WO2009034424A2 - Method and system for processing of images - Google Patents
Method and system for processing of images Download PDFInfo
- Publication number
- WO2009034424A2 WO2009034424A2 PCT/IB2008/001155 IB2008001155W WO2009034424A2 WO 2009034424 A2 WO2009034424 A2 WO 2009034424A2 IB 2008001155 W IB2008001155 W IB 2008001155W WO 2009034424 A2 WO2009034424 A2 WO 2009034424A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- streams
- bit depth
- single stream
- image
- bit
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2365—Multiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4347—Demultiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440227—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64784—Data processing by the network
- H04N21/64792—Controlling the complexity of the content stream, e.g. by dropping packets
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/77—Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
- H04N9/8205—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
- H04N9/8227—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal the additional signal being at least another television signal
Definitions
- This invention relates to processing of images and in particular, to processing multiple streams of image data.
- a camera system may include multiple cameras each producing a stream of images.
- a camera may include, for example, two fish eye lenses and/or a zoom lens each producing streams of images.
- Fish eye lenses have a wide- angle-field-of-view and many variants exist.
- a typical fish eye lens can form an image from a 180-degree hemisphere full circle.
- the two fish eye lenses may, thus, be positioned back to back to capture the entire environment.
- the zoom lens may zoom in on selected areas of the environment in order to show them in more detail.
- the images captured by the zoom lens may be of high definition format.
- HD resolution video is characterised by its wide format (generally 16:9 aspect ratio) and its high image definition (1920 x 1080 pixels and 1280 x 720 pixels are usual frame sizes, as compared with standard video definition (SD) formats where 720 x 576 pixel size is a usual frame size).
- SD standard video definition
- the images captured by the fish eye lenses, mounted on an appropriate camera may be very high definition (XHD) resolution images.
- Very high definition (XHD) format achieves pictures of larger size than high definition (HD) format video. This is desirable in many applications since it increases the user's ability to digitally zoom into the environment.
- Each of the images generally has a colour depth which is supported by computers and processing hardware.
- Colour depth describes the number of bits used to represent the colour of a single pixel in a bitmapped image or video frame buffer, and is sometimes referred to as bits per pixel. Higher colour depth gives a broader range of distinct colours.
- Truecolour has 16.7 million distinct colours and mimics many colours found in the real world.
- the range of colours produced approaches the level at which the human eye can distinguish colours for most photographic images.
- some limitations may be revealed when the images are manipulated, or are black-and-white images (which are restricted to 256 levels with true colour) or "pure" generated images.
- images are captured at 24 or 32 bit colour depth in current standards.
- 24-bit truecolour uses 8 bits to represent red, 8 bits to represent blue, and 8 bits to represent green. This gives 256 shades for each of these three colours. Therefore, the shades can be combined to give a total of 16,777,216 mixed colours (256 x 256 x 256).
- 32-bit colour comprises 24-bit colour with an additional 8 bits, either as empty padding space or to represent an alpha channel.
- Many computers process data internally in units of 32 bits. Therefore, using 32 bit colour depth may be desirable since it allows speed optimisations. However, this is at the detriment of increasing the installed video memory.
- Streams either HD or XHD 1 have a known digital data format.
- the pixels represented by a standard number of bits (known colour depth), make up a bit stream of 1's and O's.
- Progressive scanning may be used where the image lines are scanned in sequential order, or interlaced scanning may be used where first the odd lines are scanned, then the even ones, for example. Generally, scanning of each line is from left to right.
- Various digital data stream formats including various numbers of headers, are possible and will be known to the skilled person.
- a known data format is any known digital format for any image format (eg HD or XHD).
- MPEG-2 is a standard defined by the Moving Picture Experts Group for digital video. It specifies the syntax of an enclosed video bit stream. In addition, it specifies semantics and methods for subsequent encoding and compression of the corresponding video streams. However, the way the actual encoding process is implemented is up to encoder design. Therefore, advantageously, all MPEG-2 compatible equipment is interoperable. At present, the MPEG-2 standard is widespread.
- MPEG-2 allows four source formats, or 'Levels', to be coded ranging from limited definition, to full HDTV - each with a range of bit rates.
- MPEG-2 allows different 'Profiles'. Each profile offers a collection of compression tools that together make up the coding system. A different profile means that a different set of compression tools is available.
- the MPEG-4 standard incorporating the H.264 compression scheme, deals with higher compression ratios covering both low and high bit rates. It is compatible with MPEG-2 streams and is set to become the predominant standard of the future.
- HDV is a commonly used recording format to produce HD video.
- the format is compatible with MPEG-2, and MPEG-2 compression may be used on the stream.
- the output from the MPEG-2 video encoders are called elementary streams (alternatively data or video bit streams).
- Elementary streams contain only one type of data and are continuous. They do not stop until the source ends. The exact format of the elementary stream will vary dependent on the codec or data carried in the stream.
- the continuous elementary bit stream may then be fed into a packetiser, which divides the elementary stream into packets of a certain number of bytes. These packets are known as Packetised Elementary Stream (PES) packets.
- PES Packetised Elementary Stream
- PES generally, contains only one type of payload data from a single encoder. Each PES packet begins with a packet header that includes a unique packet ID. The header data also identifies the source of the payload as well as ordering and timing information.
- various other stream formats building on the Packetised Elementary stream are possible.
- a hierarchy of headers may be introduced for some applications.
- the bit stream may include an overall sequence header, a group of pictures header, an individual picture header and a slice of a picture header.
- a method for processing image data representing pixels arranged in frames comprising: processing two or more streams of image data to reduce the bit depth of data representing the pixels to produce reduced bit depth streams; combining the reduced bit depth streams into a single stream having a bit depth at least equal to the sum of the bit depths of the reduced bit depth streams; delivering the single stream in a known format; and converting the single stream back into two or more streams of image data.
- An advantage of the embodiment of the invention is simultaneous processing, and hence, viewing of multiple streams. If two streams, for example, were transmitted separately over a communication link, one could end up with data from one of the streams arriving before or after the other stream, which would then give problems in concurrently displaying that data on a display.
- the embodiment of the invention avoids this problem by combining two or more streams of image data and presenting this as a single stream in a format such as HD using MPEG-2 encoding. This single stream can be transmitted and processed using conventional hardware. The synchronisation of the data from the two or more streams is guaranteed because the data is combined together to form a single stream.
- an advantage of an embodiment of the present invention is that one may guarantee the data, representing two or more streams of images, remain synchronised during transmission. That is one may guarantee that pixels of frames from one source arrive at a destination at a known time difference or at the same time as pixels from another source. For example, these frames may correspond substantially in relation to the time of capture, thus enabling simultaneous viewing of the image streams. This is advantageous for many applications, including monitoring a production line and various 360 video applications where it is desirable to view the entire environment (captured, for example, by multiple cameras) in real time.
- An additional benefit of the invention is that by reducing the colour depth, the bandwidth is reduced prior to transmission of the data.
- a reduced colour depth may be sufficient for many applications, so it is acceptable to reduce the bandwidth in this way. For example, only 8 bits colour depth (a maximum of 256 colours) is required for images taken from night time cameras. Consequently, reducing the bit depth from say 24 bits captured to 8 bits does not cause a problematic loss of quality.
- the streams can be combined into a single stream of known format.
- the length of the resultant stream need not be longer than the longest input stream. This is advantageous leading to the possibility of processing the stream using known techniques and hardware, and particularly doing so in real time. Processing only one stream also simplifies hardware arrangements for delivery of the streams, in comparison to delivering multiple streams over separate communication links. Whilst embodiments of the invention are advantageous in the processing of multiple video streams where synchronisation is desired, the invention may also be used in a wide range of other applications where it is desirable to process multiple images as a single stream.
- the images from the separate streams merged together correspond to each other, such as being captured at the same time from different sources.
- the video may be made more secure.
- a look up table may be used to convert the merged images back into their original separated form.
- FIG. 1 is a schematic overview of the functional components of an embodiment of the invention
- Figure 2 is a schematic diagram of the encoder device of the embodiment
- Figure 3 is a schematic diagram of an optional second stage of encoding of an embodiment of the invention.
- Figure 4 is a schematic diagram illustrating the decoding device of the embodiment.
- Figure 5 is a schematic diagram illustrating the encoder process for reducing and combining the reduced bit streams to produce a single stream of known format.
- the embodiment of the invention allows multiple image streams to be merged and processed as a single image stream and then converted back to the separate image streams.
- these images are captured by separate image sources which are video sources, but this is only one example.
- the embodiment to be described has three separate image sources, with an optional fourth image source. All the video sources may be in real time or a file sequence.
- the image sources are part of a camera system monitoring a production line.
- Two camera imagers are equipped with ultra-wide- angle lenses, such as fish eye lenses, which are positioned back to back to capture the entire surrounding environment (360 degrees).
- these camera imagers capture very high definition (XHD) video, which is desirable to enable the user to digitally zoom into the images effectively.
- XHD very high definition
- XHD encompasses any definition higher than HD.
- each XHD source has the same number of pixels for each image frame, since the two camera imagers are identical, producing the same format and aspect ratio of images.
- each image frame may have the same or a different number of pixels, as the XHD image frames.
- the camera system described may also combine a fourth HD camera imager.
- the embodiment is not limited to a certain number of video sources and the techniques to be described may be used with many other combinations of image sources.
- the embodiment is particularly useful for processing images of different image formats, it is not limited to such images.
- the images to be processed may be of the same format or various differing formats. These image formats may be standard or non- standard.
- Figure 1 shows the functional components of a device embodying the invention having three image sources and optional fourth image source 1 , 2, 3 and 4.
- the captured data may be processed by the processor 5, which may be equipped with memory and/or storage capacity 6 and 7 respectively.
- the image streams are processed by a device 8, which undertakes a process of merging the streams.
- the functional components of the processor 5, memory 6, storage 7 and device 8 may be embodied in a single device.
- the image sources 1, 2, 3 may be simple image capture devices such as CCD or CMOS sensors with appropriate optics and drive electronics and the processor 5, memory 6 and storage 7 undertake the processing to turn the raw image data into streams.
- the image sources 1 , 2, 3 may themselves be image cameras that produce image streams in XHD or HD format and the processor 5, memory 6 and storage 7 then has less processing formation to perform.
- colour depth reducers 12, 13 and 14 reduce the colour depth of each image stream from 24 to 8-12 bits. That is each pixel is now represented by 8 -12 bits, and the number of colours that may be represented is reduced. For example, 8 bit colour depth gives a maximum number of colours displayed at any one time of 256.
- Colour depth reducers to perform this reduction are well known in the art, using for example sampling and quantisation. Many variants exist.
- a simple technique to reduce the colour depth involves combining bits together, so that the number 0 to 65,536 is represented as the first bit, the number 65,536 to 131 ,072 is represented by the second bit and so on.
- each stream is reduced to a uniform colour depth.
- a colour depth of 8 bits or greater is suitable/sufficient for many applications, including camera systems monitoring production lines and many 360 camera applications. It should be appreciated that other reductions in colour depth may also suit or be sufficient for various other applications.
- a stream merger 15 merges the two XHD and HD video streams, with reduced colour depth, into a single stream which has an overall colour depth of 16-32 bits.
- the processor to perform the merging is called an XHD stream merger, since the image format of the resultant stream in this case is XHD.
- the merged image stream has a known digital data format and has a colour depth at least equal to the sum of the bit depths of the reduced bit depth streams. In this case, the merged image stream has a maximum bit depth of 32 bits per pixel. Standard 24 or 32 bits colour depth are preferred.
- the merged image stream takes the format size of the largest input stream - in this case, XHD.
- the pixels in the HD image may be rearranged to fit into the XHD image format. Any additional bits needed to unify the colour depth of the resulting stream may be empty padding space.
- each of 8 bits may be merged to create a single stream of 24 bits.
- the two XHD streams may have 12 bits and the HD stream 8 bits, resulting in a total colour depth of 32 bits.
- the two XHD streams, each of 12 bits could also be combined alone to create a resulting stream of 24 bits. This may be desirable, for example, if the XHD stream length is longer than the HD stream length. In the case where there are four input streams (2 x XHD and 2 x HD), all the streams, if reduced to 8 bits colour depth, could be merged to create a resulting stream of 32 bits colour depth.
- FIG. 5 shows one way of merging the three image sources, 24, 25 and 26 considering the actual digital data information.
- each of the streams has a header and data frames (ie pixels) with 24 bits.
- the number of bits per data frame (pixel) is reduced to 8 bits, as previously described.
- the 8 bit data frames from each of the sources are concatenated to produce a 24 bit data field in the standard format corresponding to one 24 bit "pixel". This produces data in a digital structure that can be processed in a standard known format but, of course, the 24 bit "pixels" would not represent an actual image. If a processor attempted to display the combined single stream, it would display images with random arrangements of pixels. In order to display the three separate image streams, the single stream must be deconstructed or decoded, as will be discussed later.
- the reduced bit depth streams may be merged to form a single stream of known format in a variety of other ways.
- alternate bits may be taken from each source's data frames to produce the merged 24 bit data frames. Such methods may be desirable to increase security.
- the two XHD streams with the same number of pixels for each image frame may be combined by taking the first pixel of a first frame from one source and the first pixel of a first frame from the second source and merging them together (by concatenation or otherwise), as described above. Similarly, the second pixel from one frame is combined with the second pixel of the other source and so on.
- Other methods for combining the streams are possible and will occur to the skilled person.
- the technique described above of concatenating or otherwise combining the reduced bit pixels may still be used.
- empty padding space may be used for example.
- image frames from the three input streams that correspond to each other are merged. Given that the images remain synchronised throughout subsequent transmission as a single stream, one can guarantee that pixels of frames from one source arrive at a destination at the same time as corresponding pixels from another source.
- multiple image frames captured at the same time would be merged into single image frames making up a single image stream.
- This enables the user to synchronise the streams, for example, according to the time point the image streams are captured.
- this enables viewing multiple video sources simultaneously in real time.
- the digital data streams would then have the first pixel, from the first image frame of one source, at exactly the same time as the first pixel from the first frame of another source. This would simplify the process of then merging the streams, since the data bit streams would already be synchronised.
- frames from one source are merged with frames taken at exactly the same time from another source. Since the images remain synchronised throughout subsequent transmission as a single stream slight misalignment may be acceptable. For example, it may be acceptable to have frames from one source merged with frames from another source that are actually a few image frames different in terms of the time they were taken. TV cameras typically have an image frame rate of 50 fields per second. It would not matter if the images merged together were a few fields or frames apart. As long as the system and the decoder knows, it can ensure that the images are displayed at the correct time at the receiver end.
- the raw 24 bit data representing each pixel from an image source is reduced in a bit depth, combined with other reduced pixels from other streams and then packaged into a chosen known format.
- the resultant data can be made compatible with MPEG-2, for example, or other compression algorithms, by applying colour reduction and merging to patterns of pixels such as 16 x 16 pattern groups.
- the chosen groupings of pixels will depend upon the chosen compression scheme.
- the bit depth reduction and merging schemes can be fixed or adaptive. If the schemes are fixed, then the encoder and decoder both need to know in advance the arrangements of the schemes. Alternatively, if the schemes are variable or adaptive, then the chosen scheme must be recorded and transmitted from the encoder to the decoder.
- the encoding scheme can be stored and transmitted as meta data, which may be referred to as a "palette combination map". This contains the information which explains how the pixels have been reduced in bit depth and combined.
- the palette combination map comprises a lookup table which explains that each pixel is reduced from 24 bits to 8 bits and then each of 3 pixels is concatenated with a corresponding pixel from a frame of another image in the order first pixel, second pixel, third pixel.
- This lookup-table or "key” can be used by the decoder to reassemble the image streams.
- the scheme used can be set once and then fixed or be adaptive as described above. If it is adaptive, the scheme could change infrequently, such as once per day, a few times a day, or could be more frequent such as changing with the changing nature of the images being transmitted. If the scheme adapts frequently, then the palette combination map will be transmitted frequently either multiplexed with the image stream data or sent by a separate channel. As this meta data is small, there should be no transmission problem, and so no risk of delay. However, to avoid the possibility that the decoder is unable to operate if the meta data fails to reach the decoder, a default fixed scheme can be used in the absence of the meta data transmisison from encoder to decoder.
- the colour depth information of the individual streams is stored.
- This information may be stored in a palette combination map generated by the XHD stream merger, where the colour depth information may be embedded in a matrix.
- This data may be encrypted to increase security.
- additional information about the individual streams is also stored, so that the merged stream may be decoded. Such information may include the number of initial images/streams, the original location of the image pixels in the separate streams.
- This data may be embedded in a matrix in the palette combination map, and may also be encrypted, so as to increase security.
- the initial image streams may now be processed as a single stream of known format. This may be done using conventional hardware, for example if the format size of the merged images is a standard. For example, as currently is the case, if the format size is HD. This format is MPEG-2 and 4 compatible. Therefore, conventional hardware could be used, for example, if the input streams to be merged were HD format.
- the format size of the resulting images is XHD.
- the compression, transportation and storage of XHD resolution video may be performed using MPEG compression which creates huge file sizes and bandwidth creating transportation and storage problems. Therefore, powerful dedicated processors and very-high-speed networks are required to enable the data to be compressed in real time for applications. These processors and networks are, at present, not widely available nor financially viable.
- a method and system for processing images acquired at a first format according to a second format may be used to convert the combined stream to a lower definition format.
- a method in which pixels are grouped into "patterns" of 16 x 16 pixels and then transmitted in a HD format could be used.
- This is shown in Figure 3 as a "Tetris" encoder and decoder.
- This is an encoding scheme for converting XHD data to HD data, but is not essential to the embodiment of the invention.
- Other conversion schemes could be used or, , indeed, the data could be kept in XHD format.
- hardware will allow XHD to be transmitted and processed and the optional conversion step shown in Figure 3 will not be needed.
- conventional HD Codec's can be used to compress and decompress the merged data if desired. In the compressed form the data can be transported and/or stored.
- Figure 2 shows an encoder 16 for converting the XHD merged stream produced by the stream merger 15, into a HD stream.
- the active picture portion of the images are divided into patterns each having a plurality of pixels.
- the patterns are assigned coordinate values and then reformatted into HD format using an encryption key which reorders the patterns.
- This encryption key may be generated by the palette combination map, but this need not be the case.
- FIG 3 shows an overview of an example of processing the single merged stream.
- the figure shows the encoder which reformats the XHD format into HD format, the resulting HD stream, and a decoder.
- the decoder converts the images back to the XHD format, by applying the reverse reordering process under the control of the key. In this case, the key is sent with the HD stream.
- This decoder is also shown in Figure 4 as decoder 17.
- the input stream information which may be stored in the palette combination map, is also sent with the single merged stream to the decoder shown in figure 4.
- the merged single stream in this case XHD
- the palette combination map including the input stream information such as number of input streams, position of images and image pixels within those streams, is also received. Using this information, the merged single stream is split back into the separated streams, two XHD and one HD, that were merged at 15. These separated streams are at the reduced colour depth.
- These separated streams may then be sent to colour depth converters at 19, 20 and 21.
- the colour depth of the separated streams may be converted back to the colour depth of the original input streams 9, 10 and 11. Therefore, converting the 8-12 bits of each reduced pixel back to 24-32 bits. It is desirable to convert the bit depth back to a standard bit depth supported by current hardware. Standard converters to perform this function are well known in the art, using techniques, such as palettes, as used by the GIF standard.
- the output streams from the colour depth convertors 19, 20 and 21 have an altered quality of colour, as compared to the input streams, due to the use of quantisation and compression used during processing.
- the inventor has appreciated that slight alteration is not obvious to the human eye and for many applications, particularly those operating in real time, such reduced quality is acceptable and outweighed by the advantages obtained.
- the output streams may now be rendered at 22 and displayed at 23.
- the display could be, for example, a 360 video player where the end user could pan tilt and zoom into a 3D world.
- the embodiment described has the advantage that multiple video streams, which may be of different formats, can be processed, that is compressed, transported and/or stored, as a single stream having a known format. This simplifies hardware arrangements required for processing. Combining the streams in this way also means that the length of the single merged stream need not be longer than the length of the longest input stream. This is useful for storage and transportation of the streams. Also, since the bandwidth is reduced prior to transmission, the method is suitable for real time applications.
- the embodiment also has the advantage that the streams remain synchronised during delivery (ie the way in which the streams are combined does not change during transmission). In this embodiment, the streams are combined so that corresponding frames in time taken are combined. In the applications described this is particularly advantageous, since it enables the full environment to be viewed simultaneously in real time.
Abstract
Description
Claims
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP08737615A EP2193660A2 (en) | 2007-09-14 | 2008-01-22 | Method and system for processing of images |
CN200880112669.2A CN101849416B (en) | 2007-09-14 | 2008-01-22 | Method and system for processing of images |
US12/678,059 US20110038408A1 (en) | 2007-09-14 | 2008-01-22 | Method and system for processing of images |
CA2699498A CA2699498A1 (en) | 2007-09-14 | 2008-01-22 | Method and system for processing of images |
JP2010524586A JP5189167B2 (en) | 2007-09-14 | 2008-01-22 | Method and system for processing images |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB0718015A GB2452765A (en) | 2007-09-14 | 2007-09-14 | Combining multiple image streams by reducing the colour depth of each bit stream combining them onto a single stream |
GBGB0718015.1 | 2007-09-14 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2009034424A2 true WO2009034424A2 (en) | 2009-03-19 |
WO2009034424A3 WO2009034424A3 (en) | 2009-05-07 |
Family
ID=38659014
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2008/001155 WO2009034424A2 (en) | 2007-09-14 | 2008-01-22 | Method and system for processing of images |
Country Status (7)
Country | Link |
---|---|
US (1) | US20110038408A1 (en) |
EP (1) | EP2193660A2 (en) |
JP (1) | JP5189167B2 (en) |
CN (1) | CN101849416B (en) |
CA (1) | CA2699498A1 (en) |
GB (1) | GB2452765A (en) |
WO (1) | WO2009034424A2 (en) |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8351766B2 (en) * | 2009-04-30 | 2013-01-08 | Honeywell International Inc. | Multi DVR video packaging for incident forensics |
US8704903B2 (en) * | 2009-12-29 | 2014-04-22 | Cognex Corporation | Distributed vision system with multi-phase synchronization |
US9503771B2 (en) | 2011-02-04 | 2016-11-22 | Qualcomm Incorporated | Low latency wireless display for graphics |
US9413985B2 (en) * | 2012-09-12 | 2016-08-09 | Lattice Semiconductor Corporation | Combining video and audio streams utilizing pixel repetition bandwidth |
CN104427378B (en) * | 2013-09-09 | 2018-03-23 | 杭州海康威视数字技术股份有限公司 | Polymorphic type business data flow transmitting device |
TWI603290B (en) * | 2013-10-02 | 2017-10-21 | 國立成功大學 | Method, device and system for resizing original depth frame into resized depth frame |
KR20160032909A (en) * | 2014-09-17 | 2016-03-25 | 한화테크윈 주식회사 | Apparatus for preprocessing of multi-image and method thereof |
WO2016123269A1 (en) * | 2015-01-26 | 2016-08-04 | Dartmouth College | Image sensor with controllable non-linearity |
CN106713922B (en) * | 2017-01-13 | 2020-03-06 | 京东方科技集团股份有限公司 | Image processing method and electronic device |
US11184599B2 (en) | 2017-03-15 | 2021-11-23 | Pcms Holdings, Inc. | Enabling motion parallax with multilayer 360-degree video |
US20200285056A1 (en) * | 2019-03-05 | 2020-09-10 | Facebook Technologies, Llc | Apparatus, systems, and methods for wearable head-mounted displays |
CN112714279A (en) * | 2019-10-25 | 2021-04-27 | 北京嗨动视觉科技有限公司 | Image display method, device and system and video source monitor |
CN115297241B (en) * | 2022-08-02 | 2024-02-13 | 白犀牛智达(北京)科技有限公司 | Image acquisition system |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0594338A2 (en) * | 1992-10-22 | 1994-04-27 | International Business Machines Corporation | Video decompression apparatus and method |
WO2001072041A2 (en) * | 2000-03-24 | 2001-09-27 | Reality Commerce Corporation | Method and system for subject video streaming |
WO2001091454A2 (en) * | 2000-05-25 | 2001-11-29 | Koninklijke Philips Electronics N.V. | Bit-plane dependent signal compression |
WO2004045217A1 (en) * | 2002-11-13 | 2004-05-27 | Koninklijke Philips Electronics N.V. | Transmission system with colour depth scalability |
US20060197777A1 (en) * | 2005-03-04 | 2006-09-07 | Samsung Electronics Co., Ltd. | Color space scalable video coding and decoding method and apparatus for the same |
US20060215875A1 (en) * | 2002-12-31 | 2006-09-28 | Medialive, A Corporation Of France | Adaptive and progressive protection of fixed images encoded in wavelets |
EP1827024A1 (en) * | 2006-02-24 | 2007-08-29 | Sharp Kabushiki Kaisha | High dynamic range video coding |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
SG44005A1 (en) * | 1992-12-11 | 1997-11-14 | Philips Electronics Nv | System for combining multiple-format multiple-source video signals |
US5948767A (en) * | 1994-12-09 | 1999-09-07 | Genzyme Corporation | Cationic amphiphile/DNA complexes |
US6249545B1 (en) * | 1997-10-14 | 2001-06-19 | Adam S. Iga | Video compression decompression and transmission |
US6560285B1 (en) * | 1998-03-30 | 2003-05-06 | Sarnoff Corporation | Region-based information compaction as for digital images |
US6263022B1 (en) * | 1999-07-06 | 2001-07-17 | Philips Electronics North America Corp. | System and method for fine granular scalable video with selective quality enhancement |
US7143432B1 (en) * | 1999-10-01 | 2006-11-28 | Vidiator Enterprises Inc. | System for transforming streaming video data |
US20060244839A1 (en) * | 1999-11-10 | 2006-11-02 | Logitech Europe S.A. | Method and system for providing multi-media data from various sources to various client applications |
JP2003274374A (en) * | 2002-03-18 | 2003-09-26 | Sony Corp | Device and method for image transmission, device and method for transmission, device and method for reception, and robot device |
KR100925195B1 (en) * | 2003-03-17 | 2009-11-06 | 엘지전자 주식회사 | Method and apparatus of processing image data in an interactive disk player |
US7487273B2 (en) * | 2003-09-18 | 2009-02-03 | Genesis Microchip Inc. | Data packet based stream transport scheduler wherein transport data link does not include a clock line |
US8683024B2 (en) * | 2003-11-26 | 2014-03-25 | Riip, Inc. | System for video digitization and image correction for use with a computer management system |
US20060282855A1 (en) * | 2005-05-05 | 2006-12-14 | Digital Display Innovations, Llc | Multiple remote display system |
JP2006339787A (en) * | 2005-05-31 | 2006-12-14 | Oki Electric Ind Co Ltd | Coding apparatus and decoding apparatus |
US20070147827A1 (en) * | 2005-12-28 | 2007-06-28 | Arnold Sheynman | Methods and apparatus for wireless stereo video streaming |
US8582658B2 (en) * | 2007-05-11 | 2013-11-12 | Raritan Americas, Inc. | Methods for adaptive video quality enhancement |
-
2007
- 2007-09-14 GB GB0718015A patent/GB2452765A/en not_active Withdrawn
-
2008
- 2008-01-22 WO PCT/IB2008/001155 patent/WO2009034424A2/en active Application Filing
- 2008-01-22 EP EP08737615A patent/EP2193660A2/en not_active Withdrawn
- 2008-01-22 JP JP2010524586A patent/JP5189167B2/en not_active Expired - Fee Related
- 2008-01-22 CA CA2699498A patent/CA2699498A1/en not_active Abandoned
- 2008-01-22 US US12/678,059 patent/US20110038408A1/en not_active Abandoned
- 2008-01-22 CN CN200880112669.2A patent/CN101849416B/en not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0594338A2 (en) * | 1992-10-22 | 1994-04-27 | International Business Machines Corporation | Video decompression apparatus and method |
WO2001072041A2 (en) * | 2000-03-24 | 2001-09-27 | Reality Commerce Corporation | Method and system for subject video streaming |
WO2001091454A2 (en) * | 2000-05-25 | 2001-11-29 | Koninklijke Philips Electronics N.V. | Bit-plane dependent signal compression |
WO2004045217A1 (en) * | 2002-11-13 | 2004-05-27 | Koninklijke Philips Electronics N.V. | Transmission system with colour depth scalability |
US20060215875A1 (en) * | 2002-12-31 | 2006-09-28 | Medialive, A Corporation Of France | Adaptive and progressive protection of fixed images encoded in wavelets |
US20060197777A1 (en) * | 2005-03-04 | 2006-09-07 | Samsung Electronics Co., Ltd. | Color space scalable video coding and decoding method and apparatus for the same |
EP1827024A1 (en) * | 2006-02-24 | 2007-08-29 | Sharp Kabushiki Kaisha | High dynamic range video coding |
Non-Patent Citations (5)
Title |
---|
CHEN YING ET AL: "4:2:0 chroma sample format for phase difference eliminating and color space scalability" ISO/IEC JTC1/SC29/WG11 AND ITU-T SG16 Q.6, XX, XX, no. JVT-O078, 16 April 2005 (2005-04-16), pages 1-13, XP002362574 * |
ERDEM A T ET AL: "Compression of 10-bit video using the tools of MPEG-2" SIGNAL PROCESSING. IMAGE COMMUNICATION, ELSEVIER SCIENCE PUBLISHERS, AMSTERDAM, NL, vol. 7, no. 1, 1 March 1995 (1995-03-01), pages 27-56, XP004047120 ISSN: 0923-5965 * |
LIU S ET AL: "A Framework for Fully Format-Independent Adaptation of Scalable Bit Streams" IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 15, no. 10, 1 October 2005 (2005-10-01), pages 1280-1290, XP011139786 ISSN: 1051-8215 * |
MALAK M ET AL: "An image database for low bandwidth communication links" DATA COMPRESSION CONFERENCE, 1991. DCC '91. SNOWBIRD, UT, USA 8-11 APRIL 1991, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 8 April 1991 (1991-04-08), pages 53-62, XP010034260 ISBN: 978-0-8186-9202-4 * |
SEGALL A ET AL: "Tone Mapping SEI Message" JOINT VIDEO TEAM (JVT) OF ISO/IEC MPEG & ITU-T VCEG(ISO/IEC JTC1/SC29/WG11 AND ITU-T SG16 Q6), XX, XX, 1 April 2006 (2006-04-01), pages 1-12, XP002435124 * |
Also Published As
Publication number | Publication date |
---|---|
CA2699498A1 (en) | 2009-03-19 |
CN101849416B (en) | 2013-07-24 |
US20110038408A1 (en) | 2011-02-17 |
EP2193660A2 (en) | 2010-06-09 |
JP2010539774A (en) | 2010-12-16 |
GB0718015D0 (en) | 2007-10-24 |
JP5189167B2 (en) | 2013-04-24 |
WO2009034424A3 (en) | 2009-05-07 |
GB2452765A (en) | 2009-03-18 |
CN101849416A (en) | 2010-09-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20110038408A1 (en) | Method and system for processing of images | |
US5995146A (en) | Multiple video screen display system | |
US5691768A (en) | Multiple resolution, multi-stream video system using a single standard decoder | |
KR101994599B1 (en) | Method and apparatus for controlling transmission of compressed picture according to transmission synchronization events | |
US5623308A (en) | Multiple resolution, multi-stream video system using a single standard coder | |
US6275263B1 (en) | Multi-function USB video capture chip using bufferless data compression | |
US20050060421A1 (en) | System and method for providing immersive visualization at low bandwidth rates | |
US20030138045A1 (en) | Video decoder with scalable architecture | |
JP2004536529A (en) | Method and apparatus for continuously receiving frames from a plurality of video channels and alternately transmitting individual frames containing information about each of the video channels to each of a plurality of participants in a video conference | |
US20070030893A1 (en) | Apparatus and method for conserving memory in a fine granularity scalability coding system | |
CN1246234A (en) | Memory architecture for multiple format video signal processor | |
JP2007306539A (en) | Method and apparatus for serving audiovisual content | |
US10334219B2 (en) | Apparatus for switching/routing image signals through bandwidth splitting and reduction and the method thereof | |
US11677972B2 (en) | Identifying tile from network abstraction unit header | |
JPH05216800A (en) | Network communication system | |
US11445160B2 (en) | Image processing device and method for operating image processing device | |
JP2009535866A (en) | Video receiver that provides video data with video attributes | |
KR20210107678A (en) | Transmission apparatus, transmission method, encoding apparatus, encoding method, reception apparatus and reception method | |
GB2482264A (en) | Combining reduced bit depth image data streams into a single, merged stream | |
KR102523959B1 (en) | Image processing device and method for operating image processing device | |
KR20100054586A (en) | System and method for multiplexing stereoscopic high-definition video through gpu acceleration and transporting the video with light-weight compression and storage media having program source thereof | |
KR102465206B1 (en) | Image processing device | |
EP0843483A2 (en) | A method for decoding encoded video data | |
JPH09247667A (en) | Dynamic image coder and dynamic image decoder | |
CA2341759A1 (en) | Compression and decompression system for digital video signals |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200880112669.2 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08737615 Country of ref document: EP Kind code of ref document: A2 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2699498 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2010524586 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2008737615 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2008737615 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 12678059 Country of ref document: US |