WO2008142561A2 - Method and system for processing of images - Google Patents

Method and system for processing of images Download PDF

Info

Publication number
WO2008142561A2
WO2008142561A2 PCT/IB2008/001376 IB2008001376W WO2008142561A2 WO 2008142561 A2 WO2008142561 A2 WO 2008142561A2 IB 2008001376 W IB2008001376 W IB 2008001376W WO 2008142561 A2 WO2008142561 A2 WO 2008142561A2
Authority
WO
WIPO (PCT)
Prior art keywords
format
image
patterns
images
reorganising
Prior art date
Application number
PCT/IB2008/001376
Other languages
French (fr)
Other versions
WO2008142561A3 (en
Inventor
Stephane Jean Louis Jacob
Original Assignee
Dooworks Fz Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dooworks Fz Co filed Critical Dooworks Fz Co
Priority to ES08751069T priority Critical patent/ES2383470T3/en
Priority to AT08751069T priority patent/ATE547773T1/en
Priority to EP08751069A priority patent/EP2171682B1/en
Priority to PL08751069T priority patent/PL2171682T3/en
Priority to US12/600,899 priority patent/US9478011B2/en
Priority to JP2010508927A priority patent/JP5473903B2/en
Priority to CA2687874A priority patent/CA2687874C/en
Priority to CN2008800254532A priority patent/CN101755287B/en
Publication of WO2008142561A2 publication Critical patent/WO2008142561A2/en
Publication of WO2008142561A3 publication Critical patent/WO2008142561A3/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4092Image resolution transcoding, e.g. client/server architecture
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46Embedding additional information in the video signal during the compression process
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/88Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00Indexing scheme for image generation or computer graphics
    • G06T2210/32Image data format
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/015High-definition television systems

Definitions

  • This invention relates to image processing and in particular, to processing images using different image formats
  • Interactive video generally allows a user to control the displayed area of a video, for example, the user may be able to zoom into an image
  • An example of interactive video is immersive video, which combines interactive panoramic photography with digital video
  • Interactive panoramic photography combines photographic hardware with specific computer software to allow virtual display of a real environment captured previously as a photograph
  • Ultra-wide f ⁇ eld-of-v ⁇ ew images can be captured using, for example, fish eye lenses
  • a fish eye lens has a wide-angle f ⁇ eld-of-v ⁇ ew
  • a typical fish eye lens can form an image from a 180-degree hemisphere full circle The images are typically captured, transported and viewed in High Definition (HD) resolution
  • HD resolution video is characterised by its wide format (generally 16 9 aspect ratio) and its high image definition (1920 x 1080 pixels is a usual frame size as compared with standard video definition (SD) formats where 720 x 576 pixel size is a usual frame size)
  • SD standard video definition
  • XHD very high definition
  • XHD very high definition
  • HD high definition
  • XHD formats are desirable, and sometimes necessary, in many video applications to allow the user to zoom correctly
  • emerging cameras using fish-eye lenses to take the pictures
  • Using XHD video greatly increases zoom capacity compared to HD video, allowing users to see well defined images even at high zoom, thereby greatly increasing the zoom range of emerging cameras and hence, in many applications, their effectiveness
  • a method for processing images acquired at a first format for subsequent processing according to a second format comprising dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, processing images in the second format, and converting images back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
  • the invention also provides a system for processing images acquired at a first format for subsequent processing according to a second format, comprising means for dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of pixels, means for reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, a processor for processing the images in the second format, and means for converting images back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
  • the invention further provides an encoder for encoding images in a first image format into a second image format, comprising means for dividing at least part of images in the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, means for reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control
  • a decoder for use with the system described above comprising means for converting images encoded at the second format back to the first format by reversing the mapping of the patterns applied on the reorganising step according to the control
  • HD format is the emerging commonly used format in the broadcast video sector (television, DVD etc ) and is beginning to become more popular within most corporate bodies that use video
  • a preferred embodiment uses HD as the commonly used format
  • Images having a greater amount of data can be rearranged into HD format and can be transported and processed without having to develop another Codec
  • the data can be stored using existing Codec's for example H264 or DIVX
  • XHD resolution video data acquired from fish eye lenses can be encoded into HD format, allowing widespread user access to fish eye XHD video without the need to develop bespoke technology
  • embodiments of the invention are advantageous in the handling of XHD ultra- wide field-of-v ⁇ ew images, embodiments of the invention may also be used in a wide range of other applications where it is desirable to process an image according to a format other than the format in which the image was acquired
  • each pattern is assigned a position reference, thereby facilitating reorganization of the patterns between the first and second formats
  • the patterns are reorganised into more than one image of the second format This enables resolution to be maintained even where the first format is a much greater resolution than the second format
  • the images at the second format are formed into data files which include the encryption key, enabling the image in the first format to be reassembled BRIEF DESCRIPTION OF THE DRAWINGS
  • Figure 1 is illustrative of a raw digital fisheye image from a 360 x 180 f ⁇ eld-of-v ⁇ ew camera
  • FIG. 2 is a schematic overview of an embodiment of the invention.
  • Figures 3a - 3f illustrate steps shown in figure 2 in more detail
  • the invention allows an image captured at a first format to be reformatted and processed in a second format and then converted back to the first format
  • first format is a higher definition than the second format, but this need not be the case
  • second format is a standard format, for example, High Definition or Standard Definition
  • the first format may be a standard or a non-standard format
  • the embodiment to be described uses very high definition (XHD) video source, which may be taken using a wide-angle lens such as a fish eye lens
  • XHD very high definition
  • the embodiment is not limited to XHD images and the techniques to be described may be used in any environment where the image source is of a higher definition than the channel within which it is processed
  • the embodiment is particularly useful for images acquired from fish eye, or from other ultra-wide field of view lenses, it is not limited to such images
  • the video may be in real time or a file sequence
  • the video is taken using a camera imager producing images of 1600 x 1600 pixels
  • This camera imager may be a 360 x 180 camera, which captures circular images of 180° hemisphere view
  • Cameras that produce XHD video could have many applications
  • the wide view XHD images are particularly desirable since the high-resolution images captured allow the user to view well-defined images even at high zoom
  • the effectiveness of the camera Figure 1 shows an XHD image that has been acquired using a XHD digital camera and a fish eye lens
  • the image is 1600 x 1600 pixels In the image, a portion near the centre has been selected for the user to zoom into It is important that image quality is preserved during digital zooming
  • FIG. 2 shows an overview of the steps involved in image acquisition and processing embodying the invention This figure will now be described with reference also to figures 3a and 3f which illustrate some of the steps in greater detail
  • Figure 2 shows the steps involved in acquiring and encoding an XHD fisheye video source into a commonly used resolution format
  • the commonly used format is HD resolution
  • the pixels that make up the images to be encoded are divided into blocks or patterns
  • the patterns are then reorganized into HD resolution format as will be described
  • the HD format is size 1920 x 1080 pixels
  • the XHD fish eye video data can then be compressed, for example, for transportation and/or storage
  • the patterns are preferably made up of a predetermined number of pixels
  • each pattern is a square of size 16 pixels x 16 pixels
  • the fish eye image is a circular image with black outside making up the square image
  • the size of the square for this embodiment 1600 pixels x 1600 pixels, as that defined when the circle touches the square once on each of its sides
  • This black region outside the circle has no image data and is therefore not defined into patterns This saves storage space and increases total transportation time when one desires to transport or store the encoded video data
  • some patterns will be included which are partly image pixels and partly non-image pixels
  • Each pattern is assigned a pattern number and an x,y coordinate position in the matrix of patterns This coordinate information is stored and is illustrated in figure 3d
  • a commonly used encoding standard such as the encoding standard (Frame P) in the MPEG1 , MPEG 2 and MPEG4 standards is used to define the pixels into patterns
  • a P frame is one that is encoded with reference to a past frame
  • An encoder searches for differences between a given image and a previous image on a block by block basis These blocks are called macroblocks, which are superimposed on the previous image
  • the present embodiment maps the patterns onto the macroblock structure
  • an algorithm compares the two images block by block and beyond a certain difference threshold it considers the block of the previous image to be different from that of the current image and applies a JPEG compression to the block of the present image
  • the embodiment maps the patterns of the image of figure 3c onto the macroblocks of an MPEG high definition image and produces a key which defines the position to which each pattern is mapped and which can be used to reassemble the image after further processing
  • the image is acquired by the camera at 100 and the image cropped at 102 to remove the black portions outside the circular image
  • the image is converted into the set of patterns shown in figure 3b and at 106 the relative position of each pattern is recorded
  • An encryption algorithm 108 is used to generate an encryption key
  • the HD resolution format in this embodiment, is 1920 x 1080 pixels This format can fit 120 patterns x 67 patterns of the 16 pixel patterns defined from the 1600 x 1600 pixel image Therefore, all the defined patterns of the original fish eye XHD image can be fitted into the HD format
  • the original position of each pattern is allocated a matrix position which is then stored
  • the positions are mapped into the HD format
  • a random function attributes a new position to each pattern using the encryption key
  • This key may be sent with the video data to facilitate decoding the video data
  • the patterns position information is encrypted in the encoder and decoder thereby making it possible, once the data has been transferred, to reconstitute the original image
  • a raw 1920 x 1080 pixel HD image is produced and sent, at 114, to an HD encoder 116
  • This image data is shown in figure 3f as a video file which includes video data and an encryption key code
  • This HD image may be compressed using conventional techniques, such as an MPEG or JPEG algorithm as shown at 118 and may be stored, transported, manipulated or otherwise processed in that compressed format as shown at 120 Following manipulation, the compressed data, at 122 is passed to a conventional HD decoder 124 which extracts the raw 1920 x 1080 pixel file 126 for each frame
  • the encryption key is received and the original XHD image can be reassembled based on the encryption key and the known coordinates in the starting image of each pattern
  • the patterns are reorganised in to the XHD format using the key to give the original patterns position at 130
  • the patterns are converted back into image data pixels and at 134 the raw cropped image is produced
  • the image may now be rendered as required as shown at 136
  • the encoded video data as a HD raw file of pixels 1920 x 1080 can be encoded using a third party, for example using an existing encoder such as H264 or DIVX
  • a conventional HD encoder can be used to compress the HD format XHD fish eye video
  • the video can be transported and/or stored, for example using Internet storage Details of encoding HD video, transporting and storing HD video, and encoders for carrying out the functions, are well known in the art
  • a conventional HD decoder can then be used to decode and decompress the compressed HD format XHD wide-angle f ⁇ eld-of-v ⁇ ew video HD decoders are also well known in the art
  • the embodiment desc ⁇ bed has the advantage that a non-standard source can be compressed and decompressed, and therefore transported and stored in a commonly used format This is greatly advantageous for the user and enables sources such as XHD fish eye video data to be compressed and decompressed using commonly used encoders and decoders, in this embodiment HD encoders and decoders This enables XHD video data to be transported and stored without the need for new XHD video data codecs to be developed Powerful dedicated processors and very-highspeed networks which are not widely available or presently financially viable are also not required Thus the embodiment enables the use of XHD wide-angle f ⁇ eld-of-v ⁇ ew video sources in multiple applications
  • the video data is encrypted differently during transport and storage of the data This may lead to the video data being more secure
  • video image size may vary from the disclosed example and that the image pixels could fit into an appropriate number of patterns of appropriate pattern size to fit the commonly used resolution format This may be HD or some other format such as standard definition (SD) Where necessary, the patterns of the source image may be mapped onto two or more images at the lower resolution

Abstract

XHD video is acquired from a camera fitted with an ultra-wide field-of-view lens such as a fish eye lens. The active picture portion of the images are divided into patterns each having a plurality of pixels. The patterns are assigned coordinate values and then reformatted into HD format using an encryption key which reorders the patterns. The images are processed in the HD format and then returned to XHD formation by applying the reverse reordering process under the control of the key.

Description

METHOD AND SYSTEM FOR PROCESSING OF IMAGES
FIELD OF THE INVENTION
This invention relates to image processing and in particular, to processing images using different image formats
BACKGROUND TO THE INVENTION
There have been many technological advances in recent years which have potential applications in the media industry For example, digital image capture technologies have been, and are continuing to be, developed It is desirable for the media industry to offer such new and innovative technologies which result in many new or improved applications
One such developing area is that of interactive video Interactive video generally allows a user to control the displayed area of a video, for example, the user may be able to zoom into an image An example of interactive video is immersive video, which combines interactive panoramic photography with digital video
Interactive panoramic photography combines photographic hardware with specific computer software to allow virtual display of a real environment captured previously as a photograph
Immersive video has begun to address the technical problems associated with switching from static (360°) panoramic images to panoramic video
Current technology enables a video sequence in immersive 3D mode to be viewed using a computer with which the user interacts via a peripheral such as a mouse, virtual head set, joystick or immersive panoramic screen or other input device
The techniques used also make it possible to replace several standard cameras with a single immersive camera This may be done by capturing ultra-wide fιeld-of-vιew images This is desirable for the user of the system, who can thus use the pan, tilt and zoom functions virtually, and can parameterise several virtual cameras This leads to a more realistic experience for the user who can become more involved in the virtual environment, and leads to a variety of applications Ultra-wide fιeld-of-vιew images can be captured using, for example, fish eye lenses A fish eye lens has a wide-angle fιeld-of-vιew Many variants exist A typical fish eye lens can form an image from a 180-degree hemisphere full circle The images are typically captured, transported and viewed in High Definition (HD) resolution
HD resolution video is characterised by its wide format (generally 16 9 aspect ratio) and its high image definition (1920 x 1080 pixels is a usual frame size as compared with standard video definition (SD) formats where 720 x 576 pixel size is a usual frame size)
It is desirable to capture ultra-wide field-of-vιew images in very high definition (XHD) format This can be achieved if the lens (for example a fish eye lens) is mounted on an appropriate camera Very high definition (XHD) format achieves pictures of larger size than high definition (HD) format video It is noted here, for the avoidance of doubt, that very high definition encompasses any definition higher than HD The XHD formats are desirable, and sometimes necessary, in many video applications to allow the user to zoom correctly For example, emerging cameras (using fish-eye lenses to take the pictures) are beginning to work beyond one megapixel and up to eight megapixels and beyond Using XHD video greatly increases zoom capacity compared to HD video, allowing users to see well defined images even at high zoom, thereby greatly increasing the zoom range of emerging cameras and hence, in many applications, their effectiveness
At present, the compression, transportation and storage of XHD resolution video obtained, for example, from cameras with fish eye lenses is performed by using MPEG compression which creates huge file sizes and bandwidth creating transportation and storage problems Therefore, powerful dedicated processors and very-high-speed networks are required to enable the data to be compressed, transported and stored quickly enough to be available in real time for applications These processors and networks are, at present, not widely available nor financially viable Thus, fish eye XHD video cannot be offered to a wide market until these processors and networks are improved, become widely available and financially viable Until then many applications cannot be realised The present invention aims to address this problem generally SUMMARY OF THE INVENTION
According to the invention there is provided a method for processing images acquired at a first format for subsequent processing according to a second format, comprising dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, processing images in the second format, and converting images back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
The invention also provides a system for processing images acquired at a first format for subsequent processing according to a second format, comprising means for dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of pixels, means for reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, a processor for processing the images in the second format, and means for converting images back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
The invention further provides an encoder for encoding images in a first image format into a second image format, comprising means for dividing at least part of images in the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, means for reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control
A decoder for use with the system described above comprising means for converting images encoded at the second format back to the first format by reversing the mapping of the patterns applied on the reorganising step according to the control
The inventor has appreciated that the problems described above may be avoided by converting very high definition (XHD) video into a commonly used format HD format is the emerging commonly used format in the broadcast video sector (television, DVD etc ) and is beginning to become more popular within most corporate bodies that use video A preferred embodiment uses HD as the commonly used format Images having a greater amount of data can be rearranged into HD format and can be transported and processed without having to develop another Codec Furthermore, the data can be stored using existing Codec's for example H264 or DIVX In a preferred embodiment XHD resolution video data acquired from fish eye lenses can be encoded into HD format, allowing widespread user access to fish eye XHD video without the need to develop bespoke technology
While embodiments of the invention are advantageous in the handling of XHD ultra- wide field-of-vιew images, embodiments of the invention may also be used in a wide range of other applications where it is desirable to process an image according to a format other than the format in which the image was acquired
The use of patterns enables an image to be reformatted without any loss of resolution By using an encryption key to control the reorganisation of the patterns into the second format the video data may be made more secure Alternatively a look-up table may be used to control reorganisation
Preferably the patterns at the first and second formats have the same number of pixels This has the advantage that resolution is not lost in the process of conversion between formats Preferably all the patterns within an image have the same number of pixels
Preferably a portion of the image is selected for division into patterns This is advantageous where the image is a circular image such as obtained from a fish-eye lens Only parts of the image that include active pixel data need be divided into patterns
Preferably on dividing the images into patterns, each pattern is assigned a position reference, thereby facilitating reorganisation of the patterns between the first and second formats
In one embodiment, the patterns are reorganised into more than one image of the second format This enables resolution to be maintained even where the first format is a much greater resolution than the second format
Preferably, the images at the second format are formed into data files which include the encryption key, enabling the image in the first format to be reassembled BRIEF DESCRIPTION OF THE DRAWINGS
An embodiment of the invention will now be described by way of example only, and with reference to the accompanying drawings, in which
Figure 1 is illustrative of a raw digital fisheye image from a 360 x 180 fιeld-of-vιew camera,
Figure 2 is a schematic overview of an embodiment of the invention, and
Figures 3a - 3f illustrate steps shown in figure 2 in more detail
DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION
The invention allows an image captured at a first format to be reformatted and processed in a second format and then converted back to the first format In the following example the first format is a higher definition than the second format, but this need not be the case It is preferred that the second format is a standard format, for example, High Definition or Standard Definition The first format may be a standard or a non-standard format
The embodiment to be described uses very high definition (XHD) video source, which may be taken using a wide-angle lens such as a fish eye lens It should be appreciated that the embodiment is not limited to XHD images and the techniques to be described may be used in any environment where the image source is of a higher definition than the channel within which it is processed In addition, although the embodiment is particularly useful for images acquired from fish eye, or from other ultra-wide field of view lenses, it is not limited to such images
The video may be in real time or a file sequence In this example, the video is taken using a camera imager producing images of 1600 x 1600 pixels This camera imager may be a 360 x 180 camera, which captures circular images of 180° hemisphere view Cameras that produce XHD video could have many applications The wide view XHD images are particularly desirable since the high-resolution images captured allow the user to view well-defined images even at high zoom Thus improving the detail of the environment that can be viewed and greatly increasing the zoom ability and therefore, in many uses, the effectiveness of the camera Figure 1 shows an XHD image that has been acquired using a XHD digital camera and a fish eye lens The image is 1600 x 1600 pixels In the image, a portion near the centre has been selected for the user to zoom into It is important that image quality is preserved during digital zooming
Figure 2 shows an overview of the steps involved in image acquisition and processing embodying the invention This figure will now be described with reference also to figures 3a and 3f which illustrate some of the steps in greater detail
Figure 2 shows the steps involved in acquiring and encoding an XHD fisheye video source into a commonly used resolution format In this embodiment the commonly used format is HD resolution The pixels that make up the images to be encoded are divided into blocks or patterns The patterns are then reorganized into HD resolution format as will be described In this embodiment the HD format is size 1920 x 1080 pixels In HD format the XHD fish eye video data can then be compressed, for example, for transportation and/or storage
The patterns are preferably made up of a predetermined number of pixels
Preferably all the patterns are made up of the same number of pixels Defining patterns has the advantage that the pattern format may comply with existing commonly used encoding standards Thus the encoding process may be simplified and be more financially viable It is presently preferred, as shown in figure 3, that each pattern is a square of size 16 pixels x 16 pixels
Preferably, only the desired image to be encoded is defined into patterns In this example the fish eye image is a circular image with black outside making up the square image We define the size of the square for this embodiment, 1600 pixels x 1600 pixels, as that defined when the circle touches the square once on each of its sides This black region outside the circle has no image data and is therefore not defined into patterns This saves storage space and increases total transportation time when one desires to transport or store the encoded video data At the edge of the image area, some patterns will be included which are partly image pixels and partly non-image pixels
All sides of a given pattern adjoin a side of another pattern except for those sides on patterns at the edge of the active image to be encoded For the original square image size of 1600 x 1600 pixels the image could be defined by 100 x 100 patterns each of 16 x 16 pixels Where the image has been acquired through a circular lens, as in the example shown, it is defined by fewer patterns with only the active area of the image being defined into patterns If the image is circular with sides touching the edge of the defined square image size, as in this embodiment, the area of defined patterns is calculated to be 7854 patterns This is illustrated in figures 3c and 3d
Each pattern is assigned a pattern number and an x,y coordinate position in the matrix of patterns This coordinate information is stored and is illustrated in figure 3d A commonly used encoding standard, such as the encoding standard (Frame P) in the MPEG1 , MPEG 2 and MPEG4 standards is used to define the pixels into patterns A P frame is one that is encoded with reference to a past frame An encoder searches for differences between a given image and a previous image on a block by block basis These blocks are called macroblocks, which are superimposed on the previous image The present embodiment maps the patterns onto the macroblock structure
Within the MPEG standards, an algorithm compares the two images block by block and beyond a certain difference threshold it considers the block of the previous image to be different from that of the current image and applies a JPEG compression to the block of the present image
As is explained below, the embodiment maps the patterns of the image of figure 3c onto the macroblocks of an MPEG high definition image and produces a key which defines the position to which each pattern is mapped and which can be used to reassemble the image after further processing Referring back to figure 2, the image is acquired by the camera at 100 and the image cropped at 102 to remove the black portions outside the circular image At 104 the image is converted into the set of patterns shown in figure 3b and at 106 the relative position of each pattern is recorded An encryption algorithm 108 is used to generate an encryption key
As shown at 112 in figure 2 an encryption key is used to reorganize the patterns to fit the commonly used resolution format The HD resolution format, in this embodiment, is 1920 x 1080 pixels This format can fit 120 patterns x 67 patterns of the 16 pixel patterns defined from the 1600 x 1600 pixel image Therefore, all the defined patterns of the original fish eye XHD image can be fitted into the HD format An advantage of this method is that the video source, which is to be defined into patterns, is not degraded when it is encoded into HD format That is each pixel in the area of the image defined into patterns is present when the patterns are reorganised into the commonly used format 1 pixel defined in a pattern in the video data = 1 pixel defined in a pattern in the commonly used format resolution This is desirable for multiple applications where all the information is important and the user may wish to zoom into, pan and tilt the image It will be appreciated from figure 3 that the XHD image has been divided into 7854 patterns whereas the HD format can accommodate up to 120 x 67 = 8040 patterns Thus, the patterns of the XHD image can comfortably fit within a single HD frame If the resolution of the XHD image was higher, such that there were more patterns in the XHD image than a single HD frame, the XHD image could be spread over two or more HD image frames
As can be seen from figure 3d the original position of each pattern is allocated a matrix position which is then stored The positions are mapped into the HD format A random function attributes a new position to each pattern using the encryption key This key may be sent with the video data to facilitate decoding the video data This is illustrated in figure 3e In an alternative, the patterns position information is encrypted in the encoder and decoder thereby making it possible, once the data has been transferred, to reconstitute the original image
Returning to figure 2, a raw 1920 x 1080 pixel HD image is produced and sent, at 114, to an HD encoder 116 This image data is shown in figure 3f as a video file which includes video data and an encryption key code
This HD image may be compressed using conventional techniques, such as an MPEG or JPEG algorithm as shown at 118 and may be stored, transported, manipulated or otherwise processed in that compressed format as shown at 120 Following manipulation, the compressed data, at 122 is passed to a conventional HD decoder 124 which extracts the raw 1920 x 1080 pixel file 126 for each frame At 128, the encryption key is received and the original XHD image can be reassembled based on the encryption key and the known coordinates in the starting image of each pattern Thus, at 128 the patterns are reorganised in to the XHD format using the key to give the original patterns position at 130 At 132 the patterns are converted back into image data pixels and at 134 the raw cropped image is produced The image may now be rendered as required as shown at 136
The encoded video data as a HD raw file of pixels 1920 x 1080 can be encoded using a third party, for example using an existing encoder such as H264 or DIVX The skilled person will appreciate that many other HD encoders could be used, including those in development or those not yet conceived of which achieve the purpose Thus a conventional HD encoder can be used to compress the HD format XHD fish eye video In this compressed form the video can be transported and/or stored, for example using Internet storage Details of encoding HD video, transporting and storing HD video, and encoders for carrying out the functions, are well known in the art Similarly, a conventional HD decoder can then be used to decode and decompress the compressed HD format XHD wide-angle fιeld-of-vιew video HD decoders are also well known in the art
The embodiment descπbed has the advantage that a non-standard source can be compressed and decompressed, and therefore transported and stored in a commonly used format This is greatly advantageous for the user and enables sources such as XHD fish eye video data to be compressed and decompressed using commonly used encoders and decoders, in this embodiment HD encoders and decoders This enables XHD video data to be transported and stored without the need for new XHD video data codecs to be developed Powerful dedicated processors and very-highspeed networks which are not widely available or presently financially viable are also not required Thus the embodiment enables the use of XHD wide-angle fιeld-of-vιew video sources in multiple applications
In a preferred application of the invention the video data is encrypted differently during transport and storage of the data This may lead to the video data being more secure
It will be appreciated by the skilled person that examples of use of the invention are for illustration only and that the invention could be utilised in many other ways It will also be appreciated that video image size may vary from the disclosed example and that the image pixels could fit into an appropriate number of patterns of appropriate pattern size to fit the commonly used resolution format This may be HD or some other format such as standard definition (SD) Where necessary, the patterns of the source image may be mapped onto two or more images at the lower resolution
The skilled person will also appreciate that the number of pixels in each pattern may vary depending on requirements of the commonly used resolution format or encoding technology Various other modifications to the embodiments described are possible and will occur to those skilled in the art without departing from the scope of the invention which is defined by the following claims.

Claims

A method for processing images acquired at a first format for subsequent processing according to a second format, comprising dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, processing images in the second format, and converting images back to the first format by reversing the mapping of the patterns applied on the reorganising step according to the control
A method according to claim 1 wherein the control is an encryption key
A method according to claim 1 wherein the control is a look-up table
A method according to claim 1 , 2 or 3, wherein the patterns at the first format and the second format have the same number of pixels
A method according to any of claims 1 to 4, wherein all patterns of the first format images have the same number of pixels
A method according to any preceding claim, wherein the step of dividing the image into a plurality of patterns comprises selecting a portion of the image area to be divided into patterns
A method according to claim 6, wherein the image source is an ultra-wide field- of-vιew lens and wherein the portion of the image selected corresponds to an active picture area
A method according to any preceding claim, wherein the step of dividing the images comprises assigning a position reference to each pattern
A method according to claim 8, wherein the reorganising of the patterns comprises mapping the position references of the patterns to a new position in the second format image A method according to any preceding claim, wherein the first format is of higher resolution than High Definition (HD)
A method according to any preceding claim, wherein the second format is High Definition (HD)
A method according to any preceding claim, wherein the first format is a higher resolution format than the second format
A method according to any preceding claim where the patterns are reorganised into more than one image at the second format
A method according to claim 2, wherein the patterns in the second format are processed as data files which include the encryption key, whereby the images can be converted back to the first format
A method according to any preceding claim, wherein the images are video images
A system for processing images acquired at a first format for subsequent processing according to a second format, comprising means for dividing at least part of images at the first format into a plurality of patterns, each pattern comprising a plurality of pixels, means for reorganising the plurality of patterns to fit the second image format, the reorganising comprising mapping each pattern from its position in the first image format to a position in the second image format according to a control, a processor for processing the images in the second format, and means for converting images back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
A system according to claim 16, wherein the control is an encryption key
A system according to claim 16, wherein the control is a look-up table
A system according to claim 16, 17 or 18, wherein the patterns at the first format and the second format have the same number of pixels 20 A system according to any of claims 1 to 4, wherein all patterns of the first format images have the same number of pixels
21 A system according to any of claims 16 to 20, wherein the step of dividing the image into a plurality of patterns comprises means for selecting a portion of the image area to be divided into patterns
22 A system according to claim 21 , wherein the image source is an ultra-wide field of view lens and wherein the portion of the image selected corresponds to an active picture area
23 A system according to any of claims 16 to 22, wherein the step of dividing the images comprises means for assigning a position reference to each pattern
24 A system according to claim 23, wherein the reorganising of the patterns comprises means for mapping the position references of the patterns to a new position in the second format image
25 A system according to any of claims 16 to 24, wherein the first format is of higher resolution than High Definition (HD)
26 A system according to any of claims 16 to 25, wherein the second format is High Definition (HD)
27 A system according to any of claims 16 to 26, wherein the first format is a higher resolution format than the second format
28 A system according to any of claims 16 to 27 where the patterns are reorganised into more than one image at the second format
29 A system according to claim 17, wherein the processor processes patterns in the second format as data files which include the encryption key, whereby the images can be converted back to the first format
30 A system according to any of claims 16 to 29, wherein the images are video images
31 An encoder for encoding images in a first image format into a second format, comprising means for dividing at least part of images in the first format into a plurality of patterns, each pattern comprising a plurality of image pixels, and means for reorganising the plurality of patterns to fit the image format of the second format, the reorganising comprising mapping each pattern from its position in the first format image to a position in the second format image according to a control
A decoder for use with a system according to any of claims 16 to 30 comprising means for converting images encoded at the second format back to the first format by reversing the mapping of the patterns applied in the reorganising step according to the control
PCT/IB2008/001376 2007-05-21 2008-01-22 Method and system for processing of images WO2008142561A2 (en)

Priority Applications (8)

Application Number Priority Date Filing Date Title
ES08751069T ES2383470T3 (en) 2007-05-21 2008-01-22 Method and system for image processing
AT08751069T ATE547773T1 (en) 2007-05-21 2008-01-22 METHOD AND SYSTEM FOR PROCESSING IMAGES
EP08751069A EP2171682B1 (en) 2007-05-21 2008-01-22 Method and system for processing of images
PL08751069T PL2171682T3 (en) 2007-05-21 2008-01-22 Method and system for processing of images
US12/600,899 US9478011B2 (en) 2007-05-21 2008-01-22 Method and system for processing of images
JP2010508927A JP5473903B2 (en) 2007-05-21 2008-01-22 Method and system for processing images
CA2687874A CA2687874C (en) 2007-05-21 2008-01-22 Method and system for processing of images
CN2008800254532A CN101755287B (en) 2007-05-21 2008-01-22 Method and system for processing of images

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB0709711.6A GB2449631B (en) 2007-05-21 2007-05-21 Method and system for processing of images
GBGB0709711.6 2007-05-21

Publications (2)

Publication Number Publication Date
WO2008142561A2 true WO2008142561A2 (en) 2008-11-27
WO2008142561A3 WO2008142561A3 (en) 2009-01-22

Family

ID=38234800

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2008/001376 WO2008142561A2 (en) 2007-05-21 2008-01-22 Method and system for processing of images

Country Status (11)

Country Link
US (1) US9478011B2 (en)
EP (1) EP2171682B1 (en)
JP (1) JP5473903B2 (en)
CN (1) CN101755287B (en)
AT (1) ATE547773T1 (en)
CA (1) CA2687874C (en)
ES (1) ES2383470T3 (en)
GB (1) GB2449631B (en)
HK (1) HK1123913A1 (en)
PL (1) PL2171682T3 (en)
WO (1) WO2008142561A2 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102340688B (en) * 2010-07-16 2017-02-15 于培宁 Image sequence encrypting and decrypting method
US9151914B2 (en) * 2013-02-08 2015-10-06 Avago Technologies General Ip (Singapore) Pte. Ltd. Optical communication systems and methods for minimizing reflective feedback
CN103248486B (en) * 2013-04-27 2016-12-28 天地融科技股份有限公司 Conversion equipment and display system
US9799376B2 (en) * 2014-09-17 2017-10-24 Xiaomi Inc. Method and device for video browsing based on keyframe
US11184599B2 (en) 2017-03-15 2021-11-23 Pcms Holdings, Inc. Enabling motion parallax with multilayer 360-degree video

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5430486A (en) 1993-08-17 1995-07-04 Rgb Technology High resolution video image transmission and storage
WO2001008412A1 (en) 1999-07-23 2001-02-01 Sarnoff Corporation A method and system for deterring electronic video piracy through image rearrangement

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2812773B2 (en) * 1990-02-19 1998-10-22 日立電子株式会社 Image data division encoding method and apparatus
JP3278881B2 (en) * 1991-12-13 2002-04-30 ソニー株式会社 Image signal generator
US5241382A (en) * 1992-04-25 1993-08-31 General Instrument Corporation Digital HDTV data packet format and receiver therefor
JPH06153151A (en) 1992-10-31 1994-05-31 Sony Corp Digital video signal recorder
JPH0730855A (en) * 1993-07-08 1995-01-31 Sony Corp Recording method for video data
US6870886B2 (en) * 1993-12-15 2005-03-22 Koninklijke Philips Electronics N.V. Method and apparatus for transcoding a digitally compressed high definition television bitstream to a standard definition television bitstream
US5960126A (en) * 1996-05-22 1999-09-28 Sun Microsystems, Inc. Method and system for providing relevance-enhanced image reduction in computer systems
JP2993455B2 (en) * 1997-03-10 1999-12-20 日本電気株式会社 Image and audio playback device
JP2003141562A (en) 2001-10-29 2003-05-16 Sony Corp Image processing apparatus and method for nonplanar image, storage medium, and computer program
JP2003224838A (en) * 2002-01-28 2003-08-08 Sony Corp Free view point type image display apparatus and method, charging apparatus and charging method, storage medium, and computer program
JP4331492B2 (en) * 2003-03-11 2009-09-16 アイベックステクノロジー株式会社 Image processing apparatus, image providing server apparatus, image processing method, and image processing program
JP4281950B2 (en) * 2003-07-10 2009-06-17 Kddi株式会社 Video distribution device
GB0320701D0 (en) * 2003-09-04 2003-10-01 Koninkl Philips Electronics Nv Method and apparatus for storing digital images
US7463816B2 (en) * 2003-09-18 2008-12-09 Cyberlink Corp. Method of encoding and decoding pictured data for enhancing image resolution
JP2005339313A (en) * 2004-05-28 2005-12-08 Toshiba Corp Method and apparatus for presenting image
EP1631089A1 (en) * 2004-08-30 2006-03-01 Matsushita Electric Industrial Co., Ltd. Video coding apparatus and decoding apparatus
US20070019949A1 (en) * 2005-07-19 2007-01-25 Clint Clemens Stand apparatus for photographic uses
EP1764736A1 (en) 2005-09-19 2007-03-21 Thomson Licensing Method and device for image cropping
US9077960B2 (en) * 2005-08-12 2015-07-07 Microsoft Corporation Non-zero coefficient block pattern coding
JP2007318597A (en) * 2006-05-29 2007-12-06 Opt Kk Compression method of image data by wide angle lens, compression apparatus, wide angle camera apparatus, and monitor system
WO2008055845A2 (en) 2006-11-10 2008-05-15 Innostarter Frothing liquid food

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5430486A (en) 1993-08-17 1995-07-04 Rgb Technology High resolution video image transmission and storage
WO2001008412A1 (en) 1999-07-23 2001-02-01 Sarnoff Corporation A method and system for deterring electronic video piracy through image rearrangement

Also Published As

Publication number Publication date
GB2449631A (en) 2008-12-03
WO2008142561A3 (en) 2009-01-22
CA2687874C (en) 2013-09-17
JP5473903B2 (en) 2014-04-16
ES2383470T3 (en) 2012-06-21
ATE547773T1 (en) 2012-03-15
HK1123913A1 (en) 2009-06-26
EP2171682A2 (en) 2010-04-07
CA2687874A1 (en) 2008-11-27
JP2010528516A (en) 2010-08-19
PL2171682T3 (en) 2012-09-28
CN101755287A (en) 2010-06-23
EP2171682B1 (en) 2012-02-29
GB0709711D0 (en) 2007-06-27
CN101755287B (en) 2012-10-10
US9478011B2 (en) 2016-10-25
US20100189252A1 (en) 2010-07-29
GB2449631B (en) 2012-02-15

Similar Documents

Publication Publication Date Title
CN109983757B (en) View dependent operations during panoramic video playback
CN112204993B (en) Adaptive panoramic video streaming using overlapping partitioned segments
CN109983500B (en) Flat panel projection of reprojected panoramic video pictures for rendering by an application
US10764494B2 (en) Adaptive panoramic video streaming using composite pictures
EP3434021B1 (en) Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices
CA2687874C (en) Method and system for processing of images
KR20050055579A (en) Method and apparatus for image compression and decoding using bitstream map, and recording medium thereof
WO2020175908A1 (en) Method and device for partitioning picture on basis of signaled information
JP2004007266A (en) Image encoder and its method, image decoder and its method, and program and recording medium
CN115243054A (en) Panoramic video image coding transmission method, panoramic video image acquisition method, related device and system
CN117014562A (en) Multi-channel video stream hybrid encoding and decoding method, device and storage medium

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 200880025453.2

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08751069

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2687874

Country of ref document: CA

Ref document number: 2010508927

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2008751069

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 12600899

Country of ref document: US