Ultra-low bitrate coding based on 3D map reconstruction and decimated sub-pictures

Patent metadata

Field	Value
Publication number	US-10609398-B2
Application number	US-201815926729-A
Country	US
Kind code	B2
Filing date	Mar 20, 2018
Priority date	Jul 28, 2017
Publication date	Mar 31, 2020
Grant date	Mar 31, 2020

Abstract

A method of ultra-low rate video encoding, including capturing a frame within a video stream, capturing a global position of the frame, capturing a heading of the frame, linking the global position and the heading to the frame, determining a foreground of the frame based on the global position of the frame and the heading of the frame, determining an at least one region of interest within the foreground, analyzing the at least one region of interest, removing a background from the frame based on the global position and the heading, wherein the background is complementary to the foreground and encoding the foreground, the global position, the heading and the at least one region of interest.

First claim

What is claimed is:

1. A method of ultra-low rate video encoding, comprising:
capturing a frame within a video stream;
capturing a global position of the frame;
capturing a heading of the frame;
linking the global position and the heading to the frame;
determining a foreground of the frame based on the global position of the frame and the heading of the frame;
determining an at least one region of interest within the foreground, wherein the region of interest includes a pedestrian and a vehicle;
analyzing the at least one region of interest;
removing a background from the frame based on the global position and the heading, wherein the background is complementary to the foreground;
encoding the foreground, the global position, the heading and the at least one region of interest;
calibrating an angle of the frame based on at least one of an accelerometer and gyroscope;
wherein, the heading is provided by at least one of the accelerometer and gyroscope.

2. The method of ultra-low rate video encoding of claim 1 further comprising, storing the global position of the frame and heading of the frame into a metadata of the frame.

3. The method of ultra-low rate video encoding of claim 2 further comprising, storing the global position and calibrated angle into the metadata of the frame.

4. The method of ultra-low rate video encoding of claim 1 further comprising, segmenting the background, the foreground and the at least one region of interest.

5. The method of ultra-low rate video encoding of claim 4 further comprising, identifying objects in the foreground.

6. The method of ultra-low rate video encoding of claim 1 further comprising, determining a scene complexity based on a remaining frame within the video stream after removal of the background.

7. The method of ultra-low rate video encoding of claim 6 further comprising, determining a scene importance based on the remaining frame within the video stream after removal of the background.

8. The method of ultra-low rate video encoding of claim 1 further comprising, quantizing the at least one object of interest in the foreground.

9. The method of ultra-low rate video encoding of claim 1 further comprising, storing a weather information and a time into a metadata of the frame.

10. The method of ultra-low rate video encoding of claim 1 further comprising, reconstructing the frame utilizing the foreground, the global position and the heading.

11. The method of ultra-low rate video encoding of claim 1 further comprising, decimating the captured frame.

12. A method of ultra-low rate video decoding, comprising:
decoding an encoded foreground of a frame within a video stream, wherein the encoded foreground includes at least a global position of the frame and a heading of the frame, wherein the foreground includes at least one region of interest that includes a pedestrian and a vehicle;
extracting the global position of the frame and the heading of the frame;
receiving a local image based on the global position of the frame and the heading of the frame;
blending the local image and the foreground; and
decoding a decimate bitstream of the frame and up-sampling if the local image is not available.

13. The method of ultra-low rate video decoding of claim 12 wherein the blending is based in part on a weather information and a time.

14. The method of ultra-low rate video decoding of claim 12 further comprising extracting an at least one region of interest from the foreground and from a metadata of the encoded foreground.

15. The method of ultra-low rate video decoding of claim 12 further comprising adaptive blending at an intersection of the local image and the foreground.

16. The method of ultra-low rate video decoding of claim 15 further comprising feature matching at the intersection of the local image and the foreground.

17. The method of ultra-low rate video decoding of claim 15 further comprising multi-band blending the intersection of the local image and the foreground.

Assignees

• Black Sesame Technologies Inc
• Black Sesame International Holding Ltd

Inventors

• Gu Qun
• Shan Jizhang

Classifications

• H04N19/23 Primary

  with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic · CPC title

• H04N19/124

  Quantisation · CPC title

• H04N21/2343

  involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} · CPC title

• H04N21/4402

  involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title