Generating substitute image frames based on camera motion

Patent metadata

Field	Value
Publication number	US-12047690-B2
Application number	US-202117498251-A
Country	US
Kind code	B2
Filing date	Oct 11, 2021
Priority date	Oct 23, 2020
Publication date	Jul 23, 2024
Grant date	Jul 23, 2024

Abstract

Methods and apparatus, including computer program products, for processing a stream of image frames captured by a camera system. When an image frame from the stream of image frames is dropped, a current camera movement is determined. A substitute image frame for the dropped image frame is generated. The substitute image frame comprises a reference to a previous image frame and includes one or more motion vectors estimated solely based on the determined current camera movement. Remaining image frames in the stream of image frames are encoded and the substitute image frame is inserted at the place of the dropped image frame in the stream of encoded image frames.

First claim

The invention claimed is:

1. A method, performed in a camera system, for processing a stream of image frames captured during a camera movement, the method comprising:
when an image frame from the stream of image frames is dropped from the stream of image frames due to an inability to complete all operations on the image frame within an allotted time period, determining a current camera movement, wherein the dropped image frame is no longer available in the stream of image frames;
generating a substitute image frame for the dropped image frame, wherein the substitute image frame comprises a reference to a previous image frame and includes one or more motion vectors estimated solely based on the determined current camera movement;
encoding remaining image frames in the stream of image frames and inserting the substitute image frame at the place of the dropped image frame in the stream of encoded image frames; and
adjusting data for one or more subsequent image frames such that the substitute image frame is not used as a reference image frame by the one or more subsequent image frames.

2. The method of claim 1 , wherein the current camera movement is determined based on readings from one or more of: a gyroscope and a motor in the camera system.

3. The method of claim 2 , wherein the current camera movement is determined by comparing current gyroscope and motor readings with corresponding gyroscope and motor readings for a previous image frame.

4. The method of claim 3 , wherein the previous image frame is the most recent previous image frame.

5. The method of claim 1 , wherein the one or more motion vectors comprise a single global motion vector for the entire substitute image frame.

6. The method of claim 1 , wherein the motion vectors comprise a plurality of motion vectors, each of which represents the motion of a block of pixels in the substitute image frame.

7. The method of claim 6 , wherein the plurality of motion vectors represent the motion of one or more interesting portions in the substitute image frame.

8. The method of claim 1 , wherein the substitute image frame is a one of: a forward predicted image frame, P-image frame, and bidirectionally predicted image frame, B-image frame.

9. The method of claim 1 , wherein the camera system is one of: a Pan Tilt Zoom, PTZ, camera and a Body Worn Camera, BWC.

10. The method of claim 1 , wherein the stream of image frames is a live view stream of image frames and wherein the method further comprises: simultaneously capturing a recording stream of image frames, wherein the recording stream of image frames does not include any substitute image frames.

11. The method of claim 1 , further comprising:
actively dropping the image frame to prevent the image frame being dropped due to an inability to complete all operations on the image frame within an allotted time period.

12. A camera system, comprising:
an image capturing part configured to capture a stream of image frames during a camera movement;
an image processing pipeline configured to:
when an image frame from the stream of image frames is dropped from the stream of image frames due to an inability to complete all operations on the image frame within an allotted time period, determine a current camera movement, wherein the dropped image frame is no longer available in the stream of image frames;
generate a substitute image frame for the dropped image frame, wherein the substitute image frame comprises a reference to a previous image frame and includes one or more motion vectors estimated solely based on the determined current camera movement; and

an encoder configured to encode remaining image frames in the stream of image frames and insert the substitute image frame at the place of the dropped image frame in the stream of encoded image frames, and
adjust data for one or more subsequent image frames such that the substitute image frame is not used as a reference image frame by the one or more subsequent image frames.

13. A computer program product for processing a stream of image frames captured by a camera system during a camera movement, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions being executable by a processor to perform a method comprising:
when an image frame from the stream of image frames is dropped from the stream of image frames due to an inability to complete all operations on the image frame within an allotted time period, determining a current camera movement, wherein the dropped image frame is no longer available in the stream of image frames;
generating a substitute image frame for the dropped image frame, wherein the substitute image frame comprises a reference to a previous image frame and includes one or more motion vectors estimated solely based on the determined current camera movement;
encoding remaining image frames in the stream of image frames and inserting the substitute image frame at the place of the dropped image frame in the stream of encoded image frames; and
adjusting data for one or more subsequent image frames such that the substitute image frame is not used as a reference image frame by the one or more subsequent image frames.

Assignees

• Axis Ab

Inventors

• Edpalm Viktor
• Yuan Song
• Teller Raphael

Classifications

• H04N19/587

  involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence · CPC title

• H04N19/517

  by encoding · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/139 Primary

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title