Method and device for latency reduction of an image processing pipeline

Patent metadata

Field	Value
Publication number	US-11521291-B1
Application number	US-202117215151-A
Country	US
Kind code	B1
Filing date	Mar 29, 2021
Priority date	Apr 8, 2020
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

In some implementations, a method of reducing latency associated with an image read-out operation is performed at a device including one or more processors, non-transitory memory, an image processing architecture, and an image capture device. The method includes: obtaining first image data corresponding to a physical environment; reading a first slice of the first image data into an input buffer; performing processing operations on the first slice of the first image data to obtain a first portion of second image data; reading a second slice of the first image data into the input buffer; performing the image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and generating an image frame of the physical environment based at least in part on the first and second portions of the second image data.

First claim

What is claimed is:

1. A method comprising:
at a device including one or more processors, non-transitory memory, an image processing architecture, and an image capture device including a photodiode and a front-end architecture:
obtaining, via the image capture device, first image data corresponding to a physical environment;
reading a first slice of the first image data into one or more input buffers of the image processing architecture;
performing, at the image processing architecture, one or more image processing operations on the first slice of the first image data to obtain a first portion of second image data;
reading a second slice of the first image data into the one or more input buffers of the image processing architecture;
performing, at the image processing architecture, the one or more image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and
generating an image frame of the physical environment based at least in part on the first and second portions of the second image data.

2. The method of claim 1 , wherein the one or more image processing operations correspond to one of: white balance correction, de-mosaicking, color correction, gamma correction, or sharpening.

3. The method of claim 1 , wherein the first slice of the first image data corresponds to a first row or line of RAW image data.

4. The method of claim 1 , wherein the first slice of the first image data corresponds to a first predefined portion of RAW image data.

5. The method of claim 1 , wherein the first image data corresponds to RAW image data, and wherein the image frame corresponds to an RGB image frame.

6. The method of claim 1 , further comprising:
generating a graphical environment by compositing the image frame with virtual content.

7. The method of claim 6 , further comprising:
rendering the virtual content based on a camera pose of the image capture device relative to the virtual content.

8. The method of claim 6 , further comprising:
presenting the graphical environment via a display device.

9. The method of claim 8 , wherein the device corresponds to a near-eye system that includes the display device, and wherein the image capture device corresponds to a scene-facing image sensor.

10. The method of claim 1 , wherein the front-end architecture digitizes analog image data into the first image data.

11. The method of claim 10 , further comprising:
determining a focal region based on contextual information; and
performing a pixel binning operation on the first image data based on the focal region to generate a quadtree version of the first image data.

12. The method of claim 11 , wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.

13. A device comprising:
one or more processors;
a non-transitory memory;
an image processing architecture;
an image capture device including a photodiode and a front-end architecture; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
obtain, via the image capture device, first image data corresponding to a physical environment;
read a first slice of the first image data into one or more input buffers of the image processing architecture;
perform, at the image processing architecture, one or more image processing operations on the first slice of the first image data to obtain a first portion of second image data;
read a second slice of the first image data into the one or more input buffers of the image processing architecture;
perform, at the image processing architecture, the one or more image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and
generate an image frame of the physical environment based at least in part on the first and second portions of the second image data.

14. The device of claim 13 , wherein the one or more programs further cause the device to:
generate a graphical environment by compositing the image frame with virtual content.

15. The device of claim 14 , wherein the one or more programs further cause the device to:
render the virtual content based on a camera pose of the image capture device relative to the virtual content.

16. The device of claim 14 , wherein the one or more programs further cause the device to:
determine a focal region based on contextual information; and
perform a pixel binning operation on the first image data based on the focal region to generate a quadtree version of the first image data.

17. The device of claim 16 , wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.

18. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an image processing architecture and an image capture device including a photodiode and a front-end architecture, cause the device to:
obtain, via the image capture device, first image data corresponding to a physical environment;
read a first slice of the first image data into one or more input buffers of the image processing architecture;
perform, at the image processing architecture, one or more image processing operations on the first slice of the first image data to obtain a first portion of second image data;
read a second slice of the first image data into the one or more input buffers of the image processing architecture;
perform, at the image processing architecture, the one or more image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and
generate an image frame of the physical environment based at least in part on the first and second portions of the second image data.

19. The non-transitory memory of claim 18 , wherein the one or more programs further cause the device to:
generate a graphical environment by compositing the image frame with virtual content.

20. The non-transitory memory of claim 19 , wherein the one or more programs further cause the device to:
render the virtual content based on a camera pose of the image capture device relative to the virtual content.

21. The non-transitory memory of claim 18 , wherein the one or more programs further cause the device to:
determine a focal region based on contextual information; and
perform a pixel binning operation on the first image data based on the focal region to generate a quadtree version of the first image data.

22. The non-transitory memory of claim 21 , wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.

Assignees

• Apple Inc

Inventors

• Nepveu Bertrand
• Chenier Marc-Andre
• Cote Yan
• Millette Yves

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T15/005

  General purpose rendering architectures · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06T7/73

  using feature-based methods · CPC title