Low motion to photon latency rapid target acquisition

Patent metadata

Field	Value
Publication number	US-12320980-B2
Application number	US-202318384572-A
Country	US
Kind code	B2
Filing date	Oct 27, 2023
Priority date	Dec 1, 2020
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

First claim

What is claimed is:

1. A system that updates a position of a bounding element included in an image, said system comprising:
at least one processor; and
at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the system to:
access a first image of an environment, the first image being generated from a first perspective view of the environment;
access a second image of the environment, the second image being generated from a second perspective view of the environment, wherein first content from the first camera image reflects a same portion of the environment as is reflected in second content from the second camera image;
generate an image by overlaying the second content onto the first content, wherein the second content is surrounded by a bounding element in the image, and wherein pixels that originated from the first content and that are encompassed by the bounding element have lower resolutions than pixels that originated from the second content and that are encompassed by the bounding element; and
shift the bounding element to a new position in the image in response to a detected event.

2. The system of claim 1 , wherein the first image is generated by a first camera, and wherein the second image is generated by a second, different camera.

3. The system of claim 1 , wherein the bounding element has a circular shape.

4. The system of claim 1 , wherein the detected event is a movement of a camera that generated at least one of the first image or the second image.

5. The system of claim 1 , wherein overlaying the second content onto the first content is performed using feature matching.

6. The system of claim 1 , wherein, in addition to the bounding element, one or more additional image artifacts are included in the image.

7. The system of claim 1 , wherein inertial measurement unit (IMU) data is relied on when determining how to shift the bounding element to the new position.

8. The system of claim 1 , wherein inertial measurement unit (IMU) data for a camera that generated at least one of the first image or the second image is obtained, the IMU data detailing a movement of the camera, and wherein the IMU data is relied on to determine how to shift the bounding element to the new position.

9. The system of claim 1 , wherein, after shifting the bounding element, the bounding element encompasses pixels that originated from both the first content and the second content.

10. The system of claim 1 , wherein, prior to shifting the bounding element, the bounding element encompassed pixels that originated only from the second content.

11. A method comprising:
accessing a first image of an environment, the first image being generated from a first perspective view of the environment;
accessing a second image of the environment, the second image being generated from a second perspective view of the environment, wherein first content from the first camera image reflects a same portion of the environment as is reflected in second content from the second camera image;
generating an image by overlaying the second content onto the first content, the second content being surrounded by a bounding element in the image, wherein pixels that originated from the first content and that are encompassed by the bounding element have lower resolutions than pixels that originated from the second content and that are encompassed by the bounding element; and
shifting the bounding element to a new position in the image in response to a detected event.

12. The method of claim 11 , wherein the detected event is a movement event of a camera that generated at least one of the first image or the second image.

13. The method of claim 12 , wherein the movement event occurred between a time when one or both of the first image and the second image was generated and when the image was generated.

14. The method of claim 11 , wherein, after the bounding element is shifted, the bounding element encompasses pixels that originated from both the first content and the second content.

15. The method of claim 11 , wherein shifting the bounding element to the new position does not trigger a reprojection of image content.

16. The method of claim 11 , wherein the bounding element is symmetrical in shape.

17. A method comprising:
accessing a first image of an environment, the first image reflecting a first perspective view of the environment;
accessing a second image of the environment, the second image reflecting a second perspective view of the environment, wherein first content from the first camera image reflects a same perspective view of the environment as is reflected in second content from the second camera image;
generating an enhanced image by overlaying the second content onto the first content, the second content in the enhanced image being surrounded by a bounding element, wherein pixels that originated from the first content and that are encompassed by the bounding element have lower resolutions than pixels that originated from the second content and that are encompassed by the bounding element; and
selecting a new position for the bounding element in response to a detected event.

18. The method of claim 17 , wherein the detected event is a movement event of a particular unit of hardware.

19. The method of claim 18 , wherein the particular unit of hardware is a camera.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Sinha Sudipta Narayan
• Edmonds Christopher Douglas
• Price Raymond Kirk

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title