Using 6DOF pose information to align images from separated cameras

Patent metadata

Field	Value
Publication number	US-12536692-B2
Application number	US-202418414839-A
Country	US
Kind code	B2
Filing date	Jan 17, 2024
Priority date	Jul 17, 2020
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.

First claim

What is claimed is:

1 . A method for aligning image content obtained from multiple different images that reflect a common environment from different poses, said method comprising:
accessing a feature map of the environment; accessing a first image showing the environment from a first pose, wherein the first image includes a first set of pixels corresponding to an object that is represented in the first image; accessing a second image showing the environment from a second pose, wherein the second image includes a second set of pixels that also correspond to the object; and triggering display of an overlaid image that is generated by:
determining that the first set of pixels and the second set of pixels commonly represent the object, wherein said determining is performed using the feature map;
selecting the second set of pixels from the second image;
aligning the second set of pixels, which are associated with the second pose, with the first set of pixels, which are associated with the first pose; and
after said aligning, overlaying the second set of pixels, which are now aligned with the first set of pixels, onto the first set of pixels in the first image, resulting in generation of the overlaid image.

2 . The method of claim 1 , wherein the first image is one of a visible light image, a low light image, or a thermal image.

3 . The method of claim 1 , wherein the second image is one of a visible light image, a low light image, or a thermal image.

4 . The method of claim 1 , wherein the first image and the second image are generated by cameras of a same modality.

5 . The method of claim 1 , wherein the feature map is a three dimensional (3D) feature map.

6 . The method of claim 1 , wherein the method further includes identifying, using machine learning, a feature point within the feature map.

7 . The method of claim 1 , wherein a field of view of the first image is larger than a field of view of the second image.

8 . The method of claim 7 , wherein the first image has a same resolution as the second image.

9 . The method of claim 1 , wherein the second image is obtained over a broadband radio connection.

10 . The method of claim 1 , wherein the second pose and the second image are obtained using a same transmission burst.

11 . A computer system that aligns image content obtained from multiple different images that reflect a common environment from different poses, said computer system comprising:
at least one processor; and at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the computer system to:
access a feature map of the environment;
access a first image showing the environment from a first pose, wherein the first image includes a first set of pixels corresponding to an object that is represented in the first image;
access a second image showing the environment from a second pose, wherein the second image includes a second set of pixels that also correspond to the object; and
trigger display of an overlaid image that is generated by:
determining that the first set of pixels and the second set of pixels commonly represent the object, wherein said determining is performed using the feature map;
selecting the second set of pixels from the second image;
aligning the second set of pixels, which are associated with the second pose, with the first set of pixels, which are associated with the first pose; and
after said aligning, overlaying the second set of pixels, which are now aligned with the first set of pixels, onto the first set of pixels in the first image, resulting in generation of the overlaid image.

12 . The computer system of claim 11 , wherein the second pose is determined using the feature map.

13 . The computer system of claim 11 , wherein overlying the second content onto the corresponding first content is performed via a reprojection operation.

14 . The computer system of claim 11 , wherein the second content represents all content from the second image.

15 . The computer system of claim 11 , wherein a border is displayed around the overlaid second content.

16 . The computer system of claim 15 , wherein the border has a circular shape.

17 . The computer system of claim 11 , wherein a size of the first image is different than a size of the second image.

18 . The computer system of claim 11 , wherein a perspective of the overlaid image is subsequently modified.

19 . A method for aligning image content obtained from multiple different images that reflect a common environment from different perspectives, said method comprising:
accessing a feature map of the environment; accessing a first image showing the environment from a first perspective, wherein the first image includes a first set of pixels corresponding to an object that is represented in the first image; accessing a second image showing the environment from a second perspective, wherein the second image includes a second set of pixels that also correspond to the object; and triggering display of an overlaid image that is generated by: determining the first set of pixels and the second set of pixels commonly represent the object, wherein said determining is performed using the feature map; selecting the second set of pixels from the second image; aligning the second set of pixels, which are now associated with the second pose, with the first set of pixels, which are associated with the first pose; and after said aligning, overlaying the second set of pixels, which are now aligned with the first set of pixels, onto the first set of pixels in the first image, resulting in generation of the overlaid image.

20 . The method of claim 19 , wherein an entirety of the second image is overlaid onto the first image to generate the overlaid image.

21 . The method of claim 1 , wherein the first image is generated by a first camera and the second image is generated by a second camera, and
wherein the method further includes: transmitting the feature map from the first camera to the second camera via an established connection that exists between the first camera and the second camera; and causing both the first camera and the second camera to relocalize using the feature map, resulting in the first camera and the second camera relying on a same coordinate system relative to the environment.

22 . The computer system of claim 1 where an optical axis of the first camera is oriented in a manner that is approximately parallel to an optical axis of the second camera.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Price Raymond Kirk
• Bleyer Michael
• Edmonds Christopher Douglas

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title