Techniques to set focus in camera in a mixed-reality environment with hand gesture interaction

Patent metadata

Field	Value
Publication number	US-11770599-B2
Application number	US-202217573613-A
Country	US
Kind code	B2
Filing date	Jan 11, 2022
Priority date	May 31, 2019
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

An adjustable-focus PV (picture/video) camera in a mixed-reality head-mounted display (HMD) device operates with an auto-focus subsystem that is configured to be triggered based on location and motion of a user's hands to reduce the occurrence of auto-focus hunting during camera operations. The HMD device is equipped with a depth sensor that is configured to capture depth data from the surrounding physical environment to detect and track the user's hand location, movements, and gestures in three-dimensions. The hand tracking data from the depth sensor may be assessed to determine hand characteristics—such as which of the user's hands or part of a hand is detected, its size, motion, speed, etc.—within a particular region of interest (ROI) in the field of view of the PV camera. The auto-focus subsystem uses the assessed hand characteristics as an input to control auto-focus of the PV camera to reduce auto-focus hunting occurrences.

First claim

What is claimed:

1. A method performed by a head-mounted display (HMD) device operable by a user to optimize an auto-focus implementation, comprising:
enabling auto-focus operations for a camera in the HMD device, the camera being configured to capture a scene in a local physical environment surrounding the HMD device over a field of view (FOV);
detecting, using a gaze detection subsystem comprising a sensor in the HMD device, a location of focus of an eye of the user within the FOV; and
controlling, using the location of focus in the FOV, the auto-focus operations for the camera according to a criterion of an auto-focus subsystem, the criterion indicating that the user is interacting with a virtual object within the FOV.

2. The method of claim 1 , wherein the virtual object includes a tag that identifies a business, a direction to a place of interest, and a gift box.

3. The method of claim 1 , wherein the virtual object is positioned relative to the local physical environment.

4. The method of claim 1 , wherein the interaction with the virtual object includes moving, selecting, shrinking, or enlarging the virtual object.

5. The method of claim 1 , wherein the interaction occurs at a spatial distance away from a location of the virtual object.

6. The method of claim 1 , further comprising rendering content from the HMD device to a computing device, the content including the interaction of the user.

7. The method of claim 1 , wherein the FOV includes a statically or dynamically defined region of interest (ROI), wherein the user is unaware of the ROI.

8. The method of claim 7 , wherein the auto-focus operations for the camera comprise automatically focusing on content within the ROI according to the criterion.

9. A method comprising:
enabling auto-focus operations for a camera in a head-mounted display (HMD) device of a user, the camera being configured to capture a scene in a physical environment surrounding the HMD device over a field of view (FOV);
controlling, using a location of focus of the user within the FOV, the auto-focus operations for the camera according to a criterion, the criterion indicating that the user is interacting with a virtual object within the FOV; and
rendering content from the HMD device of the user on a computing device of a remote user, the content including the interaction with the virtual object.

10. The method of claim 9 , wherein the virtual object includes a tag that identifies a business, a direction to a place of interest, and a gift box.

11. The method of claim 9 , wherein the virtual object is positioned relative to the user.

12. The method of claim 9 , wherein the interaction with the virtual object includes moving, selecting, shrinking, or enlarging the virtual object.

13. The method of claim 9 , wherein the interaction occurs at a spatial distance away from a location of the virtual object.

14. The method of claim 9 , wherein the FOV includes a statically or dynamically defined region of interest (ROI), wherein the user is unaware of the ROI.

15. The method of claim 14 , wherein the auto-focus operations for the camera comprise automatically focusing within the ROI according to the criterion.

16. A head-mounted display (HMD) device comprising:
a camera configured to capture a scene in a physical environment surrounding the HMD device over a field of view (FOV);
a processor; and
a memory storing instructions which, when executed by the processor, cause the HMD device to:
enable auto-focus operations for the camera;
detect a location of focus within the FOV; and
controlling, using the detected location of focus, the auto-focus operations for the camera according to a criterion, the criterion indicating that a user is interacting with a virtual object within the FOV.

17. The HMD device of claim 16 , wherein the interaction occurs at a spatial distance away from a location of the virtual object.

18. The HMD device of claim 16 , the instructions when executed by the processor further cause the HMD device to render content from the HMD device of the user to a computing device of a remote user, the content including rendering the interaction of the user on the computing device of the remote user.

19. The HMD device of claim 16 , wherein the FOV includes a statically or dynamically defined region of interest (ROI), wherein the user is unaware of the ROI.

20. The HMD device of claim 19 , wherein the auto-focus operations for the camera comprise automatically focusing on content within the ROI according to the criterion.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lei Maria Cheng
• Jain Vishal
• Dangi Vikramaditya

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G03B13/36

  Autofocus systems · CPC title

• G02B7/28

  Systems for automatic generation of focusing signals · CPC title

• G02B27/0172

  characterised by optical features · CPC title