Techniques to set focus in camera in a mixed-reality environment with hand gesture interaction

Patent metadata

Field	Value
Publication number	US-11245836-B2
Application number	US-202016933413-A
Country	US
Kind code	B2
Filing date	Jul 20, 2020
Priority date	May 31, 2019
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

An adjustable-focus PV (picture/video) camera in a mixed-reality head-mounted display (HMD) device operates with an auto-focus subsystem that is configured to be triggered based on location and motion of a user's hands to reduce the occurrence of auto-focus hunting during camera operations. The HMD device is equipped with a depth sensor that is configured to capture depth data from the surrounding physical environment to detect and track the user's hand location, movements, and gestures in three-dimensions. The hand tracking data from the depth sensor may be assessed to determine hand characteristics—such as which of the user's hands or part of a hand is detected, its size, motion, speed, etc.—within a particular region of interest (ROI) in the field of view of the PV camera. The auto-focus subsystem uses the assessed hand characteristics as an input to control auto-focus of the PV camera to reduce auto-focus hunting occurrences.

First claim

What is claimed:

1. A method performed by a head-mounted display (HMD) device operable by a user to optimize an auto-focus implementation, comprising:
enabling auto-focus operations for a camera disposed in the HMD device, the camera being configured to capture a scene in a local physical environment surrounding the HMD device over a field of view (FOV), wherein the camera is a member of a set of one or more sensors that are operatively coupled to the HMD device;
providing a gaze detection subsystem, comprising one or more sensors in the HMD device, that is configured to detect locations of focus within the FOV of one or more eyes of the user;
designating a region of interest (ROI) within the FOV, wherein a criterion of the auto-focus subsystem includes the detected location indicating that the user is interacting with one or more virtual objects within the ROI; and
using the detected focus locations to control the auto-focus operations for the camera according to one or more criteria of an auto-focus subsystem.

2. The method of claim 1 , in which the set of sensors gather data which describes the local physical environment.

3. The method of claim 2 , in which the HMD device includes a see-through mixed-reality display through which a local user observes the local physical environment, and on which the HMD device renders one or more virtual objects.

4. The method of claim 3 , in which the scene captured by the camera over the FOV and the rendered virtual objects are transmitted as content to a remote computing device.

5. The method of claim 3 , in which the scene captured by the camera over the FOV and the rendered virtual objects are mixed by the HMD device into a composite signal that is recorded.

6. The method of claim 2 , in which the auto-focus operations are triggered based at least on a portion of the physical environment in the FOV.

7. The method of claim 1 , in which the ROI includes a three-dimensional space in the local physical environment.

8. The method of claim 1 , in which the ROI is dynamically variable in at least one of size, shape, or location.

9. The method of claim 1 , in which the set of sensors include a depth sensor and further comprising capturing characteristics of one or more of the user's hands using the depth sensor for tracking one or more of the HMD device user's hands to determine whether the detected locations satisfy one or more criterion of the auto-focus subsystem.

10. The method of claim 9 , in which the characteristics of the hands include a pace of hand movements.

11. The method of claim 9 , in which the characteristics of the hands include what part of a hand.

12. The method of claim 9 , in which the characteristics of the hands include a duration of time that the one or more hands are positioned in the ROI.

13. The method of claim 1 , in which the camera comprises a PV (Photo/Video) camera.

14. One or more hardware-based non-transitory computer-readable memory devices storing computer-readable instructions which, when executed by one or more processors in a computing device that is employable by a user, cause the computing device to:
utilize a gaze detecting subsystem to capture data for locations of focus of a user's eye as the user is employing the computing device in a physical environment;
identify a region of interest (ROI) within a field of view (FOV) of the computing device, in which the computing device renders one or more virtual objects on a see-through display coupled to the computing device to enable the user to simultaneously view the physical environment and the one or more virtual objects as a mixed-reality user experience;
determine, from the captured focus location data, an extent to which the user is interacting with virtual objects that are located within the ROI; and
in response to the determining, trigger or disable an auto-focus operation of a front-facing camera, disposed on the computing device, that is configured to capture a scene comprising at least a portion of the physical environment in the FOV, in which
the auto-focus operation is triggered responsive to a determination that user interaction with the virtual objects is a point of focus within the ROI; and
the auto-focus operation is disabled responsive to a determination that user interaction with the virtual objects is transitory within the ROI.

15. The one or more hardware-based non-transitory computer-readable memory devices of claim 14 , in which the instructions further cause the computing device to capture hand tracking data from a depth sensor that is operatively coupled to the computing device and using the captured hand tracking data to trigger or disable auto-focus operations of the camera.

16. The one or more hardware-based non-transitory computer-readable memory devices of claim 14 , in which the computing device includes a head-mounted display (HMD) device, a smartphone, a tablet computer, or a portable computer.

17. The one or more hardware-based non-transitory computer-readable memory devices of claim 14 , in which the ROI is located at a central region of the FOV.

18. The one or more hardware-based non-transitory computer-readable memory devices of claim 17 , in which the gaze detection subsystem comprises an inward-facing sensor that is configured to capture images of one or more of the user's eyes.

19. A computing device configurable to be worn on a head of a user, the computing device configured to reduce unwanted hunting effects of an auto-focus subsystem associated with the computing device, comprising:
a display configured to render holograms;
an adjustable-focus PV (picture/video) camera that is operatively coupled to the auto-focus subsystem and that is configured to capture adjustably focused images of a physical environment in which the user is located;
a gaze detection subsystem configured to detect the user's gaze in the physical environment comprising one or more of gaze direction, focus direction, or focus location;
one or more processors; and
one or more hardware-based memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the computing device to:
render at least one hologram on the display, the hologram comprising a virtual object that is located in the physical environment at a known location;
capture, using the gaze detection subsystem, gaze tracking data for the user's gaze in the physical environment;
determine from the gaze tracking data whether the user's one or more hands are interacting with the hologram at the known location;
trigger operations of the auto-focus subsystem in response to a determination that the user is not interacting with the hologram; and
suppress operations of the auto-focus subsystem in response to a determination that the user is interacting with the hologram.

20. The computing device of claim 19 , in which operations of the auto-focus subsystem are suppressed in response to a determination from the gaze tracking data that the user's interactions with the hologram comprise an in-out movement of one or more of the user's hands.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lei Maria Cheng
• Jain Vishal
• Dangi Vikramaditya

Classifications

• H04N23/67

  Focus control based on electronic image sensor signals · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/62 Primary

  Control of parameters via user interfaces · CPC title

• H04N23/675

  comprising setting of focusing regions · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title