Head-mounted display with low light operation

Patent metadata

Field	Value
Publication number	US-11733518-B2
Application number	US-202016856080-A
Country	US
Kind code	B2
Filing date	Apr 23, 2020
Priority date	Apr 26, 2019
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

A display system includes a controller and a head-mounted display. The head-mounted display includes a display, a head support coupled to the display for supporting the display on a head of a user to be viewed by the user, and sensors coupled to the head support for sensing an environment from the head-mounted display unit in low light. The sensors include one or more of an infrared sensor for sensing the environment with infrared electromagnetic radiation, or a depth sensor for sensing distances to objects of the environment, and also include an ultrasonic sensor for sensing the environment with ultrasonic sound waves. The controller determines graphical content according to the sensing of the environment with the one or more of the infrared sensor or the depth sensor and with the ultrasonic sensor, and operates the display to provide the graphical content concurrent with the sensing of the environment.

First claim

What is claimed is:

1. A display system comprising:
a controller; and
a head-mounted display unit having a display, a head support coupled to the display for supporting the display on a head of a user to be viewed by the user, and low light sensors coupled to the head support for sensing an environment from the head-mounted display unit in low light, the sensors comprising:
an infrared sensor for sensing the environment with infrared electromagnetic radiation;
a depth sensor for sensing distances to physical objects in the environment; and
an ultrasonic sensor that transmits ultrasonic waves and detects the ultrasonic waves reflected by physical objects in the environment for sensing the physical objects in the environment;
wherein the controller determines graphical content including a rendering of the environment, the rendering determined according to the sensing of the environment with the infrared sensor, the depth sensor, and the ultrasonic sensor, and operates the display to provide the graphical content concurrent with the sensing of the environment.

2. The display system according to claim 1 , wherein the graphical content includes a stored visible light image or a rendering of another object detected by the one or more of the infrared sensor or the depth sensor.

3. The display system according to claim 1 , wherein the rendering includes an infrared graphical component determined according to the infrared sensor, a depth graphical component determined according to the depth sensor, and an ultrasonic graphical component determined according to the ultrasonic sensor.

4. The display system according to claim 3 , wherein the infrared graphical component, the depth graphical component, and the ultrasonic graphical component are determined independent of each other.

5. The display system according to claim 1 , wherein the rendering includes a combined graphical component determined cooperatively according to sensing of the environment with the infrared sensor, the ultrasonic sensor, and the depth sensor.

6. The display system according to claim 5 , further comprising a visible light camera coupled to the head-mounted display unit for sensing the environment with visible light in high light prior to the sensing the environment in the low light, wherein visible light sensor data obtained from the visible light camera is stored with a storage of the display system; and
wherein the controller determines the combined graphical component according to the visible light sensor data in the storage.

7. The display system according to claim 3 , wherein the depth sensor is a structured light sensor that senses the environment with structured light.

8. The display system according to claim 1 , further comprising a visible light camera coupled to the head-mounted display unit for sensing the environment with visible light in high light prior to the sensing the environment in the low light, wherein visible light sensor data obtained from the visible light camera is stored with a storage of the display system; and
wherein the controller determines the graphical content according to the visible light sensor data in the storage.

9. The display system according to claim 8 , wherein the visible light sensor data includes visible light images, and the graphical content includes the visible light images.

10. The display system according to claim 8 , wherein sensor data obtained from the infrared sensor, the depth sensor, and the ultrasonic sensor in high light is stored in association with the visible light sensor data obtained concurrent therewith.

11. The display system according to claim 1 , wherein the low light is ambient light having luminance below approximately 10{circumflex over ( )}−3.5 cd/m{circumflex over ( )}2.

12. A display system comprising:
a controller; and
a head-mounted display unit having a display for displaying graphical content to a user wearing the head-mounted display unit and sensors for sensing an environment from the head-mounted display unit, the sensors including a visible light camera and a movement sensor;
wherein in high light conditions, visible light images of the environment are obtained by the visible light camera, information describing a position and orientation of the head-mounted display unit is obtained by the movement sensor, and the visible light images are stored in storage in association with the information describing the position and orientation of the head-mounted display unit; and
wherein in low light conditions, a current position and orientation of the head-mounted display unit is obtained from the movement sensor, the visible light images are recalled from the storage according to a comparison of the current position and orientation of the head-mounted display unit to the information describing the position and orientation of the head-mounted display unit that is stored in association with the visible light images, the controller determines the graphical content that includes a rendering of the environment based on the visible light images recalled from the storage, and outputs the graphical content using the display.

13. The display system according to claim 12 , wherein the controller determines the graphical content to include the visible light images.

14. The display system according to claim 12 , wherein the controller compares the current position and orientation of the head-mounted display unit to a previous position and orientation of the head-mounted display unit obtained in high light conditions, recalls the visible light images according to the comparison, and determines the graphical content according to the recalling of the visible light images.

15. The display system according to claim 12 , wherein the current position and orientation of the head-mounted display unit matches the previous position and orientation of the head-mounted display unit.

16. A method of providing graphical content with a display system, comprising:
sensing an environment with sensors coupled to a head-mounted display unit to obtain sensor data in low light, the sensors including an infrared sensor, a depth sensor, and an ultrasonic sensor that transmits ultrasonic waves to the environment and detects the ultrasonic waves reflected by physical objects in the environment for sensing the physical objects;
determining a rendering of the environment that includes an ultrasonic graphical component based on the sensor data obtained with the ultrasonic sensor, an infrared graphical component based on the sensor data obtained with the infrared sensor, and a depth graphical component based on the sensor data obtained with the depth sensor; and
outputting the rendering of the environment with a display of the head-mounted display unit.

17. The method according to claim 16 , wherein the rendering of the environment further includes stored images of the objects that were previously obtained in high light.

18. The method according to claim 16 , further comprising:
comparing, by the controller, the sensor data with stored sensor data obtained with the sensors in high light, and recalling visible light images of the stored sensor data;
wherein the rendering of the environment further includes the visible light images.

19. The method according to claim 16 , wherein determining the rendering of the environment includes independently processing the sensor data obtained from the infrared sensor, the depth sensor, and the ultrasonic sensor;
wherein determining the graphical content includes determining the infrared graphical

Assignees

• Apple Inc

Inventors

• Poore Killian J
• Dey Stephen E
• Ness Trevor J
• Rothkopf Fletcher R

Classifications

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G02B27/017

  Head mounted · CPC title