Video pipeline

Patent metadata

Field	Value
Publication number	US-11727619-B2
Application number	US-202117352080-A
Country	US
Kind code	B2
Filing date	Jun 18, 2021
Priority date	Apr 28, 2017
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

First claim

What is claimed is:

1. A device, comprising:
one or more processors;
one or more cameras configured to capture frames that include views of a user's environment; and
a display subsystem for displaying a 3D virtual view to the user;
wherein the one or more processors are configured to:
transform the frames captured by the one or more cameras into a warp space;
for each frame, resample the warp space at equal angles to generate a warp space frame;
transmit the warp space frames to a base station over a wireless connection; and
decompress compressed rendered frames received from the base station over the wireless connection and provide the rendered frames to the display subsystem for display.

2. The device as recited in claim 1 , wherein the base station renders, compresses, and transmits slices of frames to the device over the wireless connection, and wherein, to decompress the compressed rendered frames received from the base station over the wireless connection and provide the rendered frames to the display subsystem for display, the one or more processors are configured to decompress the compressed slices of the frames and provide the rendered frames to the display subsystem for display.

3. The device as recited in claim 1 , wherein the rendered frames include computer generated virtual content composited with views of the user's environment or representations of objects in the user's environment composited with views of a computer generated three-dimensional (3D) virtual world.

4. The device as recited in claim 1 , wherein the one or more processors are configured to compress the warp space frames before transmission to the base station over the wireless connection.

5. The device as recited in claim 1 , wherein the device further comprises one or more gaze tracking cameras configured to capture images of the user's eyes, wherein the one or more processors are further configured to transmit the images captured by the gaze tracking cameras to the base station over the wireless connection, and wherein the base station is configured to render the frames based at least in part on a gaze direction determined from the images captured by the one or more gaze tracking cameras and received from the device over the wireless connection.

6. The device as recited in claim 1 , wherein the device further comprises a plurality of sensors configured to capture data about the user and the user's environment, wherein the one or more processors are further configured to transmit the sensor data to the base station over the wireless connection, and wherein the base station is configured to render the frames based at least in part on the sensor data received from the device.

7. The device as recited in claim 6 , wherein the plurality of sensors includes one or more of:
one or more sensors configured to capture depth information in the environment;
one or more sensors configured to track gaze direction of the user's eyes;
one or more sensors configured to track position and motion of the device in the environment; or
one or more sensors configured to track expressions of the user's face.

8. The device as recited in claim 1 , wherein the device further comprises one or more depth sensors configured to capture range information for objects in the environment, wherein the one or more processors are further configured to transmit the range information to the base station over the wireless connection, wherein the base station is configured to render the frames based at least in part on the range information from the one or more depth sensors.

9. The device as recited in claim 1 , wherein the device further comprises an inertial-measurement unit (IMU) and one or more head pose cameras configured to track position and motion of the user in the user's environment, wherein the one or more processors are configured to:
determine position of the user's head and predict motion of the user's head based on images captured by one or more head pose cameras augmented with information received from the IMU; and
transmit head position and head motion prediction information to the base station over the wireless connection;
wherein the base station is configured to render the frames based at least in part on the head position and head motion prediction information received from the device.

10. The device as recited in claim 1 , wherein the one or more processors are further configured to:
monitor the wireless connection to the base station;
in response to detecting that the wireless connection to the base station has been lost:
render, by the one or more processors, one or more frames that include views of the user's environment based on the frames captured by the one or more cameras; and
provide the rendered one or more frames to the display subsystem for display.

11. The device as recited in claim 1 , wherein the device further comprises a memory comprising program instructions executable by at least one of the one or more processors of the device to implement a rendering application configured to generate virtual content, wherein the one or more processors are configured to composite the virtual content generated by the device with at least one of the one or more frames rendered by the device.

12. The device as recited in claim 1 , wherein the device further comprises a current frame decoder and a previous frame decoder each configured to decompress and process frames received from the base station over the wireless connection, wherein the one or more processors are configured to:
receive a compressed current frame from the base station over the wireless connection;
write the compressed current frame to a previous frame buffer and pass the compressed current frame to the current frame decoder to decompress and process the current frame; and
while the current frame decoder is decompressing and processing the current frame, simultaneously decompress and process a previous frame from the previous frame buffer on the previous frame decoder.

13. The device as recited in claim 12 , wherein the one or more processors are further configured to:
monitor the receiving of the compressed current frames from the base station over the wireless connection and the decompressing and processing of the current frames by the current frame decoder to detect missing or incomplete frames; and
upon detecting that a current frame is missing or incomplete, display the previous frame that was decompressed and processed by the previous frame decoder in place of the missing or incomplete current frame.

14. The device as recited in claim 12 , wherein processing the previous frame on the previous frame decoder includes rotating the previous frame based on a head pose prediction determined from sensor data collected by one or more sensors of the device.

15. A method, comprising:
performing, by one or more computing devices:
transforming frames that include views of a user's environment captured by one or more cameras into a warp space;
for each frame, resampling the warp space at equal angles to generate a warp space frame;
transmitting the warp space frames to a base station over a wireless connection; and
decompressing compressed rendered frames received from the base station over the wireless connection and providing the rendered frames to a display subsystem for display.

16. The method of claim 15 , further comprising:
receiving slices of frames from the base station over the wireless connection, wherein decompressing compressed rendered frames and providing the rendered frames to the display subsystem further comprises:
dec

Assignees

• Apple Inc

Inventors

• Zhang Arthur Y
• Chang Ray L
• Oriol Timothy R
• Su Ling
• Saund Gurjeet S
• Cote Guy
• Chou Jim C
• Pan Hao
• Eble Tobias
• Bar-Zeev Avi
• Zhang Sheng
• Hensley Justin A
• Stahl Geoffrey

Classifications

• H04N19/162 Primary

  User input · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T15/005 Primary

  General purpose rendering architectures · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title