Low latency stabilization for head-worn displays

Patent metadata

Field	Value
Publication number	US-9898866-B2
Application number	US-201414776244-A
Country	US
Kind code	B2
Filing date	Mar 13, 2014
Priority date	Mar 13, 2013
Publication date	Feb 20, 2018
Grant date	Feb 20, 2018

Abstract

Methods, systems, and computer readable media for low latency stabilization for head-worn displays are disclosed. According to one aspect, the subject matter described herein includes a system for low latency stabilization of a head-worn display. The system includes a low latency pose tracker having one or more rolling-shutter cameras that capture a 2D image by exposing each row of a frame at a later point in time than the previous row and that output image data row by row, and a tracking module for receiving image data row by row and using that data to generate a local appearance manifold. The generated manifold is used to track camera movements, which are used to produce a pose estimate.

First claim

What is claimed is:

1. A system for low latency stabilization of a head-worn display, the system comprising:
a low latency pose tracker, including:
at least one rolling-shutter camera that captures a 2D image by exposing each row of a frame at a later point in time than a previous row and outputs image data row by row;
a tracking module for receiving image data row by row from the at least one camera and generating a local appearance manifold, using the local appearance manifold to track camera movements, and producing a pose estimate; and
a rendering unit for:
receiving pose estimates from the tracking module;
using the pose estimates to predict a future pose at a time a pixel will be drawn, the pixel being a part of an augmented reality image to be displayed; and
rendering the pixel just before the pixel is drawn.

2. The system of claim 1 wherein the at least one rolling-shutter camera comprises a plurality of rolling-shutter cameras and wherein the tracking module generates the local appearance manifold using image data received from the plurality of cameras.

3. The system of claim 2 wherein the plurality of rolling-shutter cameras are mounted in a known configuration of orientations and baseline distances between each camera.

4. The system of claim 1 wherein each of the at least one rolling-shutter camera stores data for a row into a row memory as soon as an exposure for the row is complete.

5. The system of claim 4 wherein the tracking module receives image data row by row by reading the data from the row memory.

6. The system of claim 1 wherein generating the local appearance manifold includes at least one of:
linearizing the local appearance manifold;
low-pass filtering the local appearance manifold; and
Kalman filtering the local appearance manifold.

7. The system of claim 1 wherein producing the pose estimate includes at least one of:
iteratively improving the pose estimate; and
incorporating data provided from an onboard inertial measurement unit.

8. The system of claim 1 wherein rendering the pixel just before the pixel is drawn comprises determining which row of the augmented reality image will be displayed next and rendering pixels of that row prior to a time that row is displayed.

9. The system of claim 1 wherein rendering the pixel just before the pixel is drawn includes considering a predicted change of pose between a first time T 1 that a row will be next displayed and a second time T 2 that the row will be next displayed after time T 1 and rendering the pixel in one of:
an estimated location L 1 of the pixel at the first time T 1 ;
an estimated location L 2 of the pixel at the second time T 2 ; and
a location L 3 between the estimated location L 1 and the estimated location L 2 .

10. The system of claim 1 wherein the head-worn display is for displaying to a user the augmented reality image comprising a plurality of pixels rendered by the rendering unit.

11. The system of claim 10 wherein the head-worn display comprises an optical see-through display having a transparent body through which the user can see a real scene and upon which the augmented reality image is displayed so that the augmented reality image overlays the real scene from a point of view of the user.

12. The system of claim 11 wherein the transparent body comprises one of:
a liquid crystal display (LCD);
an organic light emitting diode (OLED) display; and
a transparent surface upon which the augmented reality image is projected.

13. The system of claim 1 wherein at least some of the at least one rolling-shutter camera, the tracking module, the rendering unit, and components of the head-worn display are integrated into a semiconductor die.

14. A method for low latency stabilization of a head-worn display, the method comprising:
generating a scan line by a rolling-shutter camera that captures a 2D image by exposing each scan line of a frame at a later point in time than a previous scan line and outputs image data scan line by scan line;
using the scan line to generate a local appearance manifold;
using the local appearance manifold to track camera movements;
using the camera movements to produce pose estimates;
using the pose estimates to predict a future pose at a time a pixel will be drawn, the pixel being a part of an augmented reality image to be displayed; and
rendering the pixel just before the pixel is drawn.

15. The method of claim 14 wherein a plurality of scan lines are generated by a plurality of rolling-shutter cameras and wherein the plurality of scan lines are used to generate the local appearance manifold.

16. The method of claim 15 wherein the plurality of rolling-shutter cameras are mounted in a known configuration of orientations and baseline distances between each camera.

17. The method of claim 14 wherein the rolling-shutter camera stores data for the scan line into a scan line memory as soon as an exposure for the scan line is complete.

18. The method of claim 17 wherein using the scan line to generate the local appearance manifold includes using the data for the scan line from the scan line memory to generate the local appearance manifold.

19. The method of claim 14 wherein generating the local appearance manifold includes at least one of:
linearizing the local appearance manifold;
low-pass filtering the local appearance manifold; and
Kalman filtering the local appearance manifold.

20. The method of claim 14 wherein producing the pose estimates includes at least one of: iteratively improving the pose estimates; and incorporating data provided from an onboard inertial measurement unit.

21. The method of claim 14 wherein rendering the pixel just before the pixel is drawn comprises determining which row of the augmented reality image will be displayed next and rendering pixels of that row prior to a time that row is displayed.

22. The method of claim 14 wherein rendering the pixel just before the pixel is drawn includes considering a predicted change of pose between a first time T 1 that a row will be next displayed and a second time T 2 that the row will be next displayed after time T 1 and rendering the pixel in one of:
an estimated location L 1 of the pixel at the first time T 1 ;
an estimated location L 2 of the pixel at the second time T 2 ; and
a location L 3 between the estimated location L 1 and the estimated location L 2 .

23. The method of claim 14 comprising using the head-worn display to display to a user the augmented reality image comprising a plurality of pixels rendered.

24. The method of claim 23 wherein the head-worn display comprises an optical see-through display having a transparent body through which the user can see a real scene and upon which the augmented reality image is displayed so that the augmented reality image overlays the real scene from a point of view of the user.

25. The method of claim 24 wherein displaying the augmented reality image on the transparent body comprises displaying the augmented reality image on one of:
a liquid crystal display (LED);
an organic light emitting diode (OLED) display; and
a transparent surface upon which the augmented reality image is projected.

26. The method of claim 14 wherein at least some of the steps of generating a scan line, using the scan line to generate a local appearance manifold, using the local appearance manifold to track camera m

Assignees

• Univ North Carolina Chapel Hill

Inventors

• Fuchs Henry
• Lastra Anselmo A
• Frahm Jan Michael
• Dierk Nate Michael
• Perra David Paul

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N5/247

  Electricity · mapped topic

• G06T7/292

  Multi-camera tracking · CPC title