Head-mounted display device configured to display a visual element at a location derived from sensor data and perform calibration

Patent metadata

Field	Value
Publication number	US-2018373318-A1
Application number	US-201715631802-A
Country	US
Kind code	A1
Filing date	Jun 23, 2017
Priority date	Jun 23, 2017
Publication date	Dec 27, 2018
Grant date	—

Abstract

A head-mounted display device includes: a camera; an inertial sensor in a fixed or adjustably fixed spatial relationship with the camera; a display; and a processor. The processor is configured to derive a first position of a feature element in an image frame in an image data sequence using an image frame, and derive a second position of the feature element using a sensor data sequence and the spatial relationship. The processor is further configured to display, with the display, a display image containing: i) a first visual element at a first location corresponding to the first position, and ii) a second visual element at a second location corresponding to the second position. The processor is further configured to perform calibration when a received signal indicates that the camera and inertial sensor are not sufficiently calibrated.

First claim

What is claimed is:

1 . A method of calibrating a relationship between a camera and an inertial sensor in a fixed or adjustably fixed spacial relationship with the camera, of a head-mounted display, the head-mounted display having a storage medium containing a first spatial relationship between the camera and the inertial sensor that was derived using: images of a real marker in a first and a second pose taken using the head-mounted display or another device, and a difference in pose between the first pose and the second pose acquired based on inertial sensor output of the head-mounted display or the other device, the method comprising:
acquiring, from the camera, an image data sequence; acquiring, from the inertial sensor, a sensor data sequence; deriving, by a processor, a first position of a feature element in an image frame in the image data sequence using the image frame; deriving a second position of the feature element using the sensor data sequence and the first spatial relationship; outputting, to a display, a display image containing: i) a first visual element at a first location corresponding to the first position, and ii) a second visual element at a second location corresponding to the second position; receiving a signal indicating whether the camera and inertial sensor are sufficiently calibrated based on a difference between the first location and the second location; and performing calibration when the received signal indicates that the camera and inertial sensor are not sufficiently calibrated.

2 . The method of claim 1 , further comprising:
calculating, by the processor, the difference between the first location and the second location after displaying the display image; and generating, by the processor, a non-calibration signal indicating that the camera and inertial sensor are not sufficiently calibrated when the difference is higher than a predetermined threshold.

3 . The method of claim 1 , wherein the signal is received from a user of the head-mounted display.

4 . The method of claim 1 , wherein calibration is performed by adjusting a stored relationship between sensor data from the inertial sensor and image data from the camera.

5 . The method of claim 4 , wherein the stored relationship includes rotational relationships on three axes.

6 . The method of claim 4 , wherein the performing calibration comprises:
receiving inputs from a user of head-mounted display while adjusting one of the camera or inertial sensor; receiving a final input from the user indicating proper calibration; and setting the stored relationship in response to receiving the final input.

7 . The method of claim 4 , wherein the adjustment is performed by the processor.

8 . The method of claim 1 , wherein the camera and inertial sensor are moved before the first and second positions of the feature element are derived.

9 . The method of claim 8 , further comprising:
deriving an initial position of the feature element in an earlier image frame in the image data sequence using the earlier image frame before deriving the first and second positions of the feature element, wherein the second position of the feature element is derived using the initial position of the feature element.

10 . The method of claim 8 , further comprising:
sending an instruction to move the camera and inertial sensor before the camera and inertial sensor are moved.

11 . A non-transitory storage medium containing a first spatial relationship between a camera and an inertial sensor in a fixed or adjustably fixed spatial relationship with the camera, that was derived using: images of a real marker in a first and a second pose taken using a device, and a difference in pose between the first pose and the second pose acquired based on inertial sensor output of the device, the non-transitory storage medium further containing program instructions that, when executed by a computer processor, cause the computer processor to perform a method, the method comprising:
acquiring an image data sequence from a camera connected to the processor; acquiring a sensor data sequence from an inertial sensor that is connected to the processor; deriving a first position of a feature element in an image frame in the image data sequence using the image frame; deriving a second position of the feature element using the sensor data sequence and the first spatial relationship; displaying, with a display connected to the processor, a display image containing: i) a first visual element at a first location corresponding to the first position, and ii) a second visual element at a second location corresponding to the second position; receiving a signal indicating whether the camera and inertial sensor are sufficiently calibrated based on a difference between the first location and the second location.

12 . The non-transitory storage medium of claim 11 , wherein the method further comprises:
performing calibration when a non-calibration signal is received indicating that the camera and inertial sensor are not sufficiently calibrated.

13 . The non-transitory storage medium of claim. 11 , wherein the method further comprises:
calculating the difference between the first location and the second location after displaying the display image; and generating a non-calibration signal indicating that the camera and inertial sensor are not sufficiently calibrated when the difference is higher than a predetermined threshold.

14 . The non-transitory storage medium of claim 11 , wherein the signal is received from a user of the camera and inertial sensor.

15 . The non-transitory storage medium of claim 11 , wherein the method further comprises:
deriving an initial position of the feature element in an earlier image frame in the image data sequence using the earlier image frame before deriving the first and second positions of the feature element, wherein the second position of the feature element is derived using the initial position of the feature element.

16 . A head-mounted display device comprising:
a camera; an inertial sensor in a fixed or adjustably fixed first spatial relationship with the camera; a storage medium containing a first spatial relationship between the camera and the inertial sensor that was derived using: images of a real marker in a first and a second pose taken using the head-mounted display or another device, and a difference in pose between the first pose and the second pose acquired based on inertial sensor output of the head-mounted display or another device; a display; and a processor configured to:
acquire an image data sequence from the camera;
acquire a sensor data sequence from the inertial sensor;
derive a first position of a feature element in an image frame in the image data sequence using the image frame;
derive a second position of the feature element using the sensor data sequence and the first spatial relationship;
display, with the display, a display image containing: i) a first visual element at a first location corresponding to the first position, and ii) a second visual element at a second location corresponding to the second position;
receive a signal indicating whether the camera and inertial sensor are sufficiently calibrated based on a difference between the first location and the second location; and
perform calibration when the received signal indicates that the camera and inertial sensor are not sufficiently calibrated.

17 . The head-mounted display device of claim 16 , wherein the processor is further configured to:
perform c

Assignees

• Seiko Epson Corp

Inventors

• Yang Yang

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title