Depth and Chroma Information Based Coalescence of Real World and Virtual World Images

Patent metadata

Field	Value
Publication number	US-2016148429-A1
Application number	US-201414550050-A
Country	US
Kind code	A1
Filing date	Nov 21, 2014
Priority date	Nov 21, 2014
Publication date	May 26, 2016
Grant date	—

Abstract

Methods and systems for selectively merging real-world objects into a virtual environment are disclosed. The method may include: receiving a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment; identifying at least one portion of the rendering of the real-world environment that is within a depth range and differentiable from a predetermined background; generating a merged rendering including the at least one portion of the rendering of the real-world environment into the rendering of the virtual environment; and displaying the merged rendering to a user.

First claim

What is claimed is:

1 . A method, comprising:
receiving a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment; identifying at least one portion of the rendering of the real-world environment that is within a depth range and differentiable from a predetermined background; generating a merged rendering including the at least one portion of the rendering of the real-world environment into the rendering of the virtual environment; and displaying the merged rendering to a user.

2 . The method of claim 1 , wherein the second input for rendering of the real-world environment is obtained from a perspective of the user.

3 . The method of claim 1 , wherein at least one chroma key is utilized to facilitate the identification of the at least one portion of the rendering of the real-world environment that is differentiable from the predetermined background.

4 . The method of claim 1 , further comprising:
synchronizing the first input for rendering of the virtual environment, the second input for rendering of the real-world environment, and the depth information regarding the rendering of the real-world environment based on time.

5 . The method of claim 1 , further comprising:
converting the first input for rendering of the virtual environment, the second input for rendering of the real-world environment, and the depth information regarding the rendering of the real-world environment to a common format for image processing.

6 . The method of claim 1 , further comprising:
applying geometric transformation to transform the first input for rendering of the virtual environment, the second input for rendering of the real-world environment, and the depth information regarding the rendering of the real-world environment to two-dimensional image frames.

7 . The method of claim 1 , wherein the rendering of the virtual environment and the rendering of the real-world environment are stereo video renderings formatted for being presented utilizing a head-mounted display.

8 . An apparatus, comprising:
at least one input port configured to receive a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment; and an image processor configured to:
identify at least one portion of the rendering of the real-world environment that is within a depth range and differentiable from a predetermined background;
generate a merged rendering including the at least one portion of the rendering of the real-world environment and the rendering of the virtual environment; and
provide the merged rendering for display to a user.

9 . The apparatus of claim 8 , further comprising:
an output port configured to provide the merged rendering to a display device.

10 . The apparatus of claim 9 , wherein the display device includes a head-mounted display, and wherein the merged rendering is a stereo video rendering formatted for being presented utilizing the head-mounted display.

11 . The apparatus of claim 10 , wherein the second input for rendering of the real-world environment is obtained from at least one camera co-located with the display device.

12 . The apparatus of claim 8 , wherein the rendering of the virtual environment and the rendering of the real-world environment are video streams.

13 . The apparatus of claim 8 , wherein the predetermined background includes at least one predetermined color hue.

14 . A system, comprising:
an image processor configured to:
receive a first input for rendering of a virtual environment, a second input for rendering of a real-world environment, and a depth information regarding the rendering of the real-world environment;
identify at least one portion of the rendering of the real-world environment that is: within a depth range and differentiable from a predetermined background; and
generate a merged rendering including the at least one identified portion of the rendering of the real-world environment and the rendering of the virtual environment; and
a display device coupled with the image processor and configured to display the merged rendering to a user.

15 . The system of claim 14 , wherein the display device includes a head-mounted display device, wherein the merged rendering is a stereo video rendering suitable for being presented utilizing the head-mounted display.

16 . The system of claim 15 , wherein the head-mounted display device is utilized to provide fully immersive training for the user.

17 . The system of claim 14 , wherein the second input for rendering of the real-world environment is obtained from a perspective of the user utilizing at least one camera co-located with the display device.

18 . The system of claim 17 , wherein the depth information regarding the rendering of the real-world environment is obtained from the perspective of the user utilizing the at least one camera.

19 . The system of claim 17 , wherein the depth information regarding the rendering of the real-world environment is obtained from the perspective of the user utilizing at least one of: a depth camera, a disparity map generated from two stereo cameras, and a set of pre-captured three-dimensional spatial data.

20 . The system of claim 14 , wherein the predetermined background includes at least one predetermined color hue.

Assignees

• Groppa Danilo P
• Pyczynski Loyal J
• Rockwell Collins Inc

Inventors

• Groppa Danilo P
• Pyczynski Loyal J

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G09G5/026

  Control of mixing and/or overlay of colours in general (G09G5/022 and G09G5/024 take precedence) · CPC title

• G09G3/003

  to produce spatial visual effects · CPC title

• G06F3/147

  using display panels · CPC title