Multiple data sources of captured data into single newly rendered video feed

Patent metadata

Field	Value
Publication number	US-10992984-B2
Application number	US-201715476946-A
Country	US
Kind code	B2
Filing date	Mar 31, 2017
Priority date	Mar 31, 2017
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

A method and apparatus for generating a stream of images representing a sequence of events related to a system. The apparatus defines a field view of a user from a 3D location within an enclosed area. The apparatus comprises a network interface module for receiving, from a first data source, the sequence of events related to the system, the sequence of events comprising actions of the user performed on one or more tangible instruments of the system, and for receiving, from a second data source, images related to at least a portion of the enclosed area within the field of view of the user. The apparatus further comprises an image generator module for determining a virtual camera position in the enclosed area from the 3D location, and for rendering the stream of images from the virtual camera position to produce a rendered stream of images.

First claim

What is claimed is:

1. A video generation apparatus for generating a stream of images representing a sequence of events comprising actions performed by a user on a tangible instrument during an interactive computer simulation, the video generation apparatus comprising:
one or more processors configured to:
define a real-world field of view of a user from a 3D location within an enclosed area of a simulator;
determine a virtual camera position in the enclosed area from the 3D location;
render the stream of images from the virtual camera position to produce a rendered stream of images by
receiving an event corresponding to an action performed by the user through the tangible instrument during the interactive computer simulation;
receiving images of the tangible instrument and of at least a portion of the enclosed area, related to the action and corresponding to the real-world field of view of the user during the interactive computer simulation, the images having been captured, during the interactive computer simulation, by one or more cameras located within the enclosed area; and
generating images based on (1) a result of the action and (2) the received images of the tangible instrument and of at least the portion of the enclosed area, related to the action;

provide the rendered stream of images for display on a display device to enable monitoring of the user's interaction with the simulator, including the actions performed by the user on the tangible instrument; and
update the position of the virtual camera in response to changes in the 3D location, the changes having been captured by a motion tracker in the simulator,

wherein the rendered stream of images represents an approximation of what the user actually sees.

2. The video generation apparatus of claim 1 , wherein the images are previously rendered and rendering the stream of images comprises applying at least one mask to the images.

3. The video generation apparatus of claim 1 , wherein the captured images comprise a 3D wire frame model related to the tangible instrument within the field of view of the user and related to at least a portion of the enclosed area within the field of view of the user.

4. The video generation apparatus of claim 3 , wherein the captured images are textures and rendering the stream of images comprises applying the textures to the 3D wire frame model.

5. The video generation apparatus of claim 1 , further comprising a display for displaying the rendered stream of images and a storage for storing the rendered stream of images.

6. The video generation apparatus of claim 1 , wherein the 3D location is a fixed location within the enclosed area.

7. The video generation apparatus of claim 1 , wherein the rendering of the stream of images comprises rotating the stream of images in response to changes in an eye-level of the user.

8. The video generation apparatus of claim 1 , wherein the simulator is an interactive flight simulator and the enclosed area is a cockpit of the flight simulator.

9. The video generation apparatus of claim 1 , wherein the simulator is an interactive hospital operating room simulator or an interactive paramedic vehicle simulator.

10. The video generation apparatus of claim 1 , the one or more processors are configured to receive one or more visual-perception inhibitors affecting the user, wherein the one or more visual-perception inhibitors affect the user while performing the actions in the simulator.

11. The video generation apparatus of claim 10 , wherein rendering the stream of images comprises selectively modifying the stream of images considering the captured one or more visual-perception inhibitors.

12. A method for generating a stream of images representing a sequence of events comprising actions performed by a user on a tangible instrument during an interactive computer simulation, comprising:
defining a real-world field of view of a user from a 3D location within an enclosed area of the simulator;
determining a virtual camera position in the enclosed area from the 3D location;
rendering the stream of images from the virtual camera position by:
receiving an event corresponding to an action performed by the user through the tangible instrument during the interactive computer simulation;
receiving images of the tangible instrument and of at least a portion of the enclosed area, related to the action and corresponding to the real-world field of view of the user during the interactive computer simulation, the images having been captured, during the interactive computer simulation, by one or more cameras located within the enclosed area; and
generating images based on (1) a result of the action and (2) the received images of the tangible instrument and of at least the portion of the enclosed area, related to the action;

providing the rendered stream of images for display on a display device to enable monitoring of the user's interaction with the simulator, including the actions performed by the user on the tangible instrument; and
updating the position of the virtual camera in response to changes in the 3D location, the changes having been captured by a motion tracker in the simulator,
wherein the rendered stream of images represents an approximation of what the user actually sees.

13. The method of claim 12 , wherein the images are previously rendered and rendering the stream of images comprises applying at least one mask to the images.

14. The method of claim 12 , wherein captured images comprise a 3D wire frame model related to the tangible instrument within the field of view of the user and related to at least a portion of the enclosed area within the field of view of the user.

15. The method of claim 12 , further comprising receiving an eye-level of the user, and rendering the stream of images comprises rotating the stream of images in response to changes in the eye-level of the user.

16. The method of claim 12 , further comprising capturing one or more visual-perception inhibitors affecting the user, wherein the one or more visual-perception inhibitors affect the user while performing the actions in the simulator.

17. A non-transitory computer readable medium storing computer-readable instructions stored thereon, which when executed by a processing unit of a simulator configure the processing unit to:
define a real-world field of view of a user from a 3D location within an enclosed area of the simulator;
determine a virtual camera position in the enclosed area from the 3D location;
render the stream of images from the virtual camera position by:
receiving an event corresponding to an action performed by the user through the tangible instrument during the interactive computer simulation;
receiving images of the tangible instrument and of at least a portion of the enclosed area, related to the action and corresponding to the real-world field of view of the user, the images having been captured by one or more cameras located within the enclosed area; and
generating images based on (1) a result of the action and (2) the received images of the tangible instrument and of at least the portion of the enclosed area, related to the action;

provide the rendered stream of images for display on a display device to enable monitoring of the user's interaction with the simulator, including the actions performed by the user on the tangible instrument; and
update the position of the virtual camera in response to changes in the 3D location, the changes having been captured by a motion tracker in the simulator,
wherein the rendered stream of images represent

Assignees

• Cae Inc

Inventors

• Ireland Alexander
• Russ Jack
• Bowness David

Classifications

• G06T15/20 Primary

  Perspective computation · CPC title

• G09B9/30

  Simulation of view from aircraft · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06F16/58

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title