Ubiquitous natural user system

Patent metadata

Field	Value
Publication number	US-9395810-B2
Application number	US-201313903242-A
Country	US
Kind code	B2
Filing date	May 28, 2013
Priority date	May 28, 2013
Publication date	Jul 19, 2016
Grant date	Jul 19, 2016

Abstract

A system is provided for that includes sensor(s) configured to provide sensed input characteristic of a user, the user's environment or the user's interaction with their environment, and including from a 3D scanner, measurements of points on a surface of an object in the user's environment. A front-end system may receive and process the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of electronic resource(s). The front-end system may form and communicate an input to the electronic resource(s) to cause the resource(s) to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a 3D model of the object. And the front-end system may receive an output including the 3D model from the resource(s), and communicate the output for display by a display device.

First claim

What is claimed is:

1. A ubiquitous natural user system comprising: one or more sensors configured to sense a user, an environment of the user or an interaction of the user with the environment, and provide sensed input characteristic thereof, the sensors including a three-dimensional scanner configured to provide measurements of points on a surface of an object in the environment, and the sensed input including the measurements provided by the three-dimensional scanner; and
a front-end system coupled to the one or more sensors, the front-end system being configured to receive and process the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of one or more electronic resources, the front-end system being configured to process the sensed input according to one or more intent algorithms using one or more information models or one or more historical intent-operation associations,
wherein the front-end system is configured to form and communicate an input to cause the one or more electronic resources to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a three-dimensional model of the object, and
wherein the front-end system is configured to receive an output including the three-dimensional model from the one or more electronic resources, and communicate the output for display by a display device.

2. The ubiquitous natural user system of claim 1 , wherein the front-end system is configured to communicate the output for display by a personal display system that includes the display device, the personal display system being configured to generate or enable a live or direct view of the environment, augmented by the output displayed by the display device.

3. The ubiquitous natural user system of claim 2 , wherein the front-end system is configured to communicate the output for display by a plurality of personal display systems that include respective display devices and are wearable by a respective plurality of users in the environment, and
wherein the three-dimensional model displayed by the respective display devices is anchored to the environment, but varied for each of the personal display systems depending on viewpoints of the respective users within the environment.

4. The ubiquitous natural user system of claim 1 , wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and
wherein the front-end system is configured to communicate the output for display to the user or another user in an instance in which the environment of the user or other user includes an empty space of the complex system designed for placement of the component, a live or direct view of the empty space being augmented by the three-dimensional model.

5. The ubiquitous natural user system of claim 1 , wherein the one or more sensors further include a sensor configured to capture a gesture of the user indicating alteration to a portion of the three-dimensional model, and the sensed input includes the gesture captured by the sensor,
wherein the front-end system is configured to receive and process the sensed input including the gesture captured by the sensor, and form and communicate another input to cause the same or another one or more electronic resources to carry out the alteration indicated by the gesture, and
wherein the front-end system is configured to receive an output including at least the altered portion of the three-dimensional model, and communicate the output for display of the three-dimensional model including the altered portion.

6. The ubiquitous natural user system of claim 1 , wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and
wherein the front-end system is further configured to receive an output from an electronic resource simulation of operation of the complex system, and communicate the output for display by the display device.

7. The ubiquitous natural user system of claim 1 , wherein the front-end system is configured to communicate the output for display by a plurality of display devices of a respective plurality of users, the output for each of the display devices being varied depending on roles of the respective users.

8. A method comprising:
sensing by one or more sensors, a user, an environment of the user or an interaction of the user with the environment, and providing sensed input characteristic thereof, the sensors including a three-dimensional scanner providing measurements of points on a surface of an object in the environment, and the sensed input including the measurements provided by the three-dimensional scanner, the sensed input being processed according to one or more intent algorithms using one or more information models or one or more historical intent-operation associations;
receiving and processing the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of one or more electronic resources;
forming and communicating an input to cause the one or more electronic resources to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a three-dimensional model of the object; and
receiving an output including the three-dimensional model from the one or more electronic resources, and communicating the output for display by a display device.

9. The method of claim 8 , wherein the output is communicated for display by a personal display system that includes the display device, the personal display system being configured to generate or enable a live or direct view of the environment, augmented by the output displayed by the display device.

10. The method of claim 9 , wherein the output is communicated for display by a plurality of personal display systems that include respective display devices and are wearable by a respective plurality of users in the environment, and
wherein the three-dimensional model displayed by the respective display devices is anchored to the environment, but varied for each of the personal display systems depending on viewpoints of the respective users within the environment.

11. The method of claim 8 , wherein the object in the environment in which the three-dimensional scanner provides measurements includes a mockup of a component of a complex system, and
wherein the output is communicated for display to the user or another user in an instance in which the environment of the user or other user includes an empty space of the complex system designed for placement of the component, a live or direct view of the empty space being augmented by the three-dimensional model.

12. The method of claim 8 , wherein the one or more sensors further include a sensor capturing a gesture of the user indicating alteration to a portion of the three-dimensional model, and the sensed input includes the gesture captured by the sensor,
wherein receiving and processing the sensed input includes the receiving and processing the gesture captured by the sensor, and forming and communicating the input includes forming and communicating another input to cause the same or another one or more electronic resources to carry out the alteration indicated by the gesture, and
wherein receiving an output includes receiving an output including at least the altered portion of the three-dimensional model, the output

Assignees

• Boeing Co

Inventors

• Laughlin Brian Dale
• Kelsey William David
• Blair Richard N

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title