Showing danger areas associated with objects using augmented-reality display techniques

Patent metadata

Field	Value
Publication number	US-9836652-B2
Application number	US-201615013278-A
Country	US
Kind code	B2
Filing date	Feb 2, 2016
Priority date	Feb 2, 2016
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A mechanism is provided for indicating an area of an object that is considered dangerous to a user via a head mounted display (HMD) system. The object being used by the user and one or more areas of the object that are considered dangerous are identified. Responsive to identifying one or more areas of the object that are considered dangerous, an overlay for each of the one or more areas of the object that are considered dangerous is generated. The overlay for each one or more areas of the object that are considered dangerous is displayed via a display layer of the HMD system so that the one or more areas of the object that are considered dangerous are identified as an overlay to the object that is being seen by the user through the HMD system.

First claim

What is claimed is:

1. A method, in a data processing system, for indicating an area of an object that is considered dangerous to a user via a head mounted display (HMD) system, the method comprising:
receiving, by the data processing system, one or more images or video captured from a camera mounted to the HMD system that is separate from the data processing system;
identifying, by the data processing system, the object being used by the user from the one or more images or video;
identifying, by the data processing system, one or more areas of the object that are considered dangerous;
responsive to identifying one or more areas of the object that are considered dangerous, thereby forming a set of dangerous areas, generating, by the data processing system, an overlay for each dangerous area of the set of dangerous areas;
displaying, by the data processing system, the overlay for each dangerous area of the set of dangerous areas that is visible to the user from the HMD system via a display layer of the HMD system so that each dangerous area of the set of dangerous areas that is visible to the user from the HMD system is identified as an overlay to the object that is visible by the user through the HMD system;
identifying, by the data processing system, a body part of the user traversing toward an identified dangerous area of the object;
calculating, by the data processing system, a distance of the identified danger area of the object with respect to the body part of the user;
based on the calculated distance, identifying, by the data processing system, a degree of warning that is to be issued to the user; and
issuing, by the data processing system, one or more appropriate actions that coincide with the degree of warning, wherein the one or more appropriate actions include at least one of a visual notification, an audio notification, or a vibration notification to the user, and wherein the one or more appropriate actions further includes an action communicated to the object that causes the object to either reverse movement of one or more parts of the object if the reversal will not cause any harm to the user or reverse movement to one or more parts of the object if the reversal will not cause any harm to the user.

2. The method of claim 1 , wherein the object that is being used by the user is identified via one or more images or video that is captured from a camera that is mounted to the HMD system, wherein the camera captures the one or more images or the video that correlates to the view of the user of the HMD system.

3. The method of claim 1 , wherein the one or more areas of the object that are considered dangerous are identified from a knowledge database that coincides with an identification of the object via one or more images or video and wherein the knowledges database is a database of information regarding objects that includes information of objects including one or more of object specifications, object behaviors, danger areas associated with objects, or social feedback regarding interactions with objects.

4. The method of claim 1 , wherein an overlay for each one or more areas of the object that are considered dangerous are larger than the actual area of the object that is considered dangerous to account for the movement of the user's head and a delay in processing one or more images or video by the data processing system.

5. The method of claim 1 , further comprising:
identifying, by the data processing system, one or more areas of the object that are considered to be warning areas;
responsive to identifying one or more areas of the object that are considered to be warning areas, generating, by the data processing system, an overlay for each of the one or more areas of the object that are considered to be warning areas; and
displaying, by the data processing system, the overlay for each one or more areas of the object that are considered to be warning areas via the display layer of the HMD system so that the one or more areas of the object that are considered to be warning areas are identified as an overlay to the object that is being seen by the user through the HMD system.

6. The method of claim 5 , wherein the overlay for each one or more areas of the object that are considered to be warning areas are displayed in a first color while the overlay for each one or more areas of the object that are considered dangerous are displayed in a second color that is different from the first color.

7. The method of claim 5 , further comprising:
identifying, by the data processing system, at least one of a movement of the object or a movement of one or more parts of the object;
either modifying, by the data processing system, one or more existing overlays or generating, by the data processing system, an additional overlay for each warning area or danger area that coincides with the movement of the object or the movement of the one or more parts of the object; and
displaying, by the data processing system, the modified or new overlay for each of the warning areas or danger areas that coincides with the movement of the object or the movement of the one or more parts of the object via the display layer of the HMD system so that each of the warning areas or danger areas that coincides with the movement of the object or the movement of the one or more parts of the object are identified as an overlay to the object that is being seen by the user through the HMD system.

8. The method of claim 1 , wherein the one or more areas of the object that are considered dangerous comprise areas that are identified as being a different level of danger and wherein the overlay for each one or more areas of the object that are considered to be a low level of danger are displayed in a first color while the overlay for each one or more areas of the object that are considered to be a high level of danger are displayed in a second color that is different from the first color.

9. The method of claim 1 , further comprising:
receiving, by the data processing system, information from the object;
determining, by the data processing system, whether the information indicates a need for a warning or danger notification;
responsive to the information indicating the need for the warning or danger notification, either modifying, by the data processing system, one or more existing overlays or generating, by the data processing system, an additional overlay for each warning or danger indicated by the information from the object; and
displaying, by the data processing system, the modified or new overlay for each warning or danger indicated by the information from the object via the display layer of the HMD system so that each of the warning or danger indicated by the information from the object are identified as an overlay to the object that is being seen by the user through the HMD system.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
receiving one or more images or video captured from a camera mounted to a head mounted display (HMD) system that is separate from the computing device;
identify an object being used by the user from the one or more images or video;
identify one or more areas of the object that are considered dangerous;
responsive to identifying one or more areas of the object that are considered dangerous, thereby forming a set of dangerous areas, generate an overlay for each dangerous area of the set of dangerous areas;
display the overlay for each dangerous area of the set of dangerous areas that is visible to the user from the HMD system

Assignees

• IBM

Inventors

• Lection David B
• Rakshit Sarbajit K
• Stevens Mark B
• Wilson John D

Classifications

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06K9/00671 Primary

  Physics · mapped topic

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title