Translating procedural documentation into contextual visual and auditory guidance

Patent metadata

Field	Value
Publication number	US-2017286766-A1
Application number	US-201615084035-A
Country	US
Kind code	A1
Filing date	Mar 29, 2016
Priority date	Mar 29, 2016
Publication date	Oct 5, 2017
Grant date	—

Abstract

A method and system are provided for assisting a user performing a procedure. The method includes capturing, by a camera, images of user activity while the user is performing the procedure. The method further includes converting, by computer processing system, the images of user activity into a text representation of user activity. The method also includes comparing, by the computer processing system, the textual representation of user activity to procedure documentation. The method additionally includes at least one of visually and audibly indicating, by a display and a speaker, a corrective action to the user responsive to a mismatch result from said comparing step.

First claim

1 . A method for assisting a user performing a procedure, the method comprising:
capturing, by a camera, images of user activity while the user is performing the procedure; converting, by computer processing system, the images of user activity into a text representation of user activity; comparing, by the computer processing system, the textual representation of user activity to procedure documentation; and at least one of visually and audibly indicating, by a display and a speaker, a corrective action to the user responsive to a mismatch result from said comparing step.

2 . The method of claim 1 , further comprising capturing, by the camera, text present in the procedure documentation, and wherein said comparing step compares the text representation of user activity to the text present in the procedure documentation.

3 . The method of claim 1 , further comprising:
capturing, by the camera, images of the procedure documentation; and converting, by the computer processing system, the images of the procedure documentation into a text representation of the procedure documentation; wherein said comparing step compares the text representation of user activity to the text representation of the procedure documentation.

4 . The method of claim 1 , further comprising:
capturing, by the camera, images of the procedure documentation; applying, by the computer processing system, language processing techniques to the images of the procedure documentation to identify tools and components involved in the procedure; and visually or audibly indicating, by the display or the speaker, the tools and the components involved in the procedure.

5 . The method of claim 1 , further comprising:
capturing, by the camera, images of the procedure documentation; applying, by the computer processing system, language processing techniques to the images of the procedure documentation to identify goals, steps, and actions involved in the procedure; and visually or audibly indicating, by the display or the speaker, the goals, steps, and actions involved in the procedure.

6 . The method of claim 1 , further comprising capturing, by the camera, images of the procedure documentation, and wherein said comparing step further compares the images of the procedure documentation to the images of user activity.

7 . The method of claim 1 , further comprising:
detecting a completion of a current step; and updating, on the display, displayed information to correspond to a next step.

8 . The method of claim 1 , further comprising:
identifying when the user is experiencing difficulty with a given step; retrieving, from a remote source, additional information pertaining to the procedure or the given step; and audibly or visually providing, by the speaker or the display, the additional information to the user.

9 . The method of claim 8 , further comprising searching, by the computer processing system, through the additional information to identify one or more relevant sub-portions, and wherein only the one or more relevant sub-portions are audibly or visually provided to the user while other portions of the additional information are skipped from being audibly or visually provided to the user.

10 . The method of claim 8 , wherein the additional information comprises an alternate method for performing the given step with which the user is experiencing difficulty.

11 . The method of claim 8 , wherein the additional information comprises instructional videos or video demonstrations of at least some of the procedure.

12 . The method of claim 1 , further comprising:
recognizing, using a gesture recognition system, gestures from the user activity; and evaluating, by the computer processing system, a progress of the user in performing the procedure by correlating the gestures with images from the procedure documentation or a text representation of the images from the procedure documentation.

13 . The method of claim 12 , wherein said evaluating step comprises comparing labels, generated by the gesture recognition system for classifying the user activity, to the text representation of the images from the procedure documentation.

14 . The method of claim 12 , further comprising capturing, by the camera, images of the procedure documentation, wherein said evaluating step comprises mapping the gestures to expected user actions depicted in the images of the procedure documentation.

15 . A non-transitory computer readable storage medium comprising a computer readable program for assisting a user performing a procedure, wherein the computer readable program when executed on a computer causes the computer to perform the steps of:
capturing, by a camera, images of user activity while the user is performing the procedure; converting, by computer processing system, the images of user activity into a text representation of user activity; comparing, by the computer processing system, the textual representation of user activity to procedure documentation; and visually or audibly indicating, by a display or a speaker, a corrective action to the user responsive to a mismatch result from said comparing step.

16 . A system for assisting a user performing a procedure, the method comprising:
a camera for capturing images of user activity while the user is performing the procedure; a computer processing system for converting the images of user activity into a text representation of user activity, and comparing the textual representation of user activity to procedure documentation; and a display or speaker for at least one of visually and audibly indicating a corrective action to the user responsive to a mismatch result from said comparing step.

17 . The system of claim 16 , wherein said camera and said display are disposed on a head-mounted device configured to camera images in front of the user and to display procedure related information to the user.

18 . The system of claim 16 , wherein the computer processing system is implemented as a server using a cloud computing configuration.

19 . The system of claim 16 , wherein the camera captures text present in the procedure documentation, and wherein the computer processing system compares the text representation of user activity to the text present in the procedure documentation.

20 . (canceled)

21 . The method of claim 1 , further comprising searching, by the computer processing system, through additional information pertaining to a given step with which the user is experiencing difficulty to identify one or more relevant sub-portions, and wherein only the one or more relevant sub-portions are audibly or visually provided to the user while other portions of the additional information are skipped from being audibly or visually provided to the user.

Assignees

• IBM

Inventors

• Castelli Vittorio
• Dinu Georgiana
• Florian Radu
• Kundu Gourab
• Moon Taesun
• Sil Avirup

Classifications

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/3331

  Query processing · CPC title

• G06F16/93

  Document management systems · CPC title

• G06K9/2054

  Physics · mapped topic

• G06F17/30657

  Physics · mapped topic