Visual perception assistant

Patent metadata

Field	Value
Publication number	US-10860847-B2
Application number	US-201715785644-A
Country	US
Kind code	B2
Filing date	Oct 17, 2017
Priority date	Oct 17, 2017
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

A method, a system, and a computer program product for visually identifying the at least one target subject within a real-time view of a current scene. The method includes capturing, by at least one image sensor, a real-time view of a current scene. The method further includes performing a visual analysis of the real-time view to identify at least one subject. The method further includes: receiving, in real-time, a natural language input which includes verbal utterances of at least one speaker; and identifying, within the natural language input, a description of at least one particular subject. The method further includes: analyzing the current scene to identify at least one target subject that matches the description of the at least one particular subject; and in response to identifying the at least one target subject, applying at least one visual identifier to the real-time view of the current scene.

First claim

What is claimed is:

1. A method comprising:
capturing, by at least one image sensor, a real-time view of a current scene;
receiving, a natural language input in real-time with the capturing of the real-time view;
identifying, within the natural language input, a description of at least one particular subject within the current scene;
analyzing the captured real time view of the current scene to identify, within the current scene, at least one target subject from among the at least one subject that matches the description of the at least one particular subject, the analyzing comprising differentiating among multiple identified subjects from other objects to identify the at least one target subject, based on a respective matching score determined between the description from the natural language input and each of the multiple identified subjects; and
in response to identifying the at least one target subject within the current scene: applying, to the real-time view of the current scene, at least one visual identifier that increases a visibility of the at least one target subject within the real-time view of the current scene; and providing, to at least one output device, the real-time view of the current scene including the at least one visual identifier.

2. The method of claim 1 , wherein:
the current scene includes a plurality of objects;
performing the visual analysis of the current scene further comprises, identifying:
the at least one subject from among the plurality of objects, wherein the at least one subject is a focus of attention in the current scene; and
a background of the current scene; and

the analyzing further comprises differentiating among the multiple identified subjects within the current scene, based on at least one prepositional phrase or relationship terms and phrases included in the description.

3. The method of claim 1 , further comprising:
monitoring a movement of the at least one subject within the current scene; and
dynamically updating a position of the at least one visual identifier within the real-time view based on the monitored movement.

4. The method of claim 1 , wherein applying the at least one visual identifier further comprises at least one of:
dynamically rendering an identifying object adjacent to the at least one target subject within the real-time view of the current scene; and
dynamically applying, to the real-time view of the current scene, a color adjustment to at least one region that includes the at least one target subject.

5. A data processing system comprising:
at least one image sensor that captures a real-time view of a current scene;
at least one input device that captures a natural language input including verbal utterances of at least one speaker, the natural language input captured in real-time with the capturing of the real-time view; and
at least one processor that:
identifies, within the natural language input, a description of at least one particular subject within the current scene;
analyzes the captured real time view of the current scene to identify at least one target subject within the current scene from among the at least one subject that matches the description of the at least one particular subject within the current scene, wherein to identify the at least one target subject the processor differentiates among multiple identified subjects from other objects, based on a respective matching score determined between the description from the natural language input and each of the multiple identified subjects; and
in response to identifying the at least one target subject within the current scene: applies, to the real-time view of the current scene, at least one visual identifier that increases a visibility of the at least one target subject within the real-time view of the current scene; and provides, to at least one output device, the real-time view of the current scene including the at least one visual identifier.

6. The data processing system of claim 5 , wherein:
the current scene includes a plurality of objects;
the at least one processor, in performing the visual analysis of the current scene, identifies:
the at least one subject from among the plurality of objects, wherein the at least one subject is a focus of attention in the current scene; and
a background of the current scene; and

in analyzing the captured real time view of the current scene, the at least one processor differentiates among the multiple identified subjects within the current scene, based on at least one prepositional phrase or relationship terms and phrases included in the description.

7. The data processing system of claim 5 , wherein the at least one processor:
monitors a movement of the at least one subject within the real-time view of the current scene; and
dynamically updates a position of the at least one visual identifier within the real-time view based on the monitored movement.

8. The data processing system of claim 5 , wherein in applying the at least one visual identifier, the at least one processor performs at least one of:
dynamically renders an identifying object adjacent to the at least one target subject within the real-time view of the current scene; and
dynamically applies, to the real-time view of the current scene, a color adjustment to at least one region that includes the at least one target subject.

9. A computer program product comprising:
a non-transitory computer readable storage device; and
program code on the computer readable storage device that, when executed by a processor associated with a data processing system, enables the data processing system to provide the functionality of:
capturing, by at least one image sensor, a real-time view of a current scene;
receiving, in real-time, a natural language input;
identifying, within the natural language input, a description of at least one particular subject within the current scene;
analyzing the captured real time view of the current scene to identify at least one target subject within the current scene from among the at least one subject that matches the description of the at least one particular subject, the analyzing comprising differentiating among multiple identified subjects from other objects to identify the at least one target subject, based on a respective matching score determined between the description from the natural language input and each of the multiple identified subjects; and
in response to identifying the at least one target subject within the current scene; applying, to the real-time view of the current scene, at least one visual identifier that increases a visibility of the at least one target subject within the real-time view of the current scene and providing, to at least one output device, the real-time view of the current scene including the at least one visual identifier.

10. The computer program product of claim 9 , wherein:
the current scene includes a plurality of objects;
the program code for performing the visual analysis of the current scene further comprising code for identifying:
the at least one subject from among the plurality of objects, wherein the at least one subject is a focus of attention in the current scene; and
a background of the current scene; and
the program code for analyzing the captured real time view of the current scene comprises code for differentiating among the multiple identified subjects within the current scene from other objects, based on at least one prepositional phrase or relationship terms and phrases included in the description.

11. The computer program product of claim 9 , the program code further comprising code for:
monitoring a movement of the at least on

Assignees

• Motorola Mobility Llc

Inventors

• Qian Ming
• Wang Yunming
• Wang Erin

Classifications

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V40/103 Primary

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title