Recognition interfaces for computing devices

Patent metadata

Field	Value
Publication number	US-9927949-B2
Application number	US-201314137783-A
Country	US
Kind code	B2
Filing date	Dec 20, 2013
Priority date	May 9, 2013
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

A computing device can utilize a recognition mode wherein an interface utilizes graphical elements, such as virtual fireflies or other such elements, to indicate objects that are recognized or identified. As objects are recognized, fireflies perform one or more specified actions to indicate recognition. A ribbon or other user-selectable icon is displayed indicates a specific action that the device can perform with respect to the respective object. As additional objects are recognized, additional ribbons are created and older ribbons can be moved off screen and stored for subsequent retrieval or search. The fireflies disperse when the objects are no longer represented in captured sensor data, and can be animated to move towards representations of new objects as features of those objects are identified as potential object features, in order to communicate a level of recognition for a current scene or environment.

First claim

What is claimed is:

1. A computing device, comprising:
at least one processor;
a display screen;
at least one camera; and
at least one memory including instructions that, when executed by the at least one processor, cause the computing device to:
receive image data using the at least one camera;
present, on the display screen, an image corresponding to the image data;
determine, based on object recognition processing, a first representation of a first object in the image data;
determining, based on object recognition processing, a second representation of a second object in the image data;
cause, on the display screen, a first plurality of graphical elements to indicate the first representation has been determined;
cause, on the display screen a second plurality of graphical elements to indicate the second representation has been determined;
present, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and
present, on the display screen while the second plurality of graphical elements indicate the second representation has been determined, a second user-selectable icon indicating a second action that the computing device is configured to perform with respect to the second object, the second user-selectable icon being presented adjacent to the first user-selectable icon.

2. The computing device of claim 1 , wherein the instructions, when executed by the at least one processor, further cause the computing device to:
receive audio data using a microphone of the computing device; and
present, on the display screen and based on the audio data, a third user-selectable icon indicating a third action that the computing device is configured to perform.

3. The computing device of claim 1 , wherein the instructions, when executed by the at least one processor, further cause the computing device to:
present the first user-selectable icon to appear to be stacked on top of the second user-selectable icon on the display screen.

4. The computing device of claim 1 , wherein the instructions, when executed by the at least one processor, further cause the computing device to:
store information for a previously-displayed user-selectable icon; and
present, on the display screen and based on a user performing a swiping motion on the display screen, the previously-displayed user-selectable icon.

5. The computing device of claim 1 , wherein the instructions, when executed by the at least one processor, further cause the computing device to:
provide a search option enabling a search to be executed with respect to information stored for any currently displayed user-selectable icon and any previously-displayed user-selectable icon.

6. A computer-implemented method, comprising:
receiving sensor data using at least one sensor of a computing device;
determining, based on object recognition processing, a first representation of a first object in the sensor data;
determining, based on object recognition processing, a second representation of a second object in the sensor data;
causing, on a display screen of the computing device, a first plurality of graphical elements to indicate that the first representation has been determined;
causing, on the display screen, a second plurality of the graphical elements to indicate that the second representation has been determined;
presenting, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and
presenting, on the display screen while the second plurality of graphical elements indicate the second representation has been determined, a second user-selectable icon indicating a second action that the computing device is configured to perform with respect to the second object, the second user-selectable icon being presented adjacent to the first user-selectable icon on the display screen.

7. The computer-implemented method of claim 6 , further comprising:
presenting the first user-selectable icon on the display screen for at least a period of time after the first representation is no longer presented on the display screen.

8. The computer-implemented method of claim 6 , wherein causing the first plurality of graphical elements to indicate the first representation has been determined comprises:
causing the first plurality of graphical elements to form a bounding box around the first a representation on the display screen.

9. The computer-implemented method of claim 6 , further comprising:
removing at least one user-selectable icon from the display screen in response to an additional user-selectable icon causing a number of user-selectable icons presented on the display screen to exceed a threshold number of user-selectable icons.

10. The computer-implemented method of claim 9 , further comprising:
storing information for any user-selectable icon no longer presented on the display screen, wherein the information is accessible using at least one of a scroll action, a touch-screen-based pull action, or a search request.

11. The computer-implemented method of claim 6 , wherein presenting the first user-selectable icon is performed using information previously stored associated with the first object.

12. The computer-implemented method of claim 6 , wherein the first user-selectable icon and the second user-selectable icon are presented as at least one of stacked or positioned in a sequence.

13. The computer-implemented method of claim 6 , wherein the first object is associated with the first action and a third action, and wherein the first action is conveyed by the first user-selectable icon based at least in part upon historical data associated with a user.

14. The computer-implemented method of claim 6 , wherein the first user-selectable icon is presented at a location on the display screen based upon at least one of a first location corresponding to the first representation and a second location corresponding to the second representation, a first type corresponding to the first object, historical user data, a date of last access, or times at which the first user-selectable icon and the second user-selectable icon were generated.

15. A computing device, comprising:
at least one processor;
a display screen;
at least one sensor; and
at least one memory including instructions that, when executed by the at least one processor, cause the computing device to:
receive sensor data using the at least one sensor;
determine, based on object recognition processing, a first representation of a first object in the sensor data;
determine, based on object recognition processing, a second representation of a second object in the sensor data;
cause, on the display screen, a first plurality of graphical elements to indicate the first representation has been determined;
cause, on the display screen, a second plurality of the graphical elements to indicate the second representation has been determined;
present, on the display screen while the first plurality of graphical elements indicate the first representation has been determined, a first user-selectable icon indicating a first action that the computing device is configured to perform with respect to the first object; and
present, on the display screen while the second plurality of graphical elements indicate th

Assignees

• Amazon Tech Inc

Inventors

• Gray Timothy Thomas
• Salazar Marc Anthony
• Sommer Steven Michael
• Cummins Charles Eugene
• Rooney Sean Anthony
• Agnetta Bryan Todd
• Park Jae Pum
• Mains Richard Leigh
• Marashi Suzan

Classifications

• G06F3/0485 Primary

  Scrolling or panning · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06V10/235

  based on user input or interaction · CPC title

• G06F3/04817 Primary

  using icons (graphical or visual programming using iconic symbols G06F8/34) · CPC title

• G06K9/2081

  Physics · mapped topic