Modular hierarchical vision system of an autonomous personal companion

Patent metadata

Field	Value
Publication number	US-11869237-B2
Application number	US-201715721637-A
Country	US
Kind code	B2
Filing date	Sep 29, 2017
Priority date	Sep 29, 2017
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

An autonomous personal companion utilizing a method of object identification that relies on a hierarchy of object classifiers for categorizing one or more objects in a scene. The classifier hierarchy is composed of a set of root classifiers trained to recognize objects based on separate generic classes. Each root acts as the parent of a tree of child nodes, where each child node contains a more specific variant of its parent object classifier. The method covers walking the tree in order to classify an object based on more and more specific object features. The system is further comprised of an algorithm designed to minimize the number of object comparisons while allowing the system to concurrently categorize multiple objects in a scene.

First claim

What is claimed is:

1. A method of identification, comprising:
capturing an image of a scene using an autonomous personal companion providing services to a user, wherein the scene is taken from a physical environment including the user;
identifying at the autonomous personal companion an object in the image of the scene;
determining a contextualization of the physical environment and the user based on a current time period;
filtering a plurality of generic classifiers of a classifier hierarchy independent of the object that is identified based on the contextualization of the physical environment and the user to obtain a group of generic classifiers that is active for the current time period, wherein each of the group of generic classifiers that is active has been searched for purposes of identifying one or more objects for the user;
determining a first plurality of probabilities that the object matches generic classifiers in the group of generic classifiers;
selecting at the autonomous personal companion a first generic classifier from the group of generic classifiers defining broad categories of objects using object data determined for the object obtained from the image, the first generic classifier selected as being representative of the object and having a highest probability of the first plurality of probabilities that meets a threshold probability, the each of the group of generic classifiers defining a corresponding parent node of a corresponding hierarchical tree of classifiers; and
walking a first hierarchical tree of classifiers of the first generic classifier by matching classifiers at one or more levels in the first hierarchical tree of classifiers to the object data until reaching an end classifier at a deepest level to identify an object class for the object, wherein at each level of the first hierarchical tree of classifiers including one or more sub-classifiers for the each level determining a second plurality of probabilities that the object matches the one or more sub-classifiers for the each level and selecting a sub-classifier for the each level having a highest probability of the second plurality of probabilities that meets the threshold probability.

2. The method of claim 1 , wherein the selecting at the autonomous personal companion the first generic classifier includes:
generating the first plurality of probabilities by executing the group of generic classifiers, wherein each of the first plurality of probabilities defines how closely the object data matches a corresponding generic classifier; and
matching the object data to the first generic classifier, wherein the first generic classifier generates the highest probability in the first plurality of probabilities.

3. The method of claim 2 , wherein the determining the first plurality of probabilities includes:
generating a first subset of probabilities by executing classifiers in an active list of generic classifiers including classifiers having recently identified object classes, the first generic classifier being in the active list; and
matching the object data to the first generic classifier generating the highest probability in the first subset of probabilities.

4. The method of claim 1 , wherein the selecting at the autonomous personal companion the first generic classifier includes:
generating the first plurality of probabilities generated by executing the group of generic classifiers, wherein each of the first plurality of probabilities defines how closely the object data matches a corresponding generic classifier; and
for each generic classifier in the group of generic classifiers generating probabilities exceeding a margin, walking a second corresponding hierarchical tree of classifiers by matching classifiers at one or more levels in the second corresponding hierarchical tree of classifiers to the object data, the classifiers that are matched generating probabilities exceeding the margin, wherein the end classifier of the first hierarchical tree of classifiers of the first generic classifier is at the deepest level of all corresponding trees of classifiers.

5. The method of claim 1 , wherein the walking the first hierarchical tree of classifiers of the first generic classifier includes:
walking the first hierarchical tree of classifiers until reaching the end classifier at the deepest level to identify the object class, the first hierarchical tree of classifiers including one or more hierarchical levels of classifiers under a parent node such that succeeding lower levels include more specific classifiers trained using more specific training data, each classifier in the first hierarchical tree of classifiers comprising a corresponding set of weights based on corresponding training data, wherein the walking the first hierarchical tree of classifiers until reaching the end classifier at the deepest level includes,
beginning with a next highest level directly below the first generic classifier as the parent node, determining at least one probability generated by executing one or more classifiers of the next highest level using the object data;
matching the object data to a matched classifier of the next highest level generating the highest probability;

determining if an adjacent lower level is connected to the matched classifier;
labeling the adjacent lower level as the next highest level; and
recursively performing until there is no adjacent lower level, wherein a last occurring matched classifier is the end classifier.

6. The method of claim 1 , further comprising:
capturing the image of the scene using an image capturing system of the autonomous personal companion; and
moving the autonomous personal companion closer to the object to better capture the object in the image.

7. The method of claim 6 , further comprising:
identifying a target area of the image, wherein the target area includes the object; and
centering the target area to a center of the image when performing the capturing the image.

8. The method of claim 1 , further comprising:
modifying the first hierarchical tree of classifiers by removing an existing classifier or adding a new classifier.

9. A non-transitory computer-readable medium storing a computer program for implementing a method of identification, the computer-readable medium comprising:
program instructions for capturing an image of a scene using an autonomous personal companion providing services to a user, wherein the scene is taken from a physical environment including the user;
program instructions for identifying at the autonomous personal companion an object in the image of the scene;
program instructions for determining a contextualization of the physical environment and the user based on a current time period;
program instructions for filtering a plurality of generic classifiers of the classifier hierarchy independent of the object that is identified based on the contextualization of the physical environment and the user to obtain a group of generic classifiers that is active for the current time period, wherein each of the group of generic classifiers that is active has been searched for purposes of identifying one or more objects for the user;
program instructions for determining a first plurality of probabilities that the object matches generic classifiers in the group of generic classifiers;
program instructions for selecting at the autonomous personal companion a first generic classifier from the group of generic classifiers defining broad categories of objects using object data determined for the object obtained from the image, the first generic classifier selected as being representative of the object and having a highest probability of the first plurality of probabilities that meets

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Bashkirov Sergey
• Taylor Michael
• Fernandez-Rico Javier

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title