Modular hierarchical vision system of an autonomous personal companion

Patent metadata

Field	Value
Publication number	US-2019102667-A1
Application number	US-201715721637-A
Country	US
Kind code	A1
Filing date	Sep 29, 2017
Priority date	Sep 29, 2017
Publication date	Apr 4, 2019
Grant date	—

Abstract

An autonomous personal companion utilizing a method of object identification that relies on a hierarchy of object classifiers for categorizing one or more objects in a scene. The classifier hierarchy is composed of a set of root classifiers trained to recognize objects based on separate generic classes. Each root acts as the parent of a tree of child nodes, where each child node contains a more specific variant of its parent object classifier. The method covers walking the tree in order to classify an object based on more and more specific object features. The system is further comprised of an algorithm designed to minimize the number of object comparisons while allowing the system to concurrently categorize multiple objects in a scene.

First claim

What is claimed is:

1 . A method of identification, comprising:
identifying an object in an image of a scene; selecting a first generic classifier from a group of generic classifiers defining broad categories of objects using object data determined for the object, the first generic classifier selected as being representative of the object, each generic classifier forming part of a corresponding hierarchical tree of classifiers with the generic classifier as a parent node of the tree; and walking a first tree of classifiers of the first generic classifier by matching classifiers at one or more levels in the first tree to the object data until reaching an end classifier at a deepest level to identify an object class for the object.

2 . The method of claim 1 , wherein the selecting a first generic classifier further comprises:
generating a plurality of probabilities by executing the group of generic classifiers, wherein each of the plurality of probabilities defines how closely the object data matches a corresponding generic classifier; and matching the object data to the first generic classifier, wherein the first generic classifier generates the highest probability in the plurality of probabilities.

3 . The method of claim 2 , wherein determining a plurality of probabilities further comprises:
generating a first subset of probabilities by executing classifiers in an active list of generic classifiers including classifiers having recently identified object classes, the first generic classifier being in the active list; and matching the object data to the first generic classifier generating the highest probability in the first subset of probabilities.

4 . The method of claim 1 , wherein the selecting a first generic classifier further comprises:
generating a plurality of probabilities generated by executing the group of generic classifiers, wherein each of the plurality of probabilities defines how closely the object data matches a corresponding generic classifier; and for each generic classifier generating probabilities exceeding a margin, walking a corresponding tree of classifiers by matching classifiers at one or more levels in the corresponding tree to the object data, the classifiers that are matched generating probabilities exceeding the margin, wherein the end classifier of the first tree of classifiers of the first generic classifier is at the deepest level of all corresponding trees of classifiers.

5 . The method of claim 1 , wherein the walking a first tree of classifiers comprises:
walking the first tree of classifiers until reaching the end classifier at the deepest level to identify the object class, the first tree including one or more hierarchical levels of classifiers under the parent node such that succeeding lower levels include more specific classifiers trained using more specific training data, each classifier in the first tree comprising a corresponding set of weights based on corresponding training data, wherein the walking includes,
beginning with a next highest level directly below the first generic classifier as the parent node, determining at least one probability generated by executing one or more classifiers of the next highest level using the object data;
matching the object data to a matched classifier of that level generating the highest probability;
determining if an adjacent lower level is connected to the matched classifier;
labeling the adjacent lower level as the next highest level; and
recursively performing until there is no adjacent lower level, wherein the matched classifier is the end classifier.

6 . The method of claim 1 , further comprising:
capturing the image of the scene using an image capturing system of an autonomous personal companion; and moving the personal companion closer to the object to better capture the image.

7 . The method of claim 6 , further comprising:
identifying a target area of the image, wherein the target area includes the object; and centering the target area in the center of the image when capturing the image.

8 . The method of claim 1 , further comprising:
modifying the first tree of classifiers by removing an existing classifier or adding a new classifier.

9 . A non-transitory computer-readable medium storing a computer program for implementing a method of identification, the computer-readable medium comprising:
program instructions for identifying an object in an image of a scene; program instructions for selecting a first generic classifier from a group of generic classifiers defining broad categories of objects using object data determined for the object, the first generic classifier selected as being representative of the object, each generic classifier forming part of a corresponding hierarchical tree of classifiers with the generic classifier as a parent node of the tree; and program instructions for walking a first tree of classifiers of the first generic classifier by matching classifiers at one or more levels in the first tree to the object data until reaching an end classifier at a deepest level to identify an object class for the object.

10 . The computer-readable medium of claim 9 , wherein the program instructions for selecting a first generic classifier further comprises:
program instructions for generating a plurality of probabilities by executing the group of generic classifiers, wherein each of the plurality of probabilities defines how closely the object data matches a corresponding generic classifier; and program instructions for matching the object data to the first generic classifier, wherein the first generic classifier generates the highest probability in the plurality of probabilities.

11 . The computer-readable medium of claim 10 , wherein the program instructions for determining a plurality of probabilities further comprises:
program instructions for generating a first subset of probabilities by executing classifiers in an active list of generic classifiers including classifiers having recently identified object classes, the first generic classifier being in the active list; and program instructions for matching the object data to the first generic classifier generating the highest probability in the first subset of probabilities.

12 . The computer-readable medium of claim 9 , wherein the program instructions for walking a first tree of classifiers comprises:
program instructions for walking the first tree of classifiers until reaching the end classifier at the deepest level to identify the object class, the first tree including one or more hierarchical levels of classifiers under the parent node such that succeeding lower levels include more specific classifiers trained using more specific training data, each classifier in the first tree comprising a corresponding set of weights based on corresponding training data, wherein the walking includes,
beginning with a next highest level directly below the first generic classifier as the parent node, program instructions for determining at least one probability generated by executing one or more classifiers of the next highest level using the object data;
program instructions for matching the object data to a matched classifier of that level generating the highest probability;
program instructions for determining if an adjacent lower level is connected to the matched classifier;
program instructions for labeling the adjacent lower level as the next highest level; and
program instructions for recursively performing until there is no adjacent lower level, wherein the matched classifier is the end classifier.

13 . The computer-readable medium of clai

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Bashkirov Sergey
• Taylor Michael
• Fernandez-Rico Javier

Classifications

• G06N3/004 Primary

  Artificial life, i.e. computing arrangements simulating life · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24323

  Tree-organised classifiers · CPC title

• G06N3/08

  Learning methods · CPC title