System and method for toy recognition

Patent metadata

Field	Value
Publication number	US-11794110-B2
Application number	US-202117196598-A
Country	US
Kind code	B2
Filing date	Mar 9, 2021
Priority date	Nov 10, 2014
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

System and method for automatic computer aided optical recognition of toys, for example, construction toy elements, detection of those elements on digital images and associating the elements with existing information is presented. The method and system may recognize toy elements of various sizes invariant of toy element distance from the image acquiring device for example camera, invariant of rotation of the toy element, invariant of angle of the camera, invariant of background, invariant of illumination and without the need of predefined region where a toy element should be placed. The system and method may detect more than one toy element on the image and identify them. The system is configured to learn to recognize and detect any number of various toy elements by training a deep convolutional neural network.

First claim

What is claimed is:

1. A recognition system for recognizing real-world toy objects from one or more images, the recognition system comprising an image capturing device and one or more processors, the recognition system configured to:
capture at least one image of a real-world toy object;
create one or more processed versions of the captured image;
use a classification model to predict a respective matching object identifier from each of a plurality of input images, the plurality of input images chosen from the captured image and the one or more processed images; and
compute an aggregated predicted object identifier from the predicted matching object identifiers.

2. The recognition system according to claim 1 , wherein the recognition system is further configured to:
detect a position of a toy object within the captured image;
create a plurality of different cropped images from the captured image, each cropped image comprising the detected position;
feed the plurality of cropped images through a classification model to predict a corresponding plurality of matching object identifiers; and
compute an aggregated predicted object identifier from the plurality of matching object identifiers.

3. The recognition system according to claim 1 , further comprising a model creation system having one or more processors and a training database configured to store annotated digital images, each annotated digital image depicting a real-world toy object and being annotated with an object identifier identifying the depicted real-world toy object, wherein:
the model creation system is configured to train a convolutional classification model based on at least a subset of the annotated digital images to predict a matching object identifier when the convolutional classification model is presented with a digital image of a real-world toy object; and
the recognition system is configured to use the trained convolutional classification model to predict the matching object identifier from the captured image.

4. The recognition system according to claim 3 , wherein:
the model creation system is further configured to process at least a first digital image so as to create one or more processed versions of the first digital image, and
training the convolutional classification model is based on one or more processed versions of the first digital image, each processed version being annotated with an object identifier associated with the first digital image.

5. The recognition system according to claim 4 , wherein the model creation system is configured to:
identify an object portion and a background portion of the first digital image, the object portion depicting the real-world toy object; and
create one or more processed images by replacing the identified background portion with one or more other background portions.

6. The recognition system according to claim 3 , wherein the model creation system is further configured to:
receive image data indicative of digital images depicting real-world toy objects;
receive image annotations indicative of object identifiers associated with respective ones of the depicted real-world toy objects; and
store annotated digital images in the training database, each annotated digital image depicting a real-world object and being annotated with an object identifier identifying the depicted real-world toy object.

7. The recognition system according to claim 3 , wherein:
the model creation system further comprises an image capturing device and a support member for receiving a real-world toy object, the support member and the image capturing device movably arranged so as to vary a viewpoint of the image capturing device relative to the support member, and
the model creation system is configured to capture multiple images from respective viewpoints of a real-world toy object positioned on the support member.

8. The recognition system according to claim 3 , wherein:
the convolutional classification model comprises a convolutional stage configured to extract one or more features from a digital image, and a classification stage configured to predict an object identifier based on the extracted one or more features,
the model creation system is configured to store a plurality of reference representations of respective outputs of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model when presented with respective ones of the annotated digital images, each stored reference representation being associated with the object identifier associated with the corresponding annotated digital image and with one or more corresponding reference attributes, and
the recognition system is configured to estimate the one or more additional attributes of the real-world toy object depicted in the captured image by comparing an output of the convolutional stage of the trained convolutional classification model produced by the trained convolutional classification model based on the captured image with one or more of the stored reference representations associated with the predicted object identifier.

9. The recognition system according to claim 1 , wherein the recognition system is further configured to:
detect one or more real-world toy objects in the captured image and to identify respective object locations within the captured image, each object location corresponding to one of the detected real-world toy objects; and
use the classification model to predict a respective object identifier for each of the detected real-world toy objects.

10. The recognition system according to claim 1 , wherein the real-world toy object includes one or more toy construction elements of a toy construction system, each toy construction element comprising coupling members for detachably connecting the toy construction element with other toy construction elements of the toy construction system.

11. The recognition system according to claim 1 , wherein the recognition system is further configured to obtain, based on the predicted object identifier, further object information about the depicted real-world toy object.

12. The recognition system according to claim 11 , wherein the further object information includes connectivity information indicative of how the real-world toy object can be detachably connected to toy construction elements of a toy construction system.

13. The recognition system according to claim 1 , wherein the classification model includes a convolutional neural network.

14. The recognition system according to claim 13 , wherein the convolutional neural network is a deep convolutional neural network.

15. The recognition system according to claim 14 , wherein the deep convolutional neural network comprises one or more convolutional layers, one or more rectification layers, one or more normalization layers and one or more pooling layers of artificial neurons.

16. The recognition system according to claim 14 , wherein
the deep convolutional neural network comprises one or more fully connected layers of artificial neurons, and
each artificial neuron of a fully connected layer receives respective inputs from at least a majority of artificial neurons of a preceding layer of the deep convolutional neural network.

17. The recognition system according to claim 1 , wherein the classification model is configured to output, when presented with a digital image, a prediction indicative of respective likelihoods that said digital image depicts respective real-world toy objects included in a training database.

Assignees

• Lego As

Inventors

• Velic Marko
• Noe Karsten Østergaard
• Mosegaard Jesper
• Christensen Brian Bunch
• Rimestad Jens

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• A63F13/65 Primary

  automatically by game devices or servers from real world data, e.g. measurement in live racing competition · CPC title

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title

• A63H33/08

  provided with complementary holes, grooves, or protuberances, e.g. dovetails · CPC title