System and method for toy recognition

Patent metadata

Field	Value
Publication number	US-10213692-B2
Application number	US-201515524944-A
Country	US
Kind code	B2
Filing date	Nov 9, 2015
Priority date	Nov 10, 2014
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

System and method for automatic computer aided optical recognition of toys, for example, construction toy elements, recognition of those elements on digital images and associating the elements with existing information is presented. The method and system may recognize toy elements of various sizes invariant of toy element distance from the image acquiring device for example camera, invariant of rotation of the toy element, invariant of angle of the camera, invariant of background, invariant of illumination and without the need of predefined region where a toy element should be placed. The system and method may detect more than one toy element on the image and identify them. The system is configured to learn to recognize and detect any number of various toy elements by training a deep convolutional neural network.

First claim

The invention claimed is:

1. A system for recognizing real-world toy objects from one or more images, the system comprising a model creation system and a recognition system:
wherein the model creation system comprises one or more processors and a training database and wherein the training database is configured to store annotated digital images, each annotated digital image depicting a real-world toy object and being annotated with an object identifier identifying the depicted real-world toy object;
wherein the model creation system is configured to:
train a convolutional classification model based on at least a subset of the annotated digital images to predict a matching object identifier when the convolutional classification model is presented with a digital image of a real-world toy object, wherein the recognition system comprises an image capturing device and one or more processors and wherein the recognition system is configured to:
capture at least one image of a real-world toy object;
use the trained convolutional classification model to predict a matching object identifier from the captured image;
obtain, based on the predicted object identifier, stored information including connectivity information indicative of how the real-world toy object can be detachably connected to toy construction elements of a toy construction system; and
insert a virtual toy object corresponding to the recognized physical toy object into a virtual world as a virtual construction element attached to a virtual construction model.

2. A system according to claim 1 wherein the recognition system is further configured:
to detect one or more real-world toy objects in the captured image and to identify respective object locations within the captured image, each object location corresponding to one of the detected real-world toy objects; and
to use the trained convolutional classification model to predict a respective object identifier for each of the detected real-world toy objects.

3. A system according to claim 1 , wherein the real-world toy objects are toy construction elements of a toy construction system, each toy construction element comprising coupling members for detachably connecting the toy construction element with other toy construction elements of the toy construction system.

4. A system according to claim 1 further configured to insert the virtual toy object corresponding to the recognised physical toy object into the virtual world as a virtual object or character in a game play.

5. A system according to claim 1 , wherein the model creation system is further configured to process at least a first digital image so as to create one or more processed versions of the first digital image; and wherein training the convolutional classification model is based on one or more processed versions of the first digital image, each processed version being annotated with an object identifier associated with the first digital image.

6. A system according to claim 5 , wherein the model creation system is configured:
to identify an object portion and a background portion of the first digital image, the object portion depicting the real-world toy object; and
to create one or more processed images by replacing the identified background portion with one or more other background portions.

7. The system according to claim 1 , wherein the convolutional classification model is a convolutional neural network.

8. The system according to claim 1 , wherein the convolutional classification model is a deep convolutional classification model.

9. The system according to claim 8 , wherein the deep convolutional classification model comprises of one or more convolutional layers, one or more rectification layers, one or more normalization layers and one or more pooling layers of artificial neurons.

10. The system according to claim 8 , wherein the deep convolutional classification model is a deep convolutional neural network comprising one or more fully connected layers of artificial neurons; wherein each artificial neuron of a fully connected layer receives respective inputs from at least a majority of artificial neurons of a preceding layer of the deep convolutional neural network.

11. The system according to claim 1 , wherein the model creation system comprises one or more graphics processing units (“GPU”) configured to execute a training process for training the convolutional classification model.

12. The system according to claim 1 , wherein the trained convolutional classification model is configured to output, when presented with a digital image, a prediction indicative of respective likelihoods that said digital image depicts respective real-world toy objects included in the said training database.

13. The system according to claim 1 , wherein the recognition system is implemented by a mobile device comprising a digital camera, a display and a processor.

14. The system according to claim 1 , wherein the recognition system is further configured to:
create one or more processed versions of the captured image;
feed a plurality of images through the trained convolutional classification model to predict a corresponding plurality of matching object identifiers; wherein the plurality of images are chosen from the captured image and the one or more processed images; and
compute an aggregated predicted object identifier from the plurality of matching object identifiers.

15. The system according to claim 14 , wherein the recognition system is further configured to:
detect a position of a real-world toy object within the captured image;
create a plurality of different cropped images from the captured image, each cropped image comprising the detected position;
feed the plurality of cropped images through the trained convolutional classification model to predict a corresponding plurality of matching object identifiers; and
compute an aggregated predicted object identifier from the plurality of matching object identifiers.

16. The system according to claim 1 , wherein the model creation system is further configured to:
receive image data indicative of digital images depicting real-world toy objects;
receive image annotations indicative of object identifiers associated with respective ones of the depicted real-world toy objects; and
store annotated digital images in the training database, each annotated digital image depicting a real-world object and being annotated with an object identifier identifying the depicted real-world toy object.

17. The system according to claim 1 , wherein the model creation system comprises an image capturing device and a support member for receiving a real-world toy object; wherein at least one of the support member and the image capturing device is movably arranged so as to vary a viewpoint of the image capturing device relative to the support member; and wherein the model creation system is configured to capture multiple images from respective viewpoints of a real-world toy object positioned on the support member.

18. The system according to claim 1 , wherein the recognition system is further configured to estimate one or more additional attributes of the real-world object depicted in the captured image, in addition to an identification of the real-world object.

19. The system according to claim 18 , wherein the convolutional classification model comprises:
a convolutional stage configured to extract one or more features from a digital image, and
a classification stage configured to predict an object identifier based on the extrac

Assignees

• Lego As

Inventors

• Velic Marko
• Noe Karsten Østergaard
• Mosegaard Jesper
• Christensen Brian Bunch
• Rimestad Jens

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V20/66

  Trinkets, e.g. shirt buttons or jewellery items (recognising microscopic objects G06V20/69) · CPC title

• G06N3/02

  Neural networks · CPC title

• A63F13/65 Primary

  automatically by game devices or servers from real world data, e.g. measurement in live racing competition · CPC title