Image processing apparatus and image processing method

Patent metadata

Field	Value
Publication number	US-10444617-B2
Application number	US-201615565968-A
Country	US
Kind code	B2
Filing date	Mar 1, 2016
Priority date	Apr 30, 2015
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

A polarized image acquisition section 20 acquires a plurality of polarized images having different polarization directions. The polarized images show, for example, an input indicator for a user interface as a recognition target object. A normal line calculation section 30 calculates normal lines for individual pixels of the recognition target object in accordance with the polarized images acquired by the polarized image acquisition section 20. The normal lines represent information based on the three-dimensional shape of the recognition target object. A recognition section 40 recognizes the object by using the normal lines calculated by the normal line calculation section 30, determines, for example, the type, position, and posture of the input indicator, and outputs the result of determination as input information on the user interface. The object can be recognized easily and with high accuracy.

First claim

The invention claimed is:

1. An image processing apparatus, comprising:
a storage device configured to store a plurality of images of a plurality of objects; and
circuitry configured to:
acquire a first plurality of polarized images of a recognition target object,
wherein a first polarized image of the first plurality of polarized images has a first polarization direction different from a second polarization direction of a second polarized image of the first plurality of polarized images;

calculate first normal lines of the recognition target object for each of a plurality of pixels in the first plurality of polarized images;
select a stored image of an object of the plurality of objects based on at least one feature of the recognition target object, wherein the plurality of images includes the stored image;
resolve, based on the selected stored image, an uncertainty of the calculated first normal lines of the recognition target object to generate uncertainty-free first normal lines; and
recognize the recognition target object based on the uncertainty-free first normal lines of the recognition target object.

2. The image processing apparatus according to claim 1 , wherein
the recognition target object is an input indicator for a user interface, and
the circuitry is further configured to output a result of the recognition of the recognition target object as input information on the user interface.

3. The image processing apparatus according to claim 1 , wherein the circuitry is further configured to:
generate a temporary recognition process image from the first plurality of polarized images;
recognize the at least one feature of the recognition target object based on the temporary recognition process image; and
resolve the uncertainty further based on a result of the recognition of the at least one feature of the recognition target object.

4. The image processing apparatus according to claim 3 , wherein the circuitry is further configured to
select the stored image of the object, based on the object being closest to the recognition target object among the plurality of objects, as the result of the recognition of the at least one feature of the recognition target object.

5. The image processing apparatus according to claim 4 , wherein
the recognition target object is a hand, and
the circuitry is further configured to:
recognize each of a position of a fingertip of the hand and a position of a finger pulp of the hand based on the temporary recognition process image and the stored plurality of images; and
resolve an uncertainty of second normal lines of a finger region of the hand to generate uncertainty-free second normal lines, wherein the uncertainty of the second normal lines is resolved based on the recognized position of the fingertip and the recognized position of the finger pulp.

6. The image processing apparatus according to claim 5 , wherein the circuitry is further configured to determine a finger pointing direction based on the generated uncertainty-free second normal lines of the finger region.

7. The image processing apparatus according to claim 4 , wherein
the recognition target object is a face, and
the circuitry is further configured to:
recognize a position of a face region based on the temporary recognition process image and the stored plurality of images; and
resolve an uncertainty of second normal lines of the face to generate uncertainty-free second normal lines, wherein the uncertainty of the second normal lines is resolved based on the recognized position of the face region.

8. The image processing apparatus according to claim 7 , wherein the circuitry is further configured to determine at least one of a shape of the face or an expression of the face based on the generated uncertainty-free second normal lines of the face region.

9. The image processing apparatus according to claim 4 , wherein
the recognition target object is a hand, and
the circuitry is further configured to:
recognize each of a position of a hand region and a skeletal structure of the hand region based on the temporary recognition process image and the stored plurality of images; and
resolve an uncertainty of second normal lines of the hand region to generate uncertainty-free second normal lines, wherein the uncertainty of the second normal lines is resolved based on the recognized position of the hand region and the skeletal structure of the hand region.

10. The image processing apparatus according to claim 9 , wherein the circuitry is further configured to determine a shape of the hand based on the generated uncertainty-free second normal lines of the hand region.

11. The image processing apparatus according to claim 1 , wherein the circuitry is further configured to:
calculate second normal lines based on a second plurality of polarized images of a learning object;
generate teacher data based on the second normal lines of the learning object,
wherein a third polarized image of the second plurality of polarized images has a third polarization direction different from a fourth polarization direction of a fourth polarized image of the second plurality of polarized images;

control the storage device to store the generated teacher data; and
recognize the recognition target object based on student data and the stored teacher data,
wherein the student data is generated based on the calculated first normal lines.

12. The image processing apparatus according to claim 11 , wherein the circuitry is further configured to
calculate the second normal lines of the learning object for each of the second plurality of polarized images.

13. The image processing apparatus according to claim 11 , wherein
the teacher data is an indication of distribution of the second normal lines of the learning object, and
the student data is an indication of distribution of the calculated first normal lines of the recognition target object.

14. The image processing apparatus according to claim 11 , wherein the circuitry is further configured to select the learning object corresponding to the teacher data, based on the student data, as a result of the recognition of the recognition target object.

15. An image processing method, comprising:
in an image processing apparatus:
acquiring a plurality of polarized images of a recognition target object,
wherein a first polarized image of the plurality of polarized images has a first polarization direction different from a second polarization direction of a second polarized image of the plurality of polarized images;

calculating normal lines of the recognition target object for each of a plurality of pixels in the plurality of polarized images;
selecting an image of an object of a plurality of objects based on at least one feature of the recognition target object, wherein the image is stored in a storage device;
resolving, based on the selected image, an uncertainty of the normal lines of the recognition target object to generate uncertainty-free normal lines; and
recognizing the recognition target object based on the uncertainty-free normal lines of the recognition target object.

16. A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a computer, cause the computer to execute operations, the operations comprising:
acquiring a plurality of polarized images of a recognition target object,
wherein a first polarized image of the plurality of polarized images has a first polarization direction different from a second polarization direction of a second polarized ima

Assignees

• Sony Corp

Inventors

• Nakatani Ayaka
• Hirasawa Yasutaka
• Kondo Yuhi
• Lu Ying

Classifications

• G01B11/00

  Measuring arrangements characterised by the use of optical techniques · CPC title

• G06T7/00

  Image analysis · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G03B35/26 Primary

  using polarised or coloured light separating different viewpoint images · CPC title

• G06V40/107 Primary

  Static hand or arm · CPC title