Dynamic image compensation for pre-touch localization on a reflective surface

Patent metadata

Field	Value
Publication number	US-2017147142-A1
Application number	US-201514947011-A
Country	US
Kind code	A1
Filing date	Nov 20, 2015
Priority date	Nov 20, 2015
Publication date	May 25, 2017
Grant date	—

Abstract

An aspect of the disclosure includes a method, a system and a computer program product for removing dynamic images when determining the position of an object relative to a reflective surface. The method includes acquiring a first image of an object and the reflective surface with a camera, the camera positioned adjacent the reflective surface. It is determined when a dynamic element is displayed on the reflective surface. A second image is generated by subtracting the dynamic element from the first image. The object is identified in one of the first image and second image. A reflection of the object on the reflective surface is identified in the second image. Three-dimensional coordinates of at least a portion of the object are determined based at least in part on the identifying of the object and the identifying of the reflection of the object.

First claim

1 . A method comprising:
acquiring a first image of an object and a reflective surface with a camera, the camera positioned on a front face of a housing adjacent the reflective surface and a mirror, the mirror positioned offset from the front face and oriented to receive light reflected off of at least a portion of the reflective surface and directly from a space adjacent the reflective surface; determining a dynamic element is displayed on the reflective surface; generating a second image by subtracting the dynamic element from the first image; identifying the object in one of the first image and second image, wherein the identifying the object in the first image comprises:
determining a convex hull enclosing a contour of the object in the first image, wherein the convex hull is a polygon representing a shape; and
comparing the convex hull to a predetermined threshold to determine an object type of the object in the first image;
identifying in the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surfaces comprises:
defining a region of interest;
analyzing a shape within the region of interest;
determining the shape to have a shape similar to the object in the first image; and
determining three-dimensional coordinates of at least a portion of the object based at least in part on the identifying of the object and the identifying of the reflection of the object.

2 . The method of claim 1 wherein the determining the dynamic element is displayed includes determining a time when the first image was acquired and determining a third image of the dynamic element displayed on the reflective surface at the time.

3 . The method of claim 2 further comprising generating a third image of the dynamic element at the same time the first image is acquired.

4 . The method of claim 1 wherein the subtracting the dynamic element from the first image includes subtracting a first plurality of pixel values of the dynamic element from a second plurality of pixel values of the first image.

5 . The method of claim 1 further comprising defining a region in the first image based at least in part on a position of the object in the first image, the region being a portion of the first image.

6 . The method of claim 5 wherein the subtracting the dynamic element from the first image is performed in the region.

7 . The method of claim 1 further comprising:
applying a brightness and color distortion compensation to the first image; and
applying a blur filter to the first image.

8 . A system comprising:
a housing; a reflective surface on a front face of the housing; a camera positioned on the front face adjacent the reflective surface; a mirror positioned offset from the front face and adjacent the reflective surface and the camera, the mirror oriented to receive light reflected off of at least a portion of the reflective surface and directly from a space adjacent the reflective surface, the mirror being removably coupled to the housing; a memory having computer readable instructions; and one or more processors for executing the computer readable instructions, the one or more processors being coupled for communication to the camera, the computer readable instructions comprising: acquiring a first image of an object and the reflective surface with the camera; determining a dynamic element is displayed on the reflective surface; generating a second image by subtracting the dynamic element from the first image; identifying the object in one of the first image and second image, wherein the identifying the object in the first image comprises:
determining a convex hull enclosing a contour of the object in the first image, wherein the convex hull is a polygon representing a shape; and
comparing the convex hull to a predetermined threshold to determine an object type of the object in the first image;
identifying in the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surfaces comprises:
defining a region of interest;
analyzing a shape within the region of interest;
determining the shape to have a shape similar to the object in the first image; and
determining three-dimensional coordinates of at least a portion of the object based at least in part on the identifying of the object and the identifying of the reflection of the object.

9 . The system of claim 8 wherein the determining the dynamic element is displayed includes determining a time when the first image was acquired and determining a third image of the dynamic element displayed on the reflective surface at the time.

10 . The system of claim 9 wherein the computer readable instructions further comprise generating a third image of the dynamic element at the same time the first image is acquired.

11 . The system of claim 8 wherein the subtracting the dynamic element from the first image includes subtracting a first plurality of pixel values of the dynamic element from a second plurality of pixel values of the first image.

12 . The system of claim 8 wherein the computer readable instructions further comprise defining a region in the first image based at least in part on a position of the object in the first image, the region being a portion of the first image.

13 . The system of claim 12 wherein the subtracting the dynamic element from the first image is performed in the region.

14 . The system of claim 8 wherein the computer readable instructions further comprise applying a brightness and color distortion compensation to the first image; and applying a blur filter to the first image.

15 . A computer program product for determining a location of an object relative to a reflective surface, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform:
acquiring a first image of an object and a reflective surface with a camera, the camera positioned on a front face of a housing adjacent the reflective surface and a mirror, the mirror positioned offset from the front face and oriented to receive light reflected off of at least a portion of the reflective surface and directly from a space adjacent the reflective surface; determining a dynamic element is displayed on the reflective surface; generating a second image by subtracting the dynamic element from the first image; identifying the object in one of the first image and second image, wherein the identifying the object in the first image comprises:
determining a convex hull enclosing a contour of the object in the first image, wherein the convex hull is a polygon representing a shape; and
comparing the convex hull to a predetermined threshold to determine an object type of the object in the first image;
identifying in the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surfaces comprises:
defining a region of interest;
analyzing a shape within the region of interest;
determining the shape to have a shape similar to the object in the first image; and
determining three-dimensional coordinates of at least a portion of the object based at least in part on the identifying of the object and the identifying of the reflection of the object.

16 . The computer program product of claim 15 wherein the determining the dynamic element is displaye

Assignees

• IBM

Inventors

• GU Yu
• Hwang Inseok
• Liu Su
• Rozner Eric J
• Wei Yaoguang
• Yoo Chungkuk

Classifications

• G06F1/169

  the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes (G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033) · CPC title

• G06F1/1686

  the I/O peripheral being an integrated camera · CPC title

• G06F1/1626

  with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs] · CPC title

• G06F1/1643

  the display being associated to a digitizer, e.g. laptops that can be used as penpads (details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615) · CPC title

• G06F3/0488 Primary

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title