Image labeling

Patent metadata

Field	Value
Publication number	US-10733742-B2
Application number	US-201816142743-A
Country	US
Kind code	B2
Filing date	Sep 26, 2018
Priority date	Sep 26, 2018
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

A method enables object label persistence between subsequent images captured by a camera. One or more processors receive a first image, which is captured by an image sensor on a camera, and which includes a depiction of an object. The processor(s) generate a label for the object, and display the first image on a display. The processor(s) subsequently receive movement data that describes a movement of the camera after the image sensor on the camera captures the first image and before the image sensor on the camera captures a second image. The processor(s) receive the second image. The processor(s) display the second image on the display, and then detect a pixel shift between the first image and the second image as displayed on the display. The processor(s) then label the object with the label on the second image as displayed on the display.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, a first image, wherein the first image is captured by an image sensor on a camera, and wherein the first image includes a depiction of an object;
generating, by one or more processors, a label for the object;
displaying, by one or more processors, the first image on a display;
receiving, by one or more processors, movement data from a movement sensor on the camera, wherein the movement data describes a movement of the camera after the image sensor on the camera captures the first image and before the image sensor on the camera captures a second image;
receiving, by one or more processors, the second image, wherein the second image is captured by the image sensor on the camera;
displaying, by one or more processors, the second image on the display;
detecting, by one or more processors, a pixel shift between the first image and the second image as displayed on the display;
determining, by one or more processors, that the second image includes the depiction of the object from the first image based on the movement of the camera and the pixel shift; and
labeling, by one or more processors, the object with the label on the second image as displayed on the display.

2. The method of claim 1 , further comprising:
inputting, by one or more processors, the first image into a Convolutional Neural Network (CNN), wherein the CNN creates an output that identifies and labels the object in the first image;
caching, by one or more processors, the output of the CNN; and
utilizing, by one or more processors, the cached output of the CNN to label the object in the second image based on the movement of the camera and the pixel shift.

3. The method of claim 2 , further comprising:
adjusting, by one or more processors, weights in nodes in the CNN based on the movement of the camera.

4. The method of claim 2 , further comprising:
adjusting, by one or more processors, weights in nodes in the CNN based on the pixel shift.

5. The method of claim 1 , further comprising:
determining, by one or more processors, a distance between the camera and the object; and
correlating, by one or more processors, the movement of the camera with the pixel shift based on the distance between the camera and the object.

6. The method of claim 1 , wherein the movement sensor is an accelerometer.

7. The method of claim 1 , wherein the movement of the camera is from a group of movements consisting of lateral movement, rotational movement, and zooming movement.

8. A computer program product comprising a computer readable storage medium having program code embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, and wherein the program code is readable and executable by a processor to perform a method comprising:
receiving a first image, wherein the first image is captured by an image sensor on a camera, and wherein the first image includes a depiction of an object;
generating a label for the object;
displaying the first image on a display;
receiving movement data from a movement sensor on the camera, wherein the movement data describes a movement of the camera after the image sensor on the camera captures the first image and before the image sensor on the camera captures a second image;
receiving the second image, wherein the second image is captured by the image sensor on the camera;
displaying the second image on the display;
detecting a pixel shift between the first image and the second image as displayed on the display;
determining that the second image includes the depiction of the object from the first image based on the movement of the camera and the pixel shift; and
labeling the object with the label on the second image as displayed on the display.

9. The computer program product of claim 8 , wherein the method further comprises:
inputting the first image into a Convolutional Neural Network (CNN), wherein the CNN creates an output that identifies and labels the object in the first image;
caching the output of the CNN; and
utilizing the cached output of the CNN to label the object in the second image based on the movement of the camera and the pixel shift.

10. The computer program product of claim 9 , wherein the method further comprises:
adjusting weights in nodes in the CNN based on the movement of the camera.

11. The computer program product of claim 9 , wherein the method further comprises:
adjusting weights in nodes in the CNN based on the pixel shift.

12. The computer program product of claim 8 , wherein the method further comprises:
determining a distance between the camera and the object; and
correlating the movement of the camera with the pixel shift based on the distance between the camera and the object.

13. The computer program product of claim 8 , wherein the movement sensor is an accelerometer.

14. The computer program product of claim 8 , wherein the movement of the camera is from a group of movements consisting of lateral movement, rotational movement, and zooming movement.

15. A camera comprising:
an image sensor, wherein the image sensor captures a first image and a second image, and wherein the first image includes a depiction of an object;
a labeling logic that generates a label for the object;
a display logic for displaying the first image;
a movement sensor that generates movement data that describes a movement of the camera after the image sensor on the camera captures the first image and before the image sensor on the camera captures the second image;
a pixel shift detector that detects a pixel shift on the display between the first image being displayed and the second image being displayed; and
an object identifier logic for determining that the second image includes the depiction of the object from the first image based on the movement of the camera and the pixel shift, wherein the object identifier logic labels the object with the label on the second image as displayed.

16. The camera of claim 15 , further comprising:
a Convolutional Neural Network (CNN), wherein the CNN creates an output that identifies and labels the object in the first image by inputting the first image into the CNN; and
a CNN cache, wherein the CNN cache caches the output of the CNN, wherein the object identifier logic labels the object with the label on the second image as displayed on the display by using the cached output of the CNN cache.

17. The camera of claim 16 , further comprising:
node adjusting logic for adjusting weights in nodes in the CNN based on the movement of the camera and the pixel shift.

18. The camera of claim 15 , further comprising:
a distance sensor, wherein the distance sensor determines a distance between the camera and the object, and wherein the movement of the camera is correlated with the pixel shift based on the distance between the camera and the object.

19. The camera of claim 15 , wherein the movement sensor is an accelerometer.

20. The camera of claim 15 , wherein the movement of the camera is from a group of movements consisting of lateral movement, rotational movement, and zooming movement.

Assignees

• IBM

Inventors

• Rozner Eric
• Yoo Chungkuk
• Hwang Inseok

Classifications

• G06V10/17

  using hand-held instruments · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T7/251 Primary

  involving models · CPC title

• G06N3/045

  Combinations of networks · CPC title