Sensor compensation using backpropagation

Patent metadata

Field	Value
Publication number	US-12596930-B2
Application number	US-202117358725-A
Country	US
Kind code	B2
Filing date	Jun 25, 2021
Priority date	Jun 25, 2021
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

An embodiment includes training a first convolutional neural network (CNN) using a plurality of training images to generate first and second trained CNNs, and then adding an interface layer to the second trained CNN. The embodiment processes a first and second images in a sequence of images using the first trained CNN to generate a first and second result vectors. The embodiment also processes the second image using the second trained CNN and sensor data input to the interface layer to generate a third result vector. The embodiment modifies the sensor data using a compensation value. The embodiment compares the third result vector to the second result vector to generate an error value, and then calculates a modified compensation value using the error value. The embodiment then generates a sensor-compensated trained CNN based on the second trained CNN with the modified compensation value.

First claim

What is claimed is:

1 . A processor-implemented method comprising:
training a first convolutional neural network (CNN) using a plurality of training images; generating, upon determining that the first CNN has been trained, a first trained CNN and a second trained CNN, wherein the first trained CNN and the second trained CNN are based on the first CNN; adding an interface layer to the second trained CNN; processing a first image in a sequence of successive images using the first trained CNN to generate a first result vector, wherein the first image receives a full-image analysis; processing a second image that is subsequent to the first image in the sequence of images using the first trained CNN to generate a second result vector; processing the second image using the second trained CNN and sensor data input to the interface layer to generate a third result vector, wherein the sensor data is collected after the first image is captured and prior to a capture of the second image, wherein the processing of the second image using the second trained CNN includes modifying the sensor data using a compensation value, and wherein, based on the modified sensor data, a cached first result vector from the first trained CNN is reused to label objects in subsequent images from the sequence of successive images based on sensor-detected movement and pixel shift; comparing the third result vector to the second result vector to generate an error value; calculating a modified compensation value using the error value via a backpropagation algorithm to reduce the error value; generating a sensor-compensated trained CNN based on the second trained CNN with the modified compensation value, wherein a sensor interface layer of the sensor-compensated trained CNN is updated with the modified compensation value and sensor data input to the interface layer of the sensor-compensated trained CNN is modified using the modified compensation value; and processing a third image using the sensor-compensated trained CNN and determining the third image comprises one or more objects depicted in the first image and the second image, tracked using the sensor data and pixel shift.

2 . The method of claim 1 , wherein the first image is captured by an image sensor on a camera.

3 . The method of claim 2 , further comprising generating the sensor data by a movement sensor on the camera, wherein the sensor data describes a movement of the camera after the image sensor on the camera captures the first image and before the image sensor on the camera captures a second image.

4 . The method of claim 3 , wherein the movement sensor is selected from a group comprising an accelerometer, a magnetometer, a gyroscope, a global positioning system, and a proximity sensor.

5 . The method of claim 3 , wherein the first image includes a depiction of an object, the method further comprising generating a label for the object.

6 . The method of claim 5 , further comprising displaying the first image and the label on a display.

7 . The method of claim 5 , further comprising detecting a pixel shift between the first image and the second image.

8 . The method of claim 7 , further comprising determining that the second image includes the depiction of the object from the first image based on the movement of the camera and the pixel shift.

9 . The method of claim 8 , further comprising labeling the object with the label on the second image.

10 . The method of claim 9 , further comprising:
caching the first result vector output from the first trained CNN; and utilizing the cached first result vector output from the first trained CNN to label the object in the second image based on the movement of the camera and the pixel shift.

11 . The method of claim 2 , further comprising processing the third image that is subsequent to the second image in the sequence of images using the sensor-compensated trained CNN and updated sensor data input to the interface layer to generate a fourth result vector.

12 . The method of claim 11 , further comprising generating the updated sensor data by a movement sensor on the camera, wherein the updated sensor data describes an updated movement of the camera after the image sensor on the camera captures the second image and before the image sensor on the camera captures the third image.

13 . The method of claim 12 , further comprising detecting an updated pixel shift between the second image and the third image based on the fourth result vector.

14 . The method of claim 13 , further comprising determining that the third image includes a depiction of an object from the first image and the second image based on the updated movement of the camera and the updated pixel shift.

15 . A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to perform operations comprising:
training a first convolutional neural network (CNN) using a plurality of training images; generating, upon determining that the first CNN has been trained, a first trained CNN and a second trained CNN, wherein the first trained CNN and the second trained CNN are based on the first CNN; adding an interface layer to the second trained CNN; processing a first image in a sequence of successive images using the first trained CNN to generate a first result vector, wherein the first image receives a full-image analysis; processing a second image that is subsequent to the first image in the sequence of images using the first trained CNN to generate a second result vector; processing the second image using the second trained CNN and sensor data input to the interface layer to generate a third result vector, wherein the sensor data is collected after the first image is captured and prior to a capture of the second image, wherein the processing of the second image using the second trained CNN includes modifying the sensor data using a compensation value, and wherein, based on the modified sensor data, a cached first result vector from the first trained CNN is reused to label objects in subsequent images from the sequence of successive images based on sensor-detected movement and pixel shift; comparing the third result vector to the second result vector to generate an error value; calculating a modified compensation value using the error value via a backpropagation algorithm to reduce the error value; generating a sensor-compensated trained CNN based on the second trained CNN with the modified compensation value, wherein a sensor interface layer of the sensor-compensated trained CNN is updated with the modified compensation value and sensor data input to the interface layer of the sensor-compensated trained CNN is modified using the modified compensation value; and processing a third image using the sensor-compensated trained CNN and determining the third image comprises one or more objects depicted in the first image and the second image, tracked using the sensor data and pixel shift.

16 . The computer program product of claim 15 , wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over a network from a remote data processing system.

17 . The computer program product of claim 15 , wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wh

Assignees

• Int Business Machines Corporation

Inventors

• Cho Minsik
• Hwang Inseok
• Yoo Chungkuk

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V10/7515

  Shifting the patterns to accommodate for positional errors · CPC title