Visuotactile operators for proximity sensing and contact control

Patent metadata

Field	Value
Publication number	US-12515352-B2
Application number	US-202318103825-A
Country	US
Kind code	B2
Filing date	Jan 31, 2023
Priority date	Feb 24, 2022
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

A method for identifying and manipulating objects may include obtaining, from an image sensor, image sensor data; identifying, using the image sensor data, a location of an object; controlling a robotic element, which includes the image sensor, to move towards the location of the object; determining a slippage based on contact between the image sensor and the object; and controlling a movement of the robotic element based on the determined slippage.

First claim

What is claimed is:

1 . A method for identifying and manipulating objects, the method comprising:
obtaining, from an image sensor, image sensor data; identifying, using the image sensor data, a location of an object; controlling a robotic element, which includes the image sensor, to move towards the location of the object; controlling a speed of the robotic element based on a time to contact the object, based on the time to contact the object being within a desired tolerance, grasping the object and modulating the image sensor to switch to tactile sensing, determining a slippage based on contact between the image sensor and the object; and controlling a movement of the robotic element based on the determined slippage, wherein the determining the slippage comprises determining a marker flow and determining an object flow, and determining a pixel-wise slip vector field as a difference between the object flow and the marker flow, wherein the identifying the location of the object comprises identifying a bounding box for the object in an image that is represented by the image sensor data, wherein the time to contact the object is determined based on an area of the bounding box.

2 . The method of claim 1 , wherein the identifying the bounding box comprises predicting coordinates of the bounding box in the image.

3 . The method of claim 1 , wherein the identifying the bounding box comprises identifying a centroid of the bounding box and determining a distance between the centroid and a center of the image sensor.

4 . The method of claim 1 , wherein the determining the slippage comprises measuring a deformation of a surface of the image sensor when the image sensor is in contact with the object.

5 . The method of claim 1 , wherein the determining the marker flow comprises identifying movement of at least one marker, and wherein determining the object flow comprises determining a motion of the object in relation to the image sensor.

6 . The method of claim 1 , further comprising combining the marker flow and the object flow using a convolutional neural network architecture.

7 . An electronic device for performing image authentication, the electronic device comprising:
at least memory storing instructions; and at least one processor configured to execute the instructions to:
obtain, from an image sensor, image sensor data;
identify, using the image sensor data, a location of an object;
control a robotic element, which includes the image sensor, to move towards the location of the object;
control a speed of the robotic element based on a time to contact the object;
based on the time to contact the object being within a desired tolerance, grasp the object and modulate the image sensor to switch to tactile sensing;
determine a slippage based on contact between the image sensor and the object; and
control a movement of the robotic element based on the determined slippage,
wherein the at least one processor is further configured to determine a marker flow and determine an object flow, and determine a pixel-wise slip vector field as a difference between the object flow and the marker flow; and identify a bounding box for the object in an image that is represented by the image sensor data, wherein the time to contact the object is determined based on an area of the bounding box.

8 . The electronic device of claim 7 , wherein the at least one processor is further configured to predict coordinates of the bounding box in the image.

9 . The electronic device of claim 7 , wherein the at least one processor is further configured to identify a centroid of the bounding box and determine a distance between the centroid and a center of the image sensor.

10 . The electronic device of claim 7 , wherein the at least one processor is further configured to measure a deformation of a surface of the image sensor when the image sensor is in contact with the object.

11 . The electronic device of claim 7 , wherein the at least one processor is further configured to identify movement of at least one marker, and wherein determining the object flow comprises determining a motion of the object in relation to the image sensor.

12 . The electronic device of claim 7 , wherein the at least one processor is further configured to combine the marker flow and the object flow using a convolutional neural network architecture.

13 . A non-transitory computer readable storage medium that stores instructions to be executed by at least one processor to perform a method for identifying and manipulating objects, the method comprising:
obtaining, from an image sensor, image sensor data; identifying, using the image sensor data, a location of an object; controlling a robotic element, which includes the image sensor, to move towards the location of the object; controlling a speed of the robotic element based on a time to contact the object; based on the time to contact the object being within a desired tolerance, grasping the object and modulating the image sensor to switch to tactile sensing; determining a slippage based on contact between the image sensor and the object; and controlling a movement of the robotic element based on the determined slippage, wherein the determining the slippage comprises determining a marker flow and determining an object flow, and determining a pixel-wise slip vector field as a difference between the object flow and the marker flow, wherein the identifying the location of the object comprises identifying a bounding box for the object in an image that is represented by the image sensor data, and wherein the time to contact the object is determined based on an area of the bounding box.

14 . The non-transitory computer readable storage medium of claim 13 , wherein the identifying the bounding box comprises predicting coordinates of the bounding box in the image.

15 . The non-transitory computer readable storage medium of claim 13 , wherein the identifying the bounding box comprises identifying a centroid of the bounding box and determining a distance between the centroid and a center of the image sensor.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Hogan Francois
• Tremblay Jean-Francois
• Baghi Bobak Hamed
• Jenkin Michael
• Siddiqi Kaleem
• Dudek Gregory Lewis

Classifications

• B25J19/023

  including video camera means · CPC title

• G05B2219/40567

  Purpose, workpiece slip sensing · CPC title

• G05B2219/39507

  Control of slip motion · CPC title

• G05B2219/40625

  Tactile sensor · CPC title

• G05B2219/40575

  Camera combined with tactile sensors, for 3-D · CPC title