Multisensory data fusion system and method for autonomous robotic operation

Patent metadata

Field	Value
Publication number	US-10493629-B2
Application number	US-201715399313-A
Country	US
Kind code	B2
Filing date	Jan 5, 2017
Priority date	May 27, 2016
Publication date	Dec 3, 2019
Grant date	Dec 3, 2019

Abstract

A robotic system includes one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a brake lever of a vehicle, a manipulator arm configured to grasp the brake lever of the vehicle, and a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of the brake lever of the vehicle. The controller is configured to control the manipulator arm to move toward, grasp, and actuate the brake lever of the vehicle based on the one or more of the location or the pose of the brake lever.

First claim

What is claimed is:

1. A robotic system comprising:
one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a vehicle; and
a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of a component of the vehicle;
wherein the controller is configured to identify potential objects from the 2D image data and the 3D image data, combine the potential objects, and examine the potential objects that are combined using simultaneous localization and mapping to identify the one or more of the location or the pose of the component of the vehicle.

2. The system of claim 1 , wherein the controller is configured to fuse the 2D image data with the 3D image data to identify the one or more of the location or the pose of the component of the vehicle.

3. The system of claim 1 , wherein the controller is configured to determine confidence parameters for the potential objects based on spatial consistency and temporal consistency of the potential objects.

4. The system of claim 1 , wherein the one or more optical sensors include at least a first camera configured to obtain and provide the 2D image data to the controller and at least a different, second camera configured to obtain and provide the 3D image data to the controller.

5. The system of claim 4 , wherein the at least a first camera includes a red-green-blue (RGB) camera and the at least a second camera includes one or more of a time of flight camera or a structured light sensor.

6. The system of claim 1 , wherein the one or more optical sensors and the controller are disposed onboard a robotic vehicle that moves the one or more optical sensors and the controller relative to the vehicle.

7. The system of claim 1 , further comprising a robotic arm, wherein the controller is configured to control the robotic arm to move toward, grasp, and actuate the component of the vehicle based on the one or more of the location or the pose of the component of the vehicle as identified from the 2D image data and the 3D image data.

8. The system of claim 1 , wherein the component of the vehicle includes a brake lever.

9. The system of claim 1 , wherein the controller is configured to compare the 2D image data with the 3D image data by identifying one or more shapes in the 2D image data, superimposing the 3D image data onto at least the one or more shapes in the 2D image data, extract potential objects from the 3D image data based on the 3D image data superimposed on the 2D image data, and select at least one of the potential objects as representative of the component of the vehicle.

10. The system of claim 1 , wherein the 3D image data is a point cloud.

11. A robotic system comprising:
one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a brake lever of a vehicle;
a manipulator arm configured to grasp the brake lever of the vehicle; and
a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of the brake lever of the vehicle, wherein the controller is configured to control the manipulator arm to move toward, grasp, and actuate the brake lever of the vehicle based on the one or more of the location or the pose of the brake lever.

12. The system of claim 11 , wherein the one or more optical sensors include at least a first camera configured to obtain and provide the 2D image data to the controller and at least a different, second camera configured to obtain and provide the 3D image data to the controller.

13. The system of claim 12 , wherein the at least a first camera includes a red-green-blue (RGB) camera and the at least a second camera includes one or more of a time of flight camera or a structured light sensor.

14. The system of claim 11 , wherein the one or more optical sensors and the controller are disposed onboard a robotic vehicle that moves the one or more optical sensors and the controller relative to the vehicle.

15. The system of claim 11 , wherein the controller is configured to compare the 2D image data with the 3D image data by identifying one or more shapes in the 2D image data, superimposing the 3D image data onto at least the one or more shapes in the 2D image data, extract potential objects from the 3D image data based on the 3D image data superimposed on the 2D image data, and select at least one of the potential objects as representative of the brake lever of the vehicle.

16. The system of claim 11 , wherein the 3D image data is a point cloud.

17. A method comprising:
obtaining two dimensional (2D) image data of a vehicle;
separately obtaining three dimensional (3D) image data of the vehicle;
comparing the 2D image data with the 3D image data;
identifying potential objects from the 2D image data and the 3D image data;
combining the potential objects, and examining the potential objects that are combined using simultaneous localization and mapping to identify the one or more of the location or the pose of the component of the vehicle; and
automatically controlling a robotic system to grasp and actuate the component of the vehicle to change a state of the vehicle based on the one or more of the location or the pose that is determined.

18. The method of claim 17 , wherein obtaining the 2D image data is performed by at least a first camera and obtaining the 3D image data is performed by at least a different, second camera.

19. The method of claim 17 , wherein the 2D image data is obtained from a red-green-blue (RGB) camera and the 3D image data is obtained from one or more of a time of flight camera or a structured light sensor.

Assignees

• Gen Electric
• Ge Global Sourcing Llc

Inventors

• Jain Arpit
• Theurer Charles Burton
• Kannan Balajee
• Sen Shiraj
• Sharma Pramod
• Li Shuai
• Liu Shubao

Classifications

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06T7/579

  from motion · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• Y10S901/47

  Optical · CPC title