Technologies for autonomous three-dimensional modeling

Patent metadata

Field	Value
Publication number	US-10455222-B2
Application number	US-201715473930-A
Country	US
Kind code	B2
Filing date	Mar 30, 2017
Priority date	Mar 30, 2017
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

An autonomous object modeler includes a modeling table, a controllable arm, a depth camera attached to the controllable arm, and a controller to control operation of the modeling table, the controllable arm, and the depth camera. The modeling table may be movable and includes a mass sensor to produce mass sensor data indicative of a mass of an object positioned on the modeling table. The controllable arm includes a force-torque sensor to produce force-torque sensor data indicative of an inertia of the object while the object is moved by the controllable arm. The controller is configured to control operation of the controllable arm to reposition the object on the modeling table to generate three-dimensional models of the object. The three-dimensional models include the mass data and the inertia data.

First claim

The invention claimed is:

1. An object modeler for autonomously modeling an object, the object modeler comprising:
a modeling table having a mass sensor to produce mass sensor data indicative of a mass of an object positioned on the modeling table;
a controllable arm having a controllable manipulator to grasp and move the object, wherein the controllable arm includes a force-torque sensor to produce force-torque sensor data indicative of an inertia of the object while the object is moved by the controllable arm;
a depth camera attached to the controller arm to capture depth images of the object; and
a controller to control operation of the modeling table, the controllable arm, and the depth camera.

2. The object modeler of claim 1 , wherein the controller is to generate a three-dimensional model of the object based on the depth images captured by the depth camera, wherein the three-dimensional model includes (i) mass data indicative of the mass of the object, (ii) an estimated center of mass of the object, and (iii) an inertia matrix to the three-dimensional model.

3. The object modeler of claim 1 , wherein the controller is to:
determine mass data indicative of the mass of the object based on the mass sensor data received from the mass sensor;
control the depth camera to capture a depth image of the object while the object is positioned on the modeling table;
determine whether a surface of the object is occluded from the depth camera;
control the controllable arm to reposition the object on the modeling table to a new position at which the occluded surface is viewable by the depth camera;
determine inertia data of the object based on the force-torque sensor data while the object is being repositioned by the controllable arm;
control the depth camera to capture a subsequent depth image of the object while in the new position;
generate a three-dimensional model of the object based on the captured depth images of the object; and
append the mass data and inertia data to the three-dimensional model.

4. The object modeler of claim 3 , wherein to control the controllable arm to reposition the object on the modeling table comprises to:
determine a grasp plan for the controllable arm based on a present point cloud model of the object, wherein the grasp plan defines movements of the controller arm required for the controller arm to grasp and move the object to the new position;
control the controllable manipulator to grasp the object based on the grasp plan; and
control the controllable arm to move the object to the new position based on the grasp plan.

5. The object modeler of claim 3 , wherein to determine the inertia data comprises to (i) determine an estimated center of mass of the object based on the force-torque sensor data and the movements of the object and (ii) determine an inertia matrix based on the force-torque sensor data and the movements of the object, wherein the inertia matrix defines an inertia of the object.

6. The object modeler of claim 3 , wherein to generate the three-dimensional model comprises to convert a point cloud model generated based on the depth images of the object to a three-dimensional mesh model.

7. The object modeler of claim 3 , wherein to append the mass data and inertia data comprises to append mass data indicative of the mass of the object, an estimated center of mass of the object, and an inertia matrix that defines an inertia of the object to the three-dimensional model.

8. The object modeler of claim 3 , wherein the controller is further to:
segment the subsequent depth image to remove background structures form the subsequent depth image;
align the segmented subsequent depth image with the present point cloud model; and
aggregate the aligned subsequent depth image with the point cloud model to generate an updated point cloud model.

9. The object modeler of claim 3 , wherein the controller is further to:
determine whether determination of the inertia data is completed;
control the controllable arm to move the object in response to a determination that the determination of the inertia data is not completed;
determine additional inertia data of the object while the object is being moved by the controllable arm; and
update the inertia data with the additional inertia data.

10. A method for autonomously modeling an object, the method comprising:
determining, by an object modeler, mass data indicative of a mass of the object while the object is positioned on a modeling table of the object modeler;
capturing, by a depth camera of the object modeler, a depth image of the object;
determining, by the object modeler, whether a surface of the object is occluded from the depth camera;
repositioning, by a controllable arm of the object modeler, the object on the modeling table to a new position at which the occluded surface is viewable by the depth camera;
determining, by the object modeler, inertia data of the object while the object is being repositioned by the controllable arm;
capturing, by the depth camera the object modeler, a subsequent depth image of the object while in the new position;
generating, by the object modeler, a three-dimensional model of the object based on the captured depth images of the object; and
appending, by the object modeler, the mass data and inertia data to the three-dimensional model.

11. The method of claim 10 , wherein determining the mass data comprises:
receiving, by a controller of the object modeler and from a weight sensor of the modeling table, sensor data indicative of the mass of the object while the object is positioned on the modeling table; and
determining, by the controller, the mass data based on the sensor data.

12. The method of claim 10 , wherein repositioning the object on the modeling table comprises:
determining, by a controller of the object modeler, a grasp plan for the controllable arm based on a present point cloud model of the object, wherein the grasp plan defines movements of the controller arm required for the controller arm to grasp and move the object to the new position;
grasping, by a manipulator of the controllable arm, the object based on the grasp plan; and
moving, by the controller arm, the object to the new position based on the grasp plan.

13. The method of claim 10 , wherein determining inertia data comprises: receiving, by a controller of the object modeler and from a force-torque sensor of the controllable arm, force-torque sensor data indicative of a torque of the controller arm while performing movements of the object with the controllable arm; determining, by the controller, an estimated center of mass of the object based on the force-torque sensor data and the movements of the object; and determining, by the controller, an inertia matrix based on the force-torque sensor data and the movements of the object, wherein the inertia matrix defines an inertia of the object.

14. The method of claim 10 , wherein appending the mass data and inertia data comprises appending mass data indicative of the mass of the object, an estimated center of mass of the object, and an inertia matrix that defines an inertia of the object to the three-dimensional model.

15. The method of claim 10 , further comprising updating a present point cloud model of the object based on the subsequent depth image of the object.

16. The method of claim 15 , wherein updating the present point cloud module comprises:
segmenting the subsequent depth image to remove background structures form the subsequent depth image;
aligning the segmented subsequent depth image with the present point clo

Assignees

• Intel Corp

Inventors

• Felip Leon Javier
• Gonzalez Aguirre David I
• Alvarez Ignacio J

Classifications

• B25J9/0096

  co-operating with a working support, e.g. work-table · CPC title

• G01G19/52

  Weighing apparatus combined with other objects, e.g. furniture (with walking sticks A45B3/08) · CPC title

• B25J19/023

  including video camera means · CPC title

• H04N13/207

  using a single two-dimensional [2D] image sensor · CPC title

• B25J15/103

  for gripping the object in three contact points · CPC title