Determining real-time timestamps for robotic sensor data

Patent metadata

Field	Value
Publication number	US-10296602-B1
Application number	US-201715490711-A
Country	US
Kind code	B1
Filing date	Apr 18, 2017
Priority date	Apr 18, 2017
Publication date	May 21, 2019
Grant date	May 21, 2019

Abstract

Methods, apparatus, systems, and computer-readable media for assigning a real-time clock domain timestamp to sensor frames from a sensor component that operates in a non-real-time time-domain. In some implementations, a real-time component receives capture instances that each indicate capturing of a corresponding sensor data frame by the sensor component. In response to a capture output instance, the real-time component or an additional real-time component assigns a real-time timestamp to the capture output instance, where the real-time timestamp is based on the real-time clock domain. Separately, a non-real-time component receives the corresponding sensor data frames captured by the sensor component, along with corresponding metadata. For each sensor data frame, it is determined whether there is a real-time timestamp that corresponds to the data frame and, if so, the real-time timestamp is assigned to the sensor data frame.

First claim

What is claimed is:

1. A method, comprising:
receiving a capture output generated by a vision component of a robot, the vision component including one or more vision sensors, and operating in a vision component clock domain,
wherein receiving the capture output is by a real-time component of the robot via a connection to an output of the vision component, the real-time component operating in a real-time clock domain, and
wherein the capture output is generated by the vision component based on capturing of an image frame, by the vision component, based on output of at least one of the vision sensors;

in response to receiving the capture output:
generating a real-time timestamp that corresponds to receiving the capture output and that is based on the real-time clock domain,
the generating being by the real-time component or an additional real-time component operating in the real-time clock domain;

receiving, from the vision component, the image frame and corresponding metadata generated by the vision component,
wherein receiving the image frame is by a non-real-time component via an additional connection to an additional output of the vision component;

determining that the image frame corresponds to the capture output based on the corresponding metadata for the image frame; and
in response to determining that the image frame corresponds to the capture output:
assigning the real-time timestamp to the image frame.

2. The method of claim 1 ,
wherein the corresponding metadata generated by the vision component comprises a vision component generated frame value; and
wherein determining that the image frame corresponds to the capture output based on the corresponding metadata for the image frame comprises:
determining that the image frame corresponds to the capture output based on the vision component generated frame value.

3. The method of claim 2 , wherein determining that the image frame corresponds to the capture output based on the vision component generated frame value comprises:
determining that the image frame corresponds to the capture output based on comparison of the vision component generated frame value to a previous vision component generated frame value, the previous vision component generated frame value corresponding to a previous capture output that precedes the capture output.

4. The method of claim 2 , further comprising, in response to receiving the capture output:
generating a real-time frame value that corresponds to the capture output;
wherein determining that the image frame corresponds to the capture output based on the vision component generated frame value comprises:
determining that the image frame corresponds to the capture output based on both the real-time frame value and the vision component generated frame value.

5. The method of claim 1 ,
wherein the corresponding metadata generated by the vision component comprises a vision component generated timestamp that is based on the vision component clock domain; and
wherein determining that the image frame corresponds to the capture output based on the corresponding metadata for the image frame comprises:
determining that the image frame corresponds to the capture output based on the vision component generated timestamp.

6. The method of claim 5 , wherein determining that the image frame corresponds to the capture output based on the vision component generated timestamp comprises:
comparing the vision component generated timestamp to a previous vision component generated timestamp, the previous vision component generated timestamp corresponding to a previous capture output that precedes the capture output.

7. The method of claim 1 , further comprising, in response to receiving the capture output:
transmitting, by the real-time component, an indication of the capture output over a real-time network synchronized with the real-time clock domain;
receiving, by the additional real-time component, the transmitted indication of the capture output;
wherein generating the real-time timestamp is by the additional real-time component and is based on a system tick of the real-time clock domain in which the transmitted indication of the capture output was received by the additional real-time component.

8. The method of claim 7 , further comprising:
transmitting, by the real-time component, the real-time timestamp to the non-real-time component;
wherein determining that the image frame corresponds to the capture output and assigning the real-time timestamp to the image frame are by the non-real-time component.

9. The method of claim 1 , wherein generating the real-time timestamp is by the real-time component and is based on a time, in the real-time clock domain, at which the capture output is received by the real-time component.

10. The method of claim 1 , wherein the image frame is a stereo image frame, and wherein the capture output is generated by the vision component based on capture of the image frame and an additional image frame that is a monocular image frame.

11. The method of claim 10 , further comprising:
receiving, from the vision component, the additional image frame and corresponding additional metadata generated by the vision component,
wherein receiving the image frame is by the non-real-time component via the additional connection to the additional output of the vision component;
determining that the additional image frame corresponds to the capture output based on the additional corresponding metadata for the additional image frame; and
in response to determining that the additional image frame corresponds to the capture output:
assigning the real-time timestamp to the additional image frame.

12. The method of claim 1 , wherein the additional connection is a universal serial bus connection.

13. The method of claim 12 , wherein the capture output is a voltage output that satisfies a threshold.

14. The method of claim 1 , further comprising:
generating control commands for one or more actuators of the robot based on the real-time timestamp assigned to the image frame.

15. A method, comprising:
transmitting a capture trigger to a vision component of a robot, the vision component of the robot operating in a vision component clock domain,
wherein transmitting the capture trigger is by a real-time component of the robot and causes a capture of an image frame by the vision component, the real-time time component operating in a real-time clock domain;
in response to transmitting the capture trigger:
generating a real-time timestamp that corresponds to the capture trigger and that is based on the real-time clock domain,
the generating being by the real-time component or an additional real-time component operating in the real-time clock domain;

receiving, from the vision component, the image frame and corresponding metadata generated by the vision component,
wherein receiving the image frame is via an additional connection to an additional output of the vision component;

determining that the image frame corresponds to the capture output based on the corresponding metadata for the image frame; and
in response to determining that the image frame corresponds to the capture output:
assigning the real-time timestamp to the image frame.

16. A method, comprising:
transmitting a sync command to a vision component of a robot, the vision component including one or more vision sensors, and operating in a vision component clock domain,
wherein the sync command causes the vision component to utilize an adjusted frame rate for an image frame generated by the vision comp

Assignees

• X Dev Llc

Inventors

• Cooper Emily
• Guilbert Matthieu
• Buschmann Thomas
• Deephanphongs David
• Gong Yuanzheng

Classifications

• H04N5/76

  Television signal recording · CPC title

• H04N5/91

  Television signal processing therefor · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title