Apparatus and methods for real time estimation of differential motion in live video

Patent metadata

Field	Value
Publication number	US-10728570-B2
Application number	US-201816210370-A
Country	US
Kind code	B2
Filing date	Dec 5, 2018
Priority date	May 22, 2014
Publication date	Jul 28, 2020
Grant date	Jul 28, 2020

Abstract

A data processing apparatus may use a video encoder in order to extract motion information from streaming video in real time. Output of the video encoder may be parsed in order to extract motion information associated with one or more objects within the video stream. Motion information may be utilized by e.g., an adaptive controller in order to detect one or more objects salient to a given task. The controller may be configured to determine a control signal associated with the given task. The control signal determination may be configured based on a characteristic of an object detected using motion information extracted from the encoded output. The control signal may be provided to a robotic device causing the device to execute the task. The use of dedicated hardware video encoder output may reduce energy consumption associated with execution of the task and/or extend autonomy of the robotic device.

First claim

What is claimed is:

1. A system for causing a robot to carry out a task responsive to motion of an object, comprising:
a memory having computer readable instructions stored thereon; and
at least one processor configured to execute the computer readable instructions to,
receive a first video frame comprising a first motion information associated with a first macroblock within the first video frame while the robot is in motion as the robot navigates from a first location to a second location;
receive a second video frame comprising a second motion information associated with a second macroblock within the second video frame while the robot is in motion as the robot navigates from the first location to the second location;
calculate a prevailing motion vector based on the first motion information and the second motion information;
associate the first macroblock and the second macroblock with the object;
detect a salient motion based at least on the first motion information and the second motion information and a residual motion vector which breaches a threshold, the residual motion vector comprises removal of the prevailing motion vector from the first and second motion information;
associate the salient motion with the motion of the object; and
send task instructions to the robot, the task instructions generated based on the salient motion associated with the object in order to cause the robot to carry out the task.

2. The system of claim 1 , wherein the detection of the salient motion further includes determining a spatio-temporal distribution of motion information in the first motion information and the second motion information.

3. The system of claim 2 , wherein the detection of the salient motion further includes detecting a change in motion over time from the spatio-temporal distribution of motion information.

4. The system of claim 1 , wherein the detection of the salient motion further includes detecting a pattern of alternating motion vectors in the first motion information and the second motion information.

5. The system of claim 1 , wherein the sending of the task instructions to the robot causes the robot to carry out the task, the task is at least one of: stop moving; turn around; orient to a particular direction; vocalize; or start moving.

6. The system of claim 1 , wherein the first video frame includes at least one channel carrying the first motion information, the first video frame encoded using a video encoding technique.

7. The system of claim 1 , wherein the detection of the salient motion is done after the first motion information is parsed from the first video frame and after the second motion information is parsed from the second video frame.

8. A method for causing a robot to carry out a task responsive to motion of an object, comprising:
receiving a first video frame comprising a first motion information associated with a first macroblock within the first video frame while the robot is in motion as the robot navigates from a first location to a second location;
receiving a second video fame comprising a second motion information associated with a second macroblock within the second video frame while the robot is in motion as the robot navigates from the first location to the second location;
calculating a prevailing motion vector based on the first motion information and the second motion information, respectively;
associating the first macroblock and the second macroblock with the object;
detecting a salient motion based at least on the first motion information and the second motion information and a residual motion vector which breaches a threshold, the residual motion vector comprises removal of the prevailing motion vector from the first and second motion information;
associating the salient motion with the motion of the object; and
sending task instructions to the robot, the task instructions generated based on the salient motion associated with the object in order to cause the robot to carry out the task.

9. The method of claim 8 , wherein the detecting of the salient motion further includes determining a spatio-temporal distribution of motion information in the first motion information and the second motion information.

10. The method of claim 9 , wherein the detecting of the salient motion further includes detecting a change in motion over time from the spatio-temporal distribution of motion information.

11. The method of claim 8 , wherein the detecting of the salient motion further includes detecting a pattern of alternating motion vectors in the first motion information and the second motion information.

12. The method of claim 8 , wherein the sending of the task instructions to the robot causes the robot to carry out the task, the task is at least one of: stop moving; turn around; orient to a particular direction; vocalize; or start moving.

13. The method of claim 8 , wherein the first video frame includes at least one channel for carrying the first motion information, the first video frame encoded using a video encoding technique.

14. The method of claim 8 , wherein the detecting of the salient motion is done after the first motion information is parsed from the first video frame and after the second motion information is parsed from the second video frame.

15. A non-transitory computer readable medium having computer readable instructions stored thereon that when executed by at least one processor configure the at least one processor to,
receive a first video frame comprising a first motion information associated with a first macroblock within the first video frame while a robot is in motion as the robot navigates from a first location to a second location;
receive a second video frame comprising a second motion information associated with a second macroblock within the second video frame while the robot is in motion as the robot navigates from the first location to the second location;
calculate a prevailing motion vector based on the first motion information and the second motion information;
associate the first macroblock and the second macroblock with the object;
detect a salient motion based at least on the first motion information and the second motion information and a residual motion vector which breaches a threshold, the residual motion vector comprises removal of the prevailing motion vector from the first and second motion information;
associate the salient motion with the motion of the object; and
send task instructions to the robot, the task instructions generated based on the salient motion associated with the object in order to cause the robot to carry out the task.

16. The non-transitory computer readable medium of claim 15 , wherein the detection of the salient motion further includes determining a spatio-temporal distribution of motion information in the first motion information and the second motion information.

17. The non-transitory computer readable medium of claim 16 , wherein the detection of the salient motion further includes detecting a change in motion over time from the spatio-temporal distribution of motion information.

18. The non-transitory computer readable medium of claim 15 , wherein the detection of the salient motion further includes detecting a pattern of alternating motion vectors in the first motion information and the second motion information.

Assignees

• Brain Corp

Inventors

• Richert Micah

Classifications

• H04N19/50 Primary

  using predictive coding (H04N19/61 takes precedence) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N19/51

  Motion estimation or motion compensation · CPC title

• G06T7/215

  Motion-based segmentation · CPC title

• H04N19/52

  by predictive encoding · CPC title