Control of host device using three-dimensional position and velocity

Patent metadata

Field	Value
Publication number	US-10444342-B2
Application number	US-201715452851-A
Country	US
Kind code	B2
Filing date	Mar 8, 2017
Priority date	Mar 8, 2017
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

A system and method of controlling operation of a host device in real-time, the host device operatively connected to an optical device and a radar device. The optical device is configured to obtain visual data of at least one object. The object is located at an incline, relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ). The radar device is configured to obtain radar data, including a radial distance (r) of the object from the host device, the azimuth angle (θ), and a range rate (dr/dt). The controller is programmed to determine a time-to-contact for the host device and the object based at least partially on a 3-D position and 3-D velocity vector. The operation of the host device is controlled based at least partially on the time-to-contact.

First claim

What is claimed is:

1. A system of controlling operation of a host device in real-time, the system comprising:
an optical device operatively connected to the host device and configured to obtain visual data of at least one object, the at least one object being located at an incline relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ);
a radar device operatively connected to the host device and configured to obtain radar data, including the azimuth angle (θ), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt);
a controller operatively connected to the host device and including a processor and tangible, non-transitory memory on which is recorded instructions, execution of the instructions by the processor causing the controller to:
obtain respective bounding boxes around the at least one object in the visual data, the respective bounding boxes including a first bounding box characterized by an initial position (u,v);
match the respective bounding boxes with the radar data;
determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt), the optical flow being characterized by a first displacement change (Δu) of the first bounding box in row coordinates and a second displacement change (Δv) of the first bounding box in column coordinates, both over a predefined unit of time (Δt);
determine a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt) based on the visual data and the radar data, including determining respective components of the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt) such that: ϕ=inverse tangent (u/f), θ=inverse tangent (v/f), (dθ/dt)=Δv/(f*Δt) and (dϕ/dt) =Δu/(f*Δt); and
control operation of the host device based at least partially on the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt).

2. The system of claim 1 , wherein the controller is configured to:
obtain a 3-D position vector (X, Y, Z) and a 3-D velocity vector (v x , v y , v z ) from the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt); and
determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v x , v y , v z ).

3. The system of claim 2 , wherein the controller is configured to:
communicate with a remote module such that the remote module is not physically connected to the controller;
transfer a location coordinate of the host device to the remote module;
receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and
determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.

4. The system of claim 2 , wherein the controlling operation of the host device includes:
sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold; and
wherein the first predefined threshold is greater than the second predefined threshold.

5. The system of claim 4 , wherein:
the host device includes an automatic braking module configured to decelerate the host device; and
controlling operation of the host device includes activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.

6. The system of claim 2 , wherein:
the controller includes an association module configured to perform the matching of the respective bounding boxes with the radar data.

7. The system of claim 2 , wherein:
the controller includes a Kalman filter module configured to perform the combining of the visual data and the radar data.

8. The system of claim 2 , wherein the time-to-contact (t) is defined as:
t =√{square root over (( X/v x ) 2 +( Y/v y ) 2 +( Z/v z ) 2 )}.

9. The system of claim 2 , wherein respective components of the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v x , v y , v z ) are defined as:
X=r sin ϕ cos θ;

Y=r sin θ sin ϕ;

Z=r cos ϕ;

v x =[( dr/dt )sin ϕ cos θ+ r ( dθ/dt )cos θ cos ϕ− r ( dϕ/dt )sin θ sin ϕ];

v y =[( dr/dt )sin θ sin ϕ+ r ( dθ/dt )cos θ sin ϕ+ r ( dϕ/dt )sin θ cos ϕ]; and

v z =[( dr/dt )cos ϕ− r ( dθ/dt )sin θ].

10. A method of controlling operation of a host device in real-time, the host device operatively connected to an optical device, a radar device and a controller having a processor and tangible, non-transitory memory, the method comprising:
obtaining visual data of at least one object via the optical device, the at least one object being located at an incline relative to the host device, the incline being characterized by an elevation angle (ϕ) and an azimuth angle (θ);
obtaining radar data via the radar device, including the azimuth angle (θ), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt);
obtaining respective bounding boxes around the at least one object in the visual data, the respective bounding boxes including a first bounding box characterized by an initial position (u,v), via the controller;
matching the respective bounding boxes with the radar data, via the controller;
determining an optical flow of the respective bounding boxes, via the controller, including determining a change rate of the elevation angle (dϕ/dt) and a change rate of the azimuth angle (dθ/dt), via the controller, the optical flow being characterized by a first displacement change (Δu) of the first bounding box in row coordinates and a second displacement change (Δv) of the first bounding box in column coordinates, both over a predefined unit of time (Δt);
determining a 3-D position matrix (r, θ, ϕ) and a 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt) based on the visual data and the radar data, via the controller, including determining respective components of the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt) as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (Δt) such that: ϕ=inverse tangent (u/f), θ=inverse tangent (v/f), (dθ/dt) =Δv/(f*Δt) and (dϕ/dt)=Δu/(f*Δt); and
controlling operation of the host device based at least partially on the 3-D position matrix (r, θ, ϕ) and the 3-D rate of change matrix (dr/dt, dθ/dt, dϕ/dt).

11. The method of claim 10 , wherein the controller is configured to:
obtain a 3-D position vector (X, Y, Z) and a 3-D velocity vector (v x , v y , v z ) from the 3-D position matrix (r, θ, ϕ) and the 3-D change rate matrix (dr/dt, dθ/dt, dϕ/dt); and
determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v x , v y , v z ).

12. The method of claim 11 , wherein the controller is configured to:
communicate with a remote module such that the remote module is not physically connected to the controller;
transfer a location coordinate of the host device to the remote module;
receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and
determine the time-to-contact only if the location assessment factor (LAF) is at or above a

Assignees

• Gm Global Tech Operations Llc

Inventors

• Zeng Shuqing
• Sim Jay H
• Naserian Mohammad

Classifications

• B60W2754/10

  Spatial relation or speed relative to objects · CPC title

• B60W2554/00

  Input parameters relating to objects · CPC title

• G01S2013/93185

  Controlling the brakes · CPC title

• B60W2300/125

  Heavy duty trucks · CPC title

• B60W2300/34

  Compact city vehicles, e.g., microcars or kei cars · CPC title