Semantic navigation of autonomous ground vehicles

Patent metadata

Field	Value
Publication number	US-11474530-B1
Application number	US-201916541755-A
Country	US
Kind code	B1
Filing date	Aug 15, 2019
Priority date	Aug 15, 2019
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Autonomous ground vehicles capture images during operation, and process the images to recognize ground surfaces or features within their vicinity, such as by providing the images to a segmentation network trained to recognize the ground surfaces or features. Semantic maps of the ground surfaces or features are generated from the processed images. A point on a semantic map is selected, and the autonomous ground vehicle is instructed to travel to a location corresponding to the selected point. The point is selected in accordance with one or more goals, such as to maintain the autonomous ground vehicle at a selected distance from a roadway or other hazardous surface, or along a centerline of a sidewalk.

First claim

What is claimed is:

1. An autonomous ground vehicle comprising:
a frame;
a first imaging device having a first field of view;
a second imaging device having a second field of view;
a third imaging device having a third field of view;
at least one wheel;
a motor disposed within the frame, wherein the motor is configured to cause the at least one wheel to rotate at a speed within a predetermined speed range;
at least one power module for powering the motor;
a position sensor;
at least one computer processor; and
at least one memory component,
wherein the at least one computer processor is configured to execute one or more sets of computer instructions for causing the autonomous ground vehicle to execute a method comprising:
identifying, by the at least one processor, a location associated with a destination for the autonomous ground vehicle;
determining a first course for the autonomous ground vehicle to the location associated with the destination;
causing the autonomous ground vehicle to travel on a first ground surface on the first course, wherein the first ground surface is at least a portion of a sidewalk;
while the autonomous ground vehicle is traveling on the first course,
capturing at least a first image by at least one of the first imaging device, the second imaging device or the third imaging device;
providing at least the first image as an input to a segmentation network executed by the at least one computer processor, wherein the segmentation network is trained to at least assign a label to at least one pixel based at least in part on a ground surface depicted in the at least one pixel;
receiving at least one output from the segmentation network;
recognizing a plurality of ground surfaces depicted within the first image based at least in part on the at least one output, wherein the first ground surface is one of the plurality of ground surfaces;
generating, by the at least one computer processor, a semantic representation of the plurality of ground surfaces;
selecting a point on the semantic representation, wherein the point is within a region of the semantic representation corresponding to the first ground surface;
selecting a second course for travel to a location on the first ground surface associated with the selected point; and
causing the autonomous ground vehicle to travel on the second course to the location on the first ground surface associated with the selected point,
wherein the second field of view overlaps with at least the first field of view,
wherein the third field of view overlaps with at least the first field of view, and
wherein the first field of view, the second field of view and the third field of view cover at least one hundred eighty degrees of a perimeter of the autonomous ground vehicle.

2. The autonomous ground vehicle of claim 1 , wherein selecting the point on the semantic representation comprises:
determining at least one goal for the autonomous ground vehicle, wherein the goal is at least one of:
maintaining a minimum distance between the autonomous ground vehicle and a second ground surface, wherein the second ground surface is one of the plurality of ground surfaces;
maintaining the second ground surface on a selected side of the autonomous ground vehicle;
avoiding an obstacle on the first ground surface; or
traveling on approximately a centerline of the first ground surface; and

selecting the point on the semantic representation in accordance with the at least one goal.

3. The autonomous ground vehicle of claim 1 , wherein the method further comprises:
while the autonomous ground vehicle is traveling on the second course,
capturing at least a second image by the at least one of the first imaging device, the second imaging device or the third imaging device;
detecting a landmark associated with the destination within at least the second image; and
in response to detecting the landmark;
determining that the autonomous ground vehicle is within a vicinity of a location associated with the destination;
selecting a third course for travel to the location associated with the destination; and
causing the autonomous ground vehicle to travel on the third course to the location associated with the destination.

4. The autonomous ground vehicle of claim 1 , wherein the semantic representation has at least one scaled dimension greater than eight meters.

5. A method comprising:
causing an autonomous ground vehicle to travel on a first course and at a first speed;
capturing, by at least one imaging device provided aboard the autonomous ground vehicle, at least a first image with the autonomous ground vehicle traveling on the first course and at the first speed;
recognizing at least a first ground surface and a second ground surface depicted within the first image;
generating a first representation of ground surfaces within a vicinity of the autonomous ground vehicle, wherein the first representation comprises a first set of pixels corresponding to the first ground surface and a second set of pixels corresponding to the second ground surface;
selecting a point within the first set of pixels on the first representation, wherein the point corresponds to a first location on the first ground surface;
selecting at least one of a second course or a second speed for the autonomous ground vehicle based at least in part on the selected point within the first set of pixels on the first representation; and
causing the autonomous ground vehicle to travel to the first location on the first ground surface on the second course or at the second speed,
wherein the at least one imaging device comprises a first imaging device having a first field of view extending forward of the autonomous ground vehicle at least in part, a second imaging device having a second field of view, and a third imaging device having a third field of view,
wherein the second field of view overlaps the first field of view,
wherein the third field of view overlaps with the first field of view, and
wherein the first field of view, the second field of view and the third field of view cover at least one hundred eighty degrees of a perimeter of the autonomous ground vehicle.

6. The method of claim 5 , further comprising:
determining, by a position sensor provided aboard the autonomous ground vehicle, a second location of the autonomous ground vehicle;
receiving, by the autonomous ground vehicle, instructions to travel to a third location;
determining a route from the second location to the third location, wherein the route comprises at least a first path; and
selecting at least the first course based at least in part on the first path.

7. The method of claim 5 , wherein the first image is captured at a first time, and
wherein the method further comprises:
capturing, by the at least one imaging device, at least a second image at a second time, wherein the second time precedes the first time;
recognizing the first ground surface and at least one of the second ground surface or a third ground surface depicted within the second image;
generating a second representation of ground surfaces within a vicinity of the autonomous ground vehicle at the second time, wherein the second representation comprises a third set of pixels corresponding to the first ground surface and a fourth set of pixels corresponding to the at least one of the second ground surface or the third ground surface;
selecting a point within the third set of pixels on the second representation, wherein the point corresponds to a second location on the first ground surface; and
selecting at least the first course based at least in part on the selected point within the third set of pixels.

8. The method of claim 5 , wherein recog

Assignees

• Amazon Tech Inc

Inventors

• Boyraz Hakan
• Chen Sheng
• Kojic Aleksandar

Classifications

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V20/38

  Outdoor scenes · CPC title

• G05D1/0253

  extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70) · CPC title

• G05D1/0251

  extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593) · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title