System for autonomous mobile device assisted communication

Patent metadata

Field	Value
Publication number	US-11368497-B1
Application number	US-201816134643-A
Country	US
Kind code	B1
Filing date	Sep 18, 2018
Priority date	Sep 18, 2018
Publication date	Jun 21, 2022
Grant date	Jun 21, 2022

Abstract

An autonomous mobile device (AMD) may be used in an environment as a communication endpoint for voice or video communications. An incoming request for communication may initiate a process in which the AMD finds a user within the environment. Information obtained from sensors onboard the AMD or in the environment may be used to determine the whereabouts of the user. If an existing communication endpoint is not available to the user or cannot support a requested communication modality, the AMD may travel to permitted areas within the environment to find the user, while avoiding areas designated as private. Once found, communication may be established with the user. If the incoming request expires, the AMD may present information indicative of the request to the user.

First claim

What is claimed is:

1. A method comprising:
receiving a communication request from a caller device to establish communication with a first user using a first communication modality, wherein the first user is associated with a first user account;
determining a first device that is associated with the first user account;
determining the first device is present in an environment associated with the first user account;
determining a first location of the first device within the environment;
sending a first instruction to search at least a portion of the environment with the first device to find the first user;
determining the first user is using a second device that is at a second location;
receiving, from the second device, data indicative of an acceptance of the communication request by the first user;
establishing, with the second device, the communication with the caller device using a second communication modality, wherein the second communication modality is different than the first communication modality;
based on the second device establishing the communication with the caller device, sending a second instruction to discontinue the search of the at least portion of the environment;
based on the second device establishing the communication with the caller device, sending a third instruction to move the first device to the second location;
determining the first device has found the first user at the second location within the environment; and
responsive to the determining the first device has found the first user:
discontinuing the communication using the second communication modality; and
establishing the communication using the first communication modality.

2. The method of claim 1 , the determining the first location of the first device comprising:
sending a request to a system for location data, wherein the system comprises one or more of a process executing on at least one processor of the first device or on at least one processor of a computing device in communication with the first device; and
receiving the location data, wherein the location data is indicative of one or more of geographic coordinates, coordinates with respect to a designated point, coordinates with respect to an occupancy map, or a room name.

3. The method of claim 1 , the determining the first user is present at the second location further comprising:
determining that the first user is depicted in one or more images obtained by the first device at the second location.

4. The method of claim 1 , wherein the first instruction further comprises one or more of:
an instruction to move the first device along a previously defined search route through the environment until the first user is found,
an instruction to request input from a second user as to a location of the first user and then to move the first device to that location, or
an instruction to receive first data from the caller device that is indicative of an expected location of the first user and then to move the first device to the expected location.

5. The method of claim 1 , further comprising:
receiving, from the first device, audio data of an utterance by the first user; and
analyzing, using an audio system, the audio data to determine first data indicative of intent of the utterance to establish the communication with the caller device using the first communication modality.

6. The method of claim 1 , further comprising:
presenting, using the first device at the second location, output indicative of the communication request to establish the communication, using the first communication modality, between the first device and the caller device.

7. The method of claim 1 , wherein the discontinuing the communication using the second communication modality and establishing the communication using the first communication modality is responsive to:
presenting, using an output device of the first device, a prompt;
receiving input data from an input device of the first device; and
determining, responsive to the input data, the first user has approved transition to the first communication modality.

8. The method of claim 1 , further comprising:
determining the caller device is associated with the first user account as being approved for automatic acceptance of requests to establish communication;
generating obscured video data in which video data produced by one or more cameras of the first device has been processed with a blurring algorithm; and
establishing the communication using the first communication modality between the first device and the caller device, wherein the first communication modality comprises a video call using the obscured video data.

9. The method of claim 1 , further comprising:
receiving data that is indicative of the first device performing a task for a second user from a first time until a second time; and
wherein the searching is delayed until after the second time.

10. A system comprising:
at least one processor;
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
receive a request from a first device to establish communication with a first user that is associated with a first user account, using a first communication modality;
determine a second device is associated with the first user account;
determine the second device is present in an environment associated with the first user account;
determine a first location of the second device within the environment;
receive, from the second device, data indicative of an acceptance of the request by the first user;
establish, with the second device, the communication between the first device and the second device using a second communication modality, wherein the second communication modality is different than the first communication modality;
receive first data obtained from other devices within the environment;
determine, based on the first data, that the first user is at the first location;
determine a route through the environment based at least in part on the determination that the first user is at the first location;
generate instructions to move a third device along the route through the environment to the first location;
determine the third device has found the first user at the first location; and
responsive to the determination that the third device has found the first user:
discontinue the communication using the second communication modality between the first device and the second device; and
establish the communication using the first communication modality from the second device to the third device.

11. The system of claim 10 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to one or more of:
determine that the first user is beyond a first threshold distance from the third device,
determine that a second user is beyond a second threshold distance from the third device, or
determine the first user is using another device that is at the first location.

12. The system of claim 10 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to one or more of:
request input from a second user as to a location of the first user and then move the third device to the location, or
receive data from the first device that is indicative of an expected location and move the third device to the expected location.

13. The system of claim 10 , wherein the at least one memory further comprises instructions that, when executed by the at least on

Assignees

• Amazon Tech Inc
• Amazon Technolgies Inc

Inventors

• Sathyanarayana Rao Shambhavi
• Hardie Tony Roy
• Santos Anna Chen

Classifications

• H04W4/33

  for indoor environments, e.g. buildings · CPC title

• H04W4/02

  Services making use of location information · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/088

  Word spotting · CPC title

• G10L25/78

  Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) · CPC title