System and method for focus and context views for telepresence and robotic teleoperation

Patent metadata

Field	Value
Publication number	US-10012982-B2
Application number	US-201514966428-A
Country	US
Kind code	B2
Filing date	Dec 11, 2015
Priority date	Dec 11, 2015
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

Example implementations include a system and method for providing a focus and context view for a movable apparatus. The focus view is indicated in a central portion of the display, and a context view is warped around the focus portion of the display to provide context. In example implementations, selection of another portion to render as the focus portion causes the movable apparatus to orient itself to the focus portion as the forward direction.

First claim

What is claimed is:

1. A first apparatus configured to navigate a movable second apparatus, the first apparatus comprising:
a processor, configured to:
process each of one or more camera images from the movable second apparatus into a first region and a second region, the second region being a warped image surrounding the first region, the first region being a central region of the each of the one or more camera images, the first region and the second region generated by a continuous distortion function; and
for receipt of a selected portion of the display from an input:
process the selected portion as the first region;
determine the second region from the one or more camera images based on the selected portion; and
instruct the movable second apparatus to be oriented in a forward direction based on the selected portion.

2. The first apparatus of claim 1 , wherein the one or more camera images are provided from at least one of a fish-eye camera and a 360 degree camera.

3. The first apparatus of claim 1 , wherein the processor is configured to process the one or more images into the first region by generating a focused portion from the one or more images.

4. The first apparatus of claim 1 , wherein the processor is configured to, for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion, provide instructions to the movable second apparatus to move based on the input on receipt from the movable second apparatus that the orienting is complete.

5. The first apparatus of claim 1 , wherein the processor is configured to:
for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion:
process the each of the one or more camera images from the movable second apparatus into the first region and the second region based on the forward orientation of the movable second apparatus;
adjust the display based on the processed first region and the second region; and
instruct the movable second apparatus to move based on the input.

6. The first apparatus of claim 1 , wherein the processor is configured to:
provide, on the display, an indication of the forward orientation of the movable second apparatus;
for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion:
process the one or more camera images from the movable second apparatus into the first region and the second region according to the input to move; and
update the indication of the forward orientation based on the input to move.

7. A method for a first apparatus configured to navigate a movable second apparatus, the method comprising:
processing each of one or more camera images from the movable second apparatus into a first region and a second region, the second region being a warped image surrounding the first region, the first region being a central region of the each of the one or more camera images, the first region and the second region generated by a continuous distortion function; and
for receipt of a selected portion of the display from an input:
processing the selected portion as the first region;
determining the second region from the one or more camera images based on the selected portion; and
instructing the movable second apparatus to be oriented in a forward direction based on the selected portion.

8. The method of claim 7 , wherein the one or more camera images are provided from at least one of a fish-eye camera and a 360 degree camera.

9. The method of claim 7 , wherein the processing the one or more images into the first region comprises generating a focused portion from the one or more images.

10. The method of claim 7 , further comprising, for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion, providing instructions to the movable second apparatus to move based on the input on receipt from the movable second apparatus that the orienting is complete.

11. The method of claim 7 , further comprising:
for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion:
processing the one or more camera images from the movable second apparatus into the first region and the second region based on the forward orientation of the movable second apparatus;
adjusting the display based on the processed first region and the second region; and
instructing the movable second apparatus to move based on the input.

12. The method of claim 7 , further comprising:
providing, on the display, an indication of the forward orientation of the movable second apparatus;
for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion:
processing the one or more camera images from the movable second apparatus into the first region and the second region according to the input to move; and
updating the indication of the forward orientation based on the input to move.

13. A non-transitory computer readable medium storing instructions for a first apparatus configured to navigate a movable second apparatus, the instructions comprising:
processing each of one or more camera images from the movable second apparatus into a first region and a second region, the second region being a warped image surrounding the first region, the first region being a central region of the each of the one or more camera images, the first region and the second region generated by a continuous distortion function; and
for receipt of a selected portion of the display from an input:
processing the selected portion as the first region;
determining the second region from the one or more camera images based on the selected portion; and
instructing the movable second apparatus to be oriented in a forward direction based on the selected portion.

14. The non-transitory computer readable medium of claim 13 , wherein the one or more camera images are provided from at least one of a fish-eye camera and a 360 degree camera.

15. The non-transitory computer readable medium of claim 13 , wherein the processing the one or more images into the first region comprises generating a focused portion from the one or more images.

16. The non-transitory computer readable medium of claim 13 , the instructions further comprising, for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion, providing instructions to the movable second apparatus to move based on the input on receipt from the movable second apparatus that the orienting is complete.

17. The non-transitory computer readable medium of claim 13 , the instructions further comprising:
for an input to move the movable second apparatus while the movable second apparatus is orienting the forward direction of the movable second apparatus based on the selected portion:
processing the one or more camera images from the movable second apparatus into the first region and the second region based on the forward orientation of the movable second apparatus;
adjusting the dis

Assignees

• Fuji Xerox Co Ltd

Inventors

• Kratz Sven
• Vaughan James
• Mizutani Ryota
• Kimber Donald

Classifications

• H04N23/633

  for displaying additional information relating to control or operation of the camera · CPC title

• H04N23/661

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G05D1/0038 Primary

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title