Display system and method for controlling display system

Patent metadata

Field	Value
Publication number	US-11131855-B2
Application number	US-201916664983-A
Country	US
Kind code	B2
Filing date	Oct 28, 2019
Priority date	Oct 29, 2018
Publication date	Sep 28, 2021
Grant date	Sep 28, 2021

Abstract

A display system includes an HMD and a PC. The HMD displays an image on a scene in a real space in an overlapped manner. The PC includes a first control unit. The first control unit causes the HMD to display a guide image indicating a direction set based on a robot to correspond to a robot arranged in the real space. On a tool of the robot, a tool coordinate system is set based on the tool. The first control unit displays the guide image indicating the tool coordinate system in accordance with a direction of the tool. The guide image includes an X-axis image, a Y-axis image, and a Z-axis image.

First claim

What is claimed is:

1. A display system, comprising:
a robot;
a head-mounted display configured to display an image on a scene in a real space in an overlapped manner;
a first control unit configured to cause the head-mounted display to display a first guide image indicating a direction set based on the robot to correspond to the robot disposed in the real space, the first guide image indicating a movement speed or a grasping force of an arm of the robot;
an input unit configured to receive an input regarding an operation of the robot; and
a second control unit configured to cause the robot to operate corresponding to the input,
wherein on the arm of the robot, a coordinate system based on the arm of the robot is set,
the first control unit is configured to
cause the head-mounted display to display the first guide image indicating the coordinate system corresponding to a direction of the arm of the robot, and
cause the head-mounted display to display a second guide image corresponding to a prohibited region in the real space, and

when the arm of the robot is in the prohibited region, the input unit does not receive the input regarding the operation of the robot.

2. The display system according to claim 1 ,
wherein the robot includes a motion mechanism configured to at least move or rotate the arm of the robot with respect to the head-mounted display.

3. The display system according to claim 1 ,
wherein a plurality of arms of the robot and a plurality of coordinate systems each based on a corresponding one the plurality of arms of the robot are set for the robot, and
the first control unit is configured to identify a location and a direction of each of the plurality of arms of the robot, and cause the head-mounted display to display the first guide image corresponding to the position and the direction of each of the plurality of arms of the robot.

4. The display system according to claim 1 ,
wherein head-mounted the display includes an imaging unit, and
the first control unit is configured to
identify a location and a direction of the arm of the robot with respect to the head-mounted display, based on a captured image captured by the imaging unit, and
cause the head-mounted display to display the first guide image corresponding to the position and the direction of the arm of the robot.

5. The display system according to claim 4 ,
wherein the first control unit is configured to detect, from the captured image, an image marker provided at the arm of the robot to identify the location and the direction of the arm of the robot with respect to the head-mounted display.

6. The display system according to claim 5 ,
wherein a plurality of arms of the robot are set on the robot, and
the first control unit is configured to
detect, from the captured image, the image marker provided at each of the plurality of arms of the robot, and
identify the location and the direction of each of the plurality of arms of the robot with respect to the head-mounted display.

7. The display system according to claim 5 ,
wherein the first control unit is configured to
identify, based on a line-of-sight of a user wearing the head-mounted display on a head of the user, one of the image markers provided respectively at the arm of the robot, and
display the first guide image corresponding to the one image marker.

8. The display system according to claim 2 ,
wherein the first control unit is configured to determine a size of the first guide image in accordance with a distance between the head-mounted display and the arm of the robot.

9. The display system according to claim 1 , comprising:
a display device having the head-mounted display; and
a control device communicably coupled to the robot and the display device, and including the first control unit.

10. The display system according to claim 1 ,
wherein the first control unit is configured to cause the head-mounted display to update display of the first guide image corresponding to the operation of the robot.

11. The display system according to claim 1 ,
wherein the first control unit is configured to cause the head-mounted display to display a state image indicating a state of the robot to correspond to the robot installed in the real space.

12. The display system according to claim 1 ,
wherein the robot has at least a plurality of joints or a plurality of shafts.

13. A control method for a display system including a robot, a head-mounted display configured to display an image on a scene in a real space in an overlapped manner, a first control unit configured to cause the head-mounted display to display an image, an input unit configured to receive an input regarding an operation of the robot, and a second control unit configured to cause the robot to operate corresponding to the input, the method comprising:
causing, by the first control unit, the head-mounted display to display a first guide image indicating a coordinate system based on an arm of the robot disposed in the real space and corresponding to a direction of the arm of the robot, the first guide image indicating a movement speed or a grasping force of the arm of the robot;
causing, by the first control unit, the head-mounted display to display a second guide image corresponding to a prohibited region in the real space;
causing, by the second control unit, the robot to operate corresponding to the input; and
when the arm of the robot is in the prohibited region, causing the input unit not to receive the input regarding the operation of the robot.

Assignees

• Seiko Epson Corp

Inventors

• Yamazaki Akio
• Takano Masahide

Classifications

• G05B2219/40161

  Visual display of machining, operation, remote viewing · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/017 Primary

  Head mounted · CPC title