Information processing device, information processing method, and program

Patent metadata

Field	Value
Publication number	US-12010288-B2
Application number	US-201917288314-A
Country	US
Kind code	B2
Filing date	Sep 30, 2019
Priority date	Nov 2, 2018
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

An information processing device including: an acquisition unit that acquires information according to a recognition result of position and posture of a viewpoint; a projection processing unit that projects a target object based on position and posture of a first viewpoint onto a first projection surface defined in association with the first viewpoint, associates second information according to relative position and posture relationship between the first viewpoint and the object with first information according to a result of the projection; a correction processing unit that restores three-dimensional information of the object based on the first and second information, and reprojects the object according to relative position and posture relationship between the restored object and a second viewpoint onto a second projection surface defined in association with the second viewpoint; and an output control unit that causes an output unit to present display information according to a result of the reprojection.

First claim

The invention claimed is:

1. An information processing device comprising:
an acquisition unit that acquires information according to a recognition result of the position and posture of a viewpoint;
a projection processing unit that projects a target object on a basis of the position and posture of a first viewpoint onto a first projection surface defined in association with the first viewpoint, and associates second information according to the relative position and posture relationship between the first viewpoint and the object with first information according to a result of the projection;
a correction processing unit that restores three-dimensional information of the object on a basis of the first information and the second information, and reprojects the object according to the relative position and posture relationship between the restored object and a second viewpoint onto a second projection surface defined in association with the second viewpoint; and
an output control unit that causes an output unit to present display information according to a result of the reprojection,
wherein the acquisition unit, the projection processing unit, the correction processing unit, and the output control unit are each implemented via at least one processor.

2. The information processing device according to claim 1 , wherein the second information includes information regarding a distance between the first viewpoint and the object.

3. The information processing device according to claim 1 , wherein
the acquisition unit acquires third information regarding movement of the object,
the projection processing unit associates the third information with the first information, and
the correction processing unit estimates, on a basis of the third information, movement of the object during a period in which the position and posture of the viewpoint change between the first viewpoint and the second viewpoint, and restores three-dimensional information of the object according to a result of the estimation.

4. The information processing device according to claim 3 , wherein the third information includes information regarding a direction in which the object moves and information regarding a speed of the object.

5. The information processing device according to claim 3 , wherein
the correction processing unit
corrects three-dimensional information of the restored object according to movement of the object based on the third information, and
reprojects the corrected object onto the second projection surface.

6. The information processing device according to claim 1 , wherein
the projection processing unit projects the object onto the first projection surface corresponding to the first viewpoint for each of a plurality of the first viewpoints having different positions and postures, and
the correction processing unit restores three-dimensional information of the object on a basis of the second information according to a projection result of the object for each of the plurality of the first viewpoints, and reprojects the restored object onto the second projection surface.

7. The information processing device according to claim 6 , wherein the correction processing unit restores three-dimensional information of the object by complementing information corresponding to a part of the object not included in the second information corresponding to some of the first viewpoints among the plurality of the first viewpoints with information corresponding to the part included in the second information corresponding to another first viewpoint.

8. The information processing device according to claim 1 , wherein
the acquisition unit acquires the first information for each part of the object,
the projection processing unit projects the object for each of the parts onto the first projection surface, and associates the first information corresponding to the part with a projection result of the part, and
the correction processing unit restores three-dimensional information of the object for each of the parts, and reprojects the restored object for each of the parts onto the second projection surface.

9. The information processing device according to claim 8 , wherein the part of the object includes at least any of one or more edges, one or more vertices, or one or more voxels included in the object.

10. The information processing device according to claim 1 , wherein
in a case where at least a part of a result of the reprojection of a first object onto the second projection surface and at least a part of a result of the reprojection of a second object different from the first object onto the second projection surface overlap,
the correction processing unit
performs control so that the display information corresponding to an object closer to the second viewpoint among the first object and the second object is preferentially displayed.

11. The information processing device according to claim 1 , wherein
the acquisition unit acquires information according to a recognition result of a body in real space;
the projection processing unit acquires, on a basis of the position and posture of the first viewpoint, fourth information according to the relative position and posture relationship between the first viewpoint and the body; and
the correction processing unit
restores three-dimensional information of the body on a basis of the fourth information, and
reprojects the restored object onto the second projection surface according to the relative position and posture relationship among the restored body, the restored object, and the second viewpoint.

12. The information processing device according to claim 1 , wherein
the projection processing unit associates identification information of the object with the first information according to a result of projection of the object onto the first projection surface, and
the correction processing unit identifies the object on a basis of the identification information associated with the first information, and restores three-dimensional information of the object.

13. The information processing device according to claim 1 , wherein for each of a first subarea and a second subarea included in a display area of the output unit, the output control unit reprojects the restored object onto the second projection surface on a basis of positions and postures of the second viewpoints at different timings.

14. The information processing device according to claim 13 , wherein
the output control unit controls, at different timings,
presentation in the first subarea of first display information according to a result of the reprojection for the first subarea, and
presentation in the second subarea of second display information according to a result of the reprojection for the second subarea.

15. The information processing device according to claim 1 , wherein the correction processing unit performs processing related to the reprojection in a shorter cycle than a cycle of processing related to the projection by the projection processing unit.

16. The information processing device according to claim 1 , wherein the projection processing unit controls the position in which to fix the object in real space, so that the object is superimposed on a body in real space according to a recognition result of the position and the posture of the viewpoint.

17. The information processing device according to claim 1 , wherein the output unit is a transmissive output unit.

18. The information processing device according to claim 17 , further comprising a suppo

Assignees

• Sony Group Corp

Inventors

• Aga Hiroyuki
• Ishihara Atsushi
• Nishibe Mitsuru
• Kawasaki Koichi
• Ichikawa Hirotake
• Wakabayashi Hajime
• Kohara Shunitsu
• Murata Ryosuke

Classifications

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/275

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title

• H04N13/344 Primary

  with head-mounted left-right displays · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N13/117

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title