Terminal device, application server, receiving method, and transmitting method

Patent metadata

Field	Value
Publication number	US-11836876-B2
Application number	US-202017764983-A
Country	US
Kind code	B2
Filing date	Dec 16, 2020
Priority date	Dec 23, 2019
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A terminal device ( 10 ) includes a transceiver ( 110 ), a camera ( 140 ), a display ( 130 ), and a processor ( 150 ). The processor ( 150 ) determines a first synchronization signal whose radio quality satisfies a predetermined threshold from a plurality of synchronization signals beamformed and transmitted from the base station ( 20 ), and reports the first synchronization signal to the base station ( 20 ). The processor ( 150 ) superimposes a virtual object corrected by using correction information on a captured image of the camera ( 140 ) and displays the image on the display ( 130 ). The correction information is information for indicating a position of an area covered by the first synchronization signal with respect to a real object. The correction information includes information regarding a direction of the virtual object to be displayed on the display ( 130 ) in the area and a distance from the real object to the area.

First claim

The invention claimed is:

1. A terminal device comprising:
a transceiver;
a camera for imaging a real object;
a display for displaying an augmented reality image in which a virtual object is superimposed on the real object imaged by the camera; and
a processor,
wherein the processor is configured to
receive, via the transceiver, at least one of a plurality of synchronization signals beamformed in directions different from each other and transmitted from a base station,
determine a first synchronization signal whose radio quality satisfies a predetermined threshold from the at least one of the received synchronization signals,
report the first synchronization signal to the base station, and
receive information regarding the augmented reality image from an application server after the reporting to the base station,
wherein the information regarding the augmented reality image is
correction information to be used for displaying the augmented reality image, or
augmented reality image data in which the virtual object is aligned with respect to the real object based on the correction information,
in a case where the information regarding the augmented reality image is the correction information, the processor aligns the virtual object with respect to the real object by using the correction information, generates the augmented reality image, and outputs the augmented reality image to the display,
in a case where the information regarding the augmented reality image is the augmented reality image data in which the virtual object is aligned with respect to the real object based on the correction information, the processor outputs the augmented reality image to the display based on the received augmented reality image data, and
the correction information
is information for indicating a position of an area, covered by the beamformed and transmitted first synchronization signal, with respect to the real object, and
includes information regarding a direction of the virtual object to be displayed on the display in the area and a distance from the real object to the area.

2. The terminal device according to claim 1 ,
wherein the reporting of the first synchronization signal to the base station is
performed by transmitting a random access preamble using a random access occasion corresponding to the first synchronization signal, and
the information regarding the augmented reality image is received from the application server after a random access processing procedure including the transmission of the random access preamble is completed.

3. The terminal device according to claim 1 , wherein the correction information is associated with an index of the beamformed and transmitted first synchronization signal.

4. The terminal device according to claim 1 ,
wherein the virtual object is a point cloud object,
the point cloud object includes a plurality of partial point cloud objects, and
the plurality of partial point cloud objects have definition according to a view of a user.

5. The terminal device according to claim 4 ,
wherein the plurality of partial point cloud objects include a first partial point cloud object that is viewable by the user, and
an octree depth of the first partial point cloud object is set to have higher definition than the other partial point cloud objects.

6. An application server comprising:
a network interface; and
a processor that generates an augmented reality image in which a virtual object is superimposed on a real object imaged by a camera mounted on a terminal device,
wherein the processor is configured to
acquire, via a base station, information of a first synchronization signal determined by the terminal device from at least one of a plurality of synchronization signals beamformed in directions different from each other and transmitted from the base station, and
transmit information regarding the augmented reality image to be displayed on a display mounted on the terminal device to the terminal device via the base station,
the information regarding the augmented reality image is
correction information to be used for displaying the augmented reality image associated in advance with the first synchronization signal, or
augmented reality image data in which the virtual object is aligned with respect to the real object based on the correction information,
in a case where the information regarding the augmented reality image is the correction information, the augmented reality image data and the correction information are transmitted to the terminal device to cause the terminal device to align the virtual object with respect to the real object by using the correction information,
in a case where the information regarding the augmented reality image is the augmented reality image data, the processor aligns the virtual object with respect to the real object by using the correction information, generates the augmented reality image, and transmits the augmented reality image to the terminal device, and
the correction information
is information for indicating a position of an area, covered by the beamformed and transmitted first synchronization signal, with respect to the real object, and
includes information regarding a direction of the virtual object to be displayed on the display in the area and a distance from the real object to the area.

7. The application server according to claim 6 , wherein the correction information is associated with an index of the beamformed and transmitted first synchronization signal.

8. The application server according to claim 6 ,
wherein the virtual object is a point cloud object,
the point cloud object includes a plurality of partial point cloud objects, and
the plurality of partial point cloud objects have definition according to a view of a user.

9. The application server according to claim 8 ,
wherein the plurality of partial point cloud objects include a first partial point cloud object that is viewable by the user, and
an octree depth of the first partial point cloud object is set to have higher definition than the other partial point cloud objects.

10. A terminal device control method for displaying an augmented reality image on a terminal device including
a transceiver,
a camera for imaging a real object,
a display for displaying the augmented reality image in which a virtual object is superimposed on the real object imaged by the camera, and
a processor, the terminal device control method comprising:
receiving, via the transceiver, at least one of a plurality of synchronization signals beamformed in directions different from each other and transmitted from a base station;
determining a first synchronization signal whose radio quality satisfies a predetermined threshold from the at least one of the received synchronization signals;
reporting the first synchronization signal to the base station; and
receiving information regarding the augmented reality image from an application server after the reporting to the base station,
wherein the information regarding the augmented reality image is
correction information to be used for displaying the augmented reality image, or
augmented reality image data in which the virtual object is aligned with respect to the real object based on the correction information,
in a case where the information regarding the augmented reality image is the correction information, the virtual object is aligned with respect to the real object by using the correction information, the augmented reality image is generated, and the augmented reality image is output to the display,
in a case where the information regarding the augmented reality image is the augmented reality imag

Assignees

• Sony Group Corp

Inventors

• Otsuki Hiroto
• Sasaki Daisuke

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04L67/131 Primary

  Protocols for games, networked simulations or virtual reality · CPC title

• H04W4/029

  Location-based management or tracking services · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• H04N21/6587

  Control parameters, e.g. trick play commands, viewpoint selection · CPC title