Method and apparatus for constructing real-geographic-space scene in real time

Patent metadata

Field	Value
Publication number	US-11836878-B2
Application number	US-202318144860-A
Country	US
Kind code	B2
Filing date	May 9, 2023
Priority date	May 9, 2022
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A method for constructing a real-geographic-space scene in real time based on a panoramic-video technique is provided. By using a measuring robot and the attitude sensors, accurately determining the geographic coordinates and the attitudes of the cameras, where the cameras may be installed in a fixed or stringing manner, where in the fixing type a plurality of neighboring videos at the same moment undergo orthographic correction and splicing, and in the stringing type the cameras are installed to a guiding device and may locally, independently and quickly move and shoot, and the videos of the neighboring cameras are spliced in real time; and fusing the videos, the geographic coordinates and the environment sounds that satisfy the delay time, to form a scene video streaming.

First claim

What is claimed is:

1. A method for constructing a real-geographic-space scene in real time based on a panoramic-video technique, wherein the method for constructing the real-geographic-space scene in real time comprises:
by a high-performance computing device, receiving image data collected by a plurality of panoramic cameras, and combining the image data into an image-data sequence, wherein the plurality of panoramic cameras are controlled by using a remote-control instruction, and the plurality of panoramic cameras are installed in the real-geographic-space scene in a fixing manner or a stringing manner, and provide marking points in the real-geographic-space scene;
by the high-performance computing device, receiving a three-dimensional geographic coordinate of each of the plurality of panoramic cameras in the scene, wherein the three-dimensional geographic coordinate is, by a measuring robot, acquired by using a prism installed at each of the plurality of panoramic cameras and subsequently sent;
by the high-performance computing device, receiving external-azimuth parameters from attitude sensors, wherein each of the plurality of panoramic cameras is installed with one of the attitude sensors, and each of the attitude sensors is configured for determining in real time an azimuth angle, a pitch angle and a roll angle of the panoramic camera where the attitude sensor is located, and providing the external-azimuth parameters in photogrammetry, to facilitate subsequent orthographic geometric correction, splicing and optimization of a panoramic image;
by the high-performance computing device, receiving a scene environment sound, wherein the scene environment sound is acquired and sent by a plurality of audio collecting devices in the real-geographic-space scene;
by the high-performance computing device, according to different installation modes, the image-data sequence, the marking points, the external-azimuth parameters and the three-dimensional geographic coordinates, performing splicing-stitching and the orthographic geometric correction or distortion correction to the image-data sequence, to obtain an orthographic image or a panoramic-image sequence, wherein the different installation modes use different methods of splicing optimization, the orthographic image is obtained by splicing-stitching and orthographic geometric correction to an image sequence obtained in a fixing-type installation mode, and the panoramic-image sequence is obtained by splicing-stitching and distortion correction to an image sequence obtained in a stringing-type installation mode;
by the high-performance computing device, performing augmented-reality optimization to the synthesized orthographic image or panoramic-image sequence with the scene environment sound and the three-dimensional geographic coordinates, to obtain a real-geographic-space-scene video streaming having augmented reality, space audios and geographic coordinates located in a geographic-coordinate frame; and
by the high-performance computing device, sending the real-geographic-space-scene video streaming to an output controlling device, so that the output controlling device displays the real-geographic-space scene in real time, and an operator sends the remote-control instruction by using the output controlling device, to remotely in real time control all or part of the panoramic cameras to shoot or move.

2. The method for constructing the real-geographic-space scene in real time according to claim 1 , wherein the marking points comprise a plurality of first marking points and a plurality of second marking points;
the plurality of panoramic cameras are installed in the real-geographic-space scene in a fixing manner or a stringing manner, and provide marking points in the real-geographic-space scene comprises:
if the plurality of panoramic cameras are installed in the real-geographic-space scene in a fixing manner, the plurality of panoramic cameras are installed fixedly and evenly in the real-geographic-space scene, an installation spacing distance is set according to a precision of images collected by the panoramic cameras, while ensuring that a shooting coverage area of each of the panoramic cameras satisfies an application demand of an actual scene, ensuring that a coinciding degree of neighboring images is greater than a threshold, and the plurality of first marking points are individually provided in overlapping view fields of the plurality of panoramic cameras; and
if the plurality of panoramic cameras are installed in the real-geographic-space scene in a stringing manner, the plurality of panoramic cameras are installed on a guiding device, the guiding device is divided into a plurality of local line segments, each of the local line segments is installed with at least one of the panoramic cameras, each of the local line segments is capable of independently driving the panoramic camera installed thereon to move, and according to movement speeds and requirement on real-time capabilities of the plurality of panoramic cameras, the plurality of second marking points are provided in the real-geographic-space scene; and
each of the local line segments is provided with anticollision rings at two ends, to prevent derailment of the at least one of the panoramic cameras installed on each of the local line segments in movement, and a shooting speed of each of the panoramic cameras matches with a movement speed thereof, to enable a degree of overlapping in the image-data sequence to be sufficient to complete large-range splicing, and to guarantee real-time capabilities of image splicing and fusion in reconstruction of the real-geographic-space scene.

3. The method for constructing the real-geographic-space scene in real time according to claim 2 , wherein the step of, by the high-performance computing device, receiving the image data collected by the plurality of panoramic cameras, and combining the image data into the image-data sequence comprises:
according to different installation modes of the plurality of panoramic cameras, using different modes to collect the image data;
if the plurality of panoramic cameras are installed in the fixing manner, according to a demand of a target area, by the high-performance computing device, forwarding the remote-control instruction, controlling the plurality of panoramic cameras to individually collect a real-time panoramic image of the target area, and combining into a real-time-panoramic-image data sequence;
if the plurality of panoramic cameras are installed in the stringing manner, according to a demand of the target area, by the high-performance computing device, forwarding the remote-control instruction, and controlling a movement mode of the plurality of panoramic cameras by using the guiding device, wherein the movement mode comprises that each of the plurality of panoramic cameras moves independently, or that some of the panoramic cameras move synchronously;
if the plurality of panoramic cameras are controlled to move synchronously, by the high-performance computing device, receiving dynamic-image data of the target area and a non-target area that are collected by each of the plurality of panoramic cameras, and combining into a dynamic-image data sequence; and
if part of the plurality of panoramic cameras are controlled to move independently, by the high-performance computing device, receiving dynamic-image data of the target area that are collected by the part of the plurality of panoramic cameras, receiving static-image data of the non-target area that are collected by motionless panoramic cameras, and combining into a dynamic-image data sequence.

4. The method for constructing the real-geographic-space scene in real time according to claim 1 , wherein the measuring robot, after acquiring the three-dimensional geographic coordinate of the prism, by usin

Assignees

• Univ Beijing
• Beijing Longruan Technologies Inc

Inventors

• Mao Shanjun
• Fan Yingbo
• Li Ben
• Chen Huazhou
• Li Xinchao

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T5/006

  Physics · mapped topic

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N5/265

  Mixing · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title