Synthetic colorization of real-time immersive environments

Patent metadata

Field	Value
Publication number	US-9639958-B2
Application number	US-201514662762-A
Country	US
Kind code	B2
Filing date	Mar 19, 2015
Priority date	Mar 19, 2015
Publication date	May 2, 2017
Grant date	May 2, 2017

Abstract

A system related to synthetic colorization of real-time immersive environments is disclosed. The system discloses converting real-time camera data on-board a machine into signatures which describe the real-time camera data. The signatures are based on image properties, such as color, intensity, and illumination. The signatures, which require much less bandwidth than the real-time camera data to transmit, are transmitted to a remote operator station. The remote operator station utilizes the signatures to synthetically color and texture the environment surrounding the machine to provide an immersive environment to a user at the remote operator station.

First claim

We claim:

1. A system for synthetically coloring and texturing an image, the system comprising:
a plurality of sensors located at a first location configured to gather data about an environment surrounding the plurality of sensors;
a first computer processor located at the first location configured to generate at least one signature, the at least one signature being a description of the data gathered by the plurality of sensors, and to transmit the at least one signature to a second location;
a second computer processor located at the second location configured to:
receive the at least one signature and to generate a synthetic environment based on the at least one signature,
compare a sub-image with adjacent sub-images in an overall image using an iterative algorithm, and
modify one or more characteristics of the sub-image to more closely match one or more image characteristics of the adjacent sub-images.

2. The system of claim 1 , wherein:
one sensor of the plurality of sensors located at the first location is a Light Detection and Ranging (LIDAR) sensor; and
the first computer processor is further configured to generate a point cloud based on data gathered by the LIDAR sensor and to transmit the point cloud from the first location to the second location.

3. The system of claim 2 , wherein the second computer processor is further configured to correlate the synthetic environment to a specific location in the point cloud.

4. The system of claim 1 , wherein the at least one signature is generated based on at least one of a color, an intensity, and an illumination.

5. The system of claim 1 , wherein the second computer processor is further configured to use the at least one signature as an index to select a synthetic color and texture from a plurality of synthetic colors and textures.

6. The system of claim 1 , wherein the first computer processor is further configured to generate a variably sized window, the size of the variably sized window is based on a shape of a three-dimensional (3D) ground surface as sensed by one sensor of the plurality of sensors.

7. The system of claim 6 , wherein the at least one signature is based on the variably sized window.

8. The system of claim 1 , wherein the second computer processor is further configured to manipulate the synthetic environment to highlight features of the area surrounding the plurality of sensors of the first location.

9. The system of claim 1 , wherein the second computer processor collects signatures over a time period to generate baseline synthetic color and texture parameters of the first location.

10. A method for synthetically coloring and texturing an image, the method comprising:
gathering data about an environment of a first location using a plurality of sensors at the first location;
generating at least one signature, the at least one signature being a description of the data gathered by the plurality of sensors, at the first location, using a first computer processor located at the first location;
transmitting the at least one signature to a second location, using the first computer processor;
receiving the at least one signature at the second location, by a second computer processor located at the second location;
generating a synthetic environment at the second location based on the at least one signature, using the second computer processor;
comparing a sub-image with adjacent sub-images in an overall image using an iterative algorithm executed by the second computer processor; and
modifying one or more characteristics of the sub-image to more closely match one or more image characteristics of the adjacent sub-images, using the second computer processor.

11. The method of claim 10 , further comprising:
generating a point cloud based on data gathered by a Light Detection and Ranging (LIDAR) sensor; and
transmitting the point cloud from the first location to the second location, using the first computer processor.

12. The method of claim 11 , wherein the generating the synthetic environment at the second location further comprises correlating the synthetic environment to a specific location in the point cloud.

13. The method of claim 10 , wherein the generating the synthetically colored and textured environment at the second location further comprises:
using the at least one signature as an index to select a synthetic color and texture from a plurality of synthetic colors and textures by the second computer processor.

14. The method of claim 10 , further comprising:
generating a variably sized window, wherein a size of the variably sized window is based on a shape of a three-dimensional (3D) ground surface as sensed by one sensor of the plurality of sensors, using the first computer processor.

15. The method of claim 14 , wherein the at least one signature is based on the variably sized window.

16. The method of claim 10 , further comprising:
manipulating the synthetically colored and textured environment to highlight features of the environment of the first location, using the second computer processor.

17. The method of claim 10 , further comprising:
collecting signatures at the second location, by the second computer processor, over a time period to generate baseline synthetic color and texture parameters of the first location.

18. A system for synthetically coloring and texturing an image, the system comprising:
a plurality of sensors located at a first location configured to gather data about an environment surrounding the plurality of sensors;
a first computer processor located at the first location configured to:
generate at least one signature, the at least one signature being a description of the data gathered by the plurality of sensors,
transmit the at least one signature to a second location; and
generate a variably sized window, the size of the variably sized window is based on a shape of a three-dimensional (3D) ground surface as sensed by one sensor of the plurality of sensors; and

a second computer processor located at the second location configured to receive the at least one signature and to generate a synthetic environment based on the at least one signature.

Assignees

• Caterpillar Inc

Inventors

• Reiners Eric Alan
• Rybski Paul Edmund

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T7/408 Primary

  Physics · mapped topic

• G06T11/001

  Physics · mapped topic

• G06T7/90 Primary

  Determination of colour characteristics · CPC title

• G06T2210/08

  Bandwidth reduction · CPC title