Object removal using lidar-based classification

Patent metadata

Field	Value
Publication number	US-9905032-B2
Application number	US-201615384153-A
Country	US
Kind code	B2
Filing date	Dec 19, 2016
Priority date	Jun 14, 2013
Publication date	Feb 27, 2018
Grant date	Feb 27, 2018

Abstract

In scenarios involving the capturing of an environment, it may be desirable to remove temporary objects (e.g., vehicles depicted in captured images of a street) in furtherance of individual privacy and/or an unobstructed rendering of the environment. However, techniques involving the evaluation of visual images to identify and remove objects may be imprecise, e.g., failing to identify and remove some objects while incorrectly omitting portions of the images that do not depict such objects. However, such capturing scenarios often involve capturing a lidar point cloud, which may identify the presence and shapes of objects with higher precision. The lidar data may also enable a movement classification of respective objects differentiating moving and stationary objects, which may facilitate an accurate removal of the objects from the rendering of the environment (e.g., identifying the object in a first image may guide the identification of the object in sequentially adjacent images).

First claim

What is claimed is:

1. A method of generating a rendering of an environment including at least one object, the method performed on at least one device comprising at least one processor and the method comprising:
generating for the environment a lidar point cloud comprising at least one lidar point;
based on lidar points in the lidar point cloud, identifying at least one object in a first representation of the environment;
based on an identified position of the object in the first representation, estimating a position of the object in a second representation of the environment;
based on the estimated position, identifying the object in a second representation of the environment; and
generating the rendering of the environment omitting at least a portion of the object based on the identification of the object in the second representation.

2. The method of claim 1 , further comprising:
selecting a movement classification of an object in the environment using the lidar point cloud.

3. The method of claim 1 , wherein
generating the rendering of the environment comprises:
focusing a portion of an image depicting the object using a movement classification of the object; and
generating the rendering of the environment omitting the portion of the image depicting the object.

4. The method of claim 3 , wherein
generating the rendering of the environment comprises:
focusing an image portion of the image depicting a selected object portion of the object using a movement classification of the object; and
generating the rendering of the environment omitting the image portion depicting the selected object portion of the object.

5. The method of claim 1 , wherein
identifying the at least one object in the first representation of the environment further comprises:
applying an optical character recognizer to the first representation of the environment to detect at least one character; and
associating the at least one character with the object in the environment.

6. The method of claim 1 , further comprising selecting the movement classification for the object, wherein the movement classification is selected from a movement classification set comprising:
a moving object;
a stationary foreground object;
a stationary background object; and
a fixed-ground object.

7. The method of claim 1
further comprising:
capturing at least one image of the environment; and omitting the at least an object portion of the respective at least one object comprising: blurring at least an image portion of at least one image depicting the at least one object portion of the at least one object.

8. The method of claim 1 , wherein:
at least one selected object comprises a personal identifier of at least one individual associated with the object; and
omitting the at least an object portion of the selected object comprises removing from the rendering of the environment the at least one personal identifier of the at least one individual.

9. The method of claim 8 wherein:
the object comprises an individual;
the personal identifier comprises at least one recognizable feature of the individual; and
omitting the at least an object portion of the selected object further comprises removing the at least one recognizable feature of the individual from the rendering of the environment.

10. A system comprising:
at least one processor; and
a memory operatively coupled to the at least one processor, the memory storing instructions that when executed by the at least one processor perform a set of operations comprising:
generating, for an environment, a lidar point cloud comprising at least one lidar point;
based on lidar points in the lidar point cloud, identifying at least one object in a first representation of the environment;
based on an identified position of the object in the first representation, estimating a position of the object in a second representation of the environment;
based on the estimated position, identifying the object in a second representation of the environment; and
generating the rendering of the environment omitting at least a portion of the object based on the identification of the object in the second representation.

11. The system of claim 10 , wherein generating the rendering of the environment further comprises, for least one background portion of the rendering of the environment that is obscured by a selected object, replacing at least one object portion of at least one object depicted in the environment with a background portion of the environment.

12. The system of claim 10 , wherein the operations further comprise, for object that is moving in the environment according to a movement classification, generate within the rendering a depiction of the object moving through the environment.

13. The system of claim 10 , wherein the operations further comprise estimating a movement vector of the object, and wherein estimating the position of the object in the second representation of the environment is further based on the movement vector of the object.

14. The system of claim 10 , wherein the operations further comprise identifying an object type for the object.

15. The system of claim 10 , wherein the object is one a vehicle or an individual.

16. A computer-readable storage device comprising instructions that, when executed on a processor of a device, cause the device to generate a rendering of an environment including at least one object by:
based on lidar points in a lidar point cloud representing the environment, identifying at least one object in a first representation of the environment;
based on an identified position of the object in the first representation, estimating a position of the object in a second representation of the environment;
based on the estimated position, identifying the object in a second representation of the environment; and
generating the rendering of the environment omitting at least a portion of the object based on the identification of the object in the second representation.

17. The computer-readable storage device of claim 16 , wherein estimating the position of the object in the second representation of the environment is further based on a movement classification for the object.

18. The computer-readable storage device of claim 17 , wherein the movement classification is one of:
a moving object;
a stationary foreground object;
a stationary background object; and
a fixed-ground object.

19. The computer-readable storage device of claim 16 , wherein estimating the position of the object in the second representation of the environment is further based on a movement vector.

20. The computer-readable storage device of claim 16 , wherein the object is one a vehicle or an individual.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Rogan Aaron Matthew
• Kadlec Benjamin James

Classifications

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G01S7/4802

  using analysis of echo signal for target characterisation; Target signature; Target cross-section · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title