Method and system for determining dynamism in a scene by processing depth image

Patent metadata

Field	Value
Publication number	US-11443446-B2
Application number	US-202017036213-A
Country	US
Kind code	B2
Filing date	Sep 29, 2020
Priority date	Sep 30, 2019
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

State of art systems in the domain of dynamism detection fail to estimate noise if depth images are being collected as input for the dynamism detection, as the noise in depth images depend on the scene being captured. Disclosed herein are method and system for determining dynamism by processing depth image of a scene. The system models depth sensor noise as ergodic stochastic process by determining that distribution estimated at each reference pixel from a plurality of neighborhood pixels in a reference image being processed is statistically same as a distribution estimated from evolution of the reference pixel over the time. After modeling the depth sensor noise in this manner, the same is eliminated/removed from the reference image, which is then processed to estimate divergence at each pixel based on temporal and spatial distribution built at pixel level in the reference image, and in turn determines dynamism in the scene.

First claim

What is claimed is:

1. A processor implemented method for determining dynamism, comprising:
collecting a depth image of a scene at time instance ‘t’ as a reference image, via one or more hardware processors, wherein a depth sensor noise associated with the collected reference image is modelled as an ergodic stochastic process;
re-projecting a plurality of historical depth images onto the time instance ‘t’, via the one or more hardware processors;
building a spatial distribution at each of a plurality of pixels of the reference image, by processing the reference image via the one or more hardware processors;
building a temporal distribution at each of the plurality of pixels of the reference image, by processing the plurality of re-projected historical depth images via the one or more hardware processors;
determining divergence at each of the plurality of pixels of the reference image, based on the spatial distribution and the temporal distribution, via the one or more hardware processors; and
determining dynamism in the scene, based on the determined divergence in one or more of the plurality of the pixels, via the one or more hardware processors.

2. The method as claimed in claim 1 , wherein the plurality of historical depth images are images of the scene taken prior to the time instance ‘t’.

3. The method as claimed in claim 1 , wherein modelling the depth sensor noise as the ergodic stochastic process comprises determining that distribution estimated at each reference pixel from a plurality of neighborhood pixels is statistically same as a distribution estimated from evolution of the reference pixel over the time.

4. The method as claimed in claim 3 , wherein the evolution of the reference pixel is a change of a pixel at the reference image, measured from the re-projected historical depth images at the pixel.

5. The method as claimed in claim 1 , wherein determining the dynamism in the scene based on the determined divergence comprises:
determining that object in at least a few of the plurality of pixels is a static object if value of determined divergence for the at least a few of the plurality of pixels is zero; and
determining that object in the at least a few of the plurality of pixels is a dynamic object if value of determined divergence for the at least a few of the plurality of pixels is a value exceeding zero.

6. The method as claimed in claim 1 , wherein extent of dynamism is determined based on value of the determined divergence between the spatial distribution and the temporal distribution.

7. A system for determining dynamism, comprising:
one or more hardware processors;
one or more communication interfaces; and
one or more memory storing a plurality of instructions, wherein the plurality of instructions when executed cause the one or more hardware processors to:
collect a depth image of a scene at time instance ‘t’ as a reference image, via one or more hardware processors, wherein a depth sensor noise associated with the collected reference image is modelled as an ergodic stochastic process;
re-project a plurality of historical depth images onto the time instance ‘t’, via one or more hardware processors;
build a spatial distribution at each of a plurality of pixels of the reference image, by processing the reference image via the one or more hardware processors;
build a temporal distribution at each of the plurality of pixels of the reference image, by processing the plurality of re-projected historical depth images via the one or more hardware processors;
determine divergence at each of the plurality of pixels of the reference image, based on the spatial distribution and the temporal distribution, via the one or more hardware processors; and
determine dynamism in the scene, based on the determined divergence in one or more of the pixels, via the one or more hardware processors.

8. The system as claimed in claim 7 , wherein the plurality of historical depth images are images of the scene taken prior to the time instance ‘t’.

9. The system as claimed in claim 7 , wherein the system is configured to model the depth sensor noise as the ergodic stochastic process by determining that distribution estimated at each reference pixel from a plurality of neighborhood pixels is statistically same as a distribution estimated from evolution of the reference pixel over the time.

10. The system as claimed in claim 9 , wherein the evolution of the reference pixel is a change of a pixel at the reference image, measured from the re-projected historical depth images at the pixel.

11. The system as claimed in claim 7 , wherein the system determines the dynamism in the scene based on the determined divergence by:
determining that object in at least a few of the plurality of pixels is a static object if value of determined divergence for the at least a few of the plurality of pixels is zero; and
determining that object in the at least a few of the plurality of pixels is a dynamic object if value of determined divergence for the at least a few of the plurality of pixels is a value exceeding zero.

12. The system as claimed in claim 7 , wherein the system determines an extent of dynamism based on value of the determined divergence between the spatial distribution and the temporal distribution.

13. A non-transitory computer readable medium for determining dynamism, the non-transitory computer readable medium comprising a plurality of instructions which when executed cause one or more hardware processors to:
collect a depth image of a scene at time instance ‘t’ as a reference image, wherein a depth sensor noise associated with the collected reference image is modelled as an ergodic stochastic process;
re-project a plurality of historical depth images onto the time instance ‘t’;
build a spatial distribution at each of a plurality of pixels of the reference image, by processing the reference image;
build a temporal distribution at each of the plurality of pixels of the reference image, by processing the plurality of re-projected historical depth images;
determine divergence at each of the plurality of pixels of the reference image, based on the spatial distribution and the temporal distribution; and
determine dynamism in the scene, based on the determined divergence in one or more of the plurality of the pixels.

14. The non-transitory computer readable medium as claimed in claim 13 , wherein modelling the depth sensor noise as the ergodic stochastic process comprises determining that distribution estimated at each reference pixel from a plurality of neighborhood pixels is statistically same as a distribution estimated from evolution of the reference pixel over the time.

15. The non-transitory computer readable medium as claimed in claim 13 , wherein determining the dynamism in the scene based on the determined divergence comprises:
determining that object in at least a few of the plurality of pixels is a static object if value of determined divergence for the at least a few of the plurality of pixels is zero; and
determining that object in the at least a few of the plurality of pixels is a dynamic object if value of determined divergence for the at least a few of the plurality of pixels is a value exceeding zero.

Assignees

• Tata Consultancy Services Ltd

Inventors

• Sinha Arnab

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/277

  involving stochastic approaches, e.g. using Kalman filters · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title