Detecting occlusion of digital ink

Patent metadata

Field	Value
Publication number	US-11720745-B2
Application number	US-201715621613-A
Country	US
Kind code	B2
Filing date	Jun 13, 2017
Priority date	Jun 13, 2017
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

An image processing apparatus is described comprising a processor configured to receive a video and digital ink annotated on the video. For at least a first frame of the video, the processor is configured to compute a model describing pixels of a bounding region of the ink. For a frame of the video, the processor is configured to compute a second region corresponding to the bounding region. The processor is configured to compute a comparison between the second region and the model and update the ink using the comparison.

First claim

The invention claimed is:

1. An image processing apparatus configured to detect occlusion of digital ink in a digitally annotated video, the image processing apparatus comprising:
a processor configured to:
receive a video;
identify an annotation made to a first frame in the video, the annotation being made after the first frame in the video has been captured, the annotation comprising digital ink applied to the first frame in the video;
for the first frame of the video, compute a model describing pixels in a bounding region containing the digital ink;
for a second frame of the video, identify an object in a second region that overlaps the bounding region;
compute a comparison between pixels in the second region and the model;
based on the computing, determine that the object is in a foreground with respect to the pixels described by the model;
based on determining that the object is in the foreground with respect to pixels described by the model, determine that a portion of the object is occluding at least a portion of the digital ink;
based on determining that the portion of the object is occluding at least the portion of the digital ink, hide at least the portion of the digital ink occluded by the portion of the object; and
update a display of the digital ink in the second frame to reflect the hiding of at least the portion of the digital ink.

2. The image processing apparatus of claim 1 , wherein the model comprises a statistical model of at least one channel of the pixels in the bounding region, and the processor is configured to compute the comparison by comparing the intensity of the at least one channel of the pixels in the second region to the model.

3. The image processing apparatus of claim 1 , wherein:
the model comprises a set of sub-models describing pixels of the first frame in a set of respective first sub-regions making up the bounding region;
the second region comprises a set of second sub-regions corresponding to the set of first sub-regions; and
the processor is configured to compute a comparison between each pixel of each second sub-region and the corresponding sub-model.

4. The image processing apparatus of claim 3 , wherein each sub-model describes a cell of a grid of pixels.

5. The image processing apparatus of claim 3 , wherein the processor is configured to interpolate the comparison at the boundaries between neighboring sub-regions.

6. The image processing apparatus of claim 1 , wherein the comparison comprises a computed similarity value between pixels of the second region and the model.

7. The image processing apparatus of claim 6 , wherein the comparison comprises an occlusion map indicating pixels of the second region which have a similarity value below a predetermined threshold value.

8. The image processing apparatus of claim 6 , wherein the comparison comprises an occlusion probability map calculated from the similarity values of each of the pixels of the second region.

9. The image processing apparatus of claim 8 , wherein the probability map is filtered using a cross bilateral filter to generate the comparison.

10. The image processing apparatus of claim 8 , wherein the processor is configured to update the ink by applying the probability map to the ink.

11. The image processing apparatus of claim 6 , wherein
the processor is further configured to:
determine data from the second region; and
update the model to describe pixels of the second region which have a similarity value above a predetermined threshold value if the data from the second region has not yet been included in the model.

12. The image processing apparatus of claim 11 , wherein the model is updated according to a learning algorithm having a learning rate.

13. The image processing apparatus of claim 12 , wherein the processor is configured to change the learning rate.

14. The image processing apparatus of claim 6 , wherein the processor is configured to compute the comparison by:
generating an occlusion map indicating occluded pixels of the second region which have a similarity value below a predetermined threshold value; and
for each pixel not indicated as occluded, marking the pixel as occluded if the number of occluded pixels in a selected neighborhood of the pixel is above a predetermined threshold value.

15. The image processing apparatus of claim 1 , wherein the processor is further configured to, prior to updating the ink or prior to generating the comparison:
check a set of criteria for the video, the criteria being indicative of the occlusion mechanism being effective; and
if the criteria is not fulfilled, then abort any remaining steps for detecting occlusion.

16. The image processing apparatus of claim 1 , wherein the processor is configured to compute a second region corresponding to the bounding region by:
selecting a plurality of template pixels of the bounding region;
matching the template pixels of the bounding region to corresponding matching template pixels in the subsequent frame:
generating a homography transform matrix using the matching; and
applying the homography transform matrix to the bounding region to generate the second region.

17. The image processing apparatus of claim 1 , wherein the bounding region is associated with an object to which the digital ink was applied in the first frame.

18. The image processing apparatus of claim 17 , wherein computing the comparison between pixels in the second region and the model further comprises computing a comparison between pixels in the second region and the object to which the digital ink was applied.

19. A computer-implemented method for detecting occlusion of digital ink in a digitally annotated video, comprising operations including:
receiving a video;
receiving an annotation made to a first frame in the video, the annotation being made after the first frame in the video has been captured, the annotation comprising digital ink applied to the first frame of the video;
for the first frame of the video, computing a model describing pixels of a bounding region containing the digital ink;
for a second frame of the video, identifying an object in a second region that overlaps the bounding region;
computing a comparison between pixels in the second region and the model;
based on the computing, determining that the object is in a foreground with respect to the pixels described by the model;
based on determining that the object is in the foreground with respect to pixels described by the model, determining that a portion of the object is occluding at least a portion of the digital ink;
based on determining that the portion of the object is occluding at least the portion of the digital ink, hiding at least the portion of the digital ink occluded by the portion of the object; and
updating a display of the digital ink in the second frame to reflect the hiding of at least the portion of the digital ink.

20. One or more device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to perform operations comprising:
receiving a video;
identifying an annotation made to a first frame in the video, the annotation being made after the first frame in the video has been captured, the annotation comprising digital ink applied to the first frame of the video;
for the first frame of the video, computing a model describing pixels of a bounding region containing the digital ink;
for a second frame of the video, identifying an object in a second region th

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Windmark Johan
• Träff Gustav

Classifications

• G06F40/171 Primary

  by use of digital ink · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F40/216

  using statistical methods · CPC title

• G11B27/02

  Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers · CPC title

• G11B27/036 Primary

  Insert-editing · CPC title