Obfuscating portions of video data

Patent metadata

Field	Value
Publication number	US-12002174-B1
Application number	US-202318178269-A
Country	US
Kind code	B1
Filing date	Mar 3, 2023
Priority date	Dec 12, 2017
Publication date	Jun 4, 2024
Grant date	Jun 4, 2024

Abstract

Described are systems and methods for detecting objects using calibrated imaging devices and obfuscating, in real-time or near real time, portions of the video data to protect the privacy of operators represented in the video data. For example, a position of an operator within a fulfillment center may be determined or tracked in video data and the pixels representative of that operator may be obfuscated using pixilation and/or other techniques so that a reviewing agent that is viewing the video data cannot determine the identity of the operator. Such obfuscation may be performed in real-time or near real-time using automated processing. In addition, only portions of the video data may be obfuscated so that events (e.g., item picks, item place) and/or other objects represented in the video data are still viewable to the reviewing agent.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, from a device, a first video frame representative of a location within a field of view of the device;
determining a first plurality of pixels of the first video frame that are representative of at least a portion of an operator positioned at the location;
generating an obfuscation mask that indicates a second plurality of pixels corresponding to the first plurality of pixels that are to be obfuscated;
generating, based on the first video frame, a second video frame, wherein the second video frame is a lower resolution video frame of the first video frame, wherein the lower resolution is such that user identifiable information about the operator cannot be determined from the second video frame; and
sending, for concurrent display, the first video frame and the second video frame such that the second video frame is overlaid atop the first video frame when presented and obfuscates the operator so that the user identifiable information about the operator cannot be determined from the concurrent display of the first video frame and the second video frame.

2. The computer-implemented method of claim 1 , further comprising:
determining a background object boundary surrounding at least a portion of a background object that is to be presented without obfuscation; and
wherein the obfuscation mask does not include an indication of the second plurality of pixels within the background object boundary.

3. The computer-implemented method of claim 2 , wherein the background object is presented without obfuscation so that an event at the background object can be detected.

4. The computer-implemented method of claim 2 , further comprising:
determining an event boundary corresponding to at least a portion of the first video frame; and
adding to the obfuscation mask, a second indication of pixels of the first video frame that correspond to a position that is outside the event boundary.

5. The computer-implemented method of claim 1 , wherein determining the first plurality of pixels of the first video frame received from the device that are representative of the operator, further comprises:
generating, based at least in part on data corresponding to an area within the field of view, a three-dimensional model (“3D model”) that includes a plurality of points;
correlating a plurality of pixels of the first video frame with the plurality of points of the 3D model; and
determining, based at least in part on the points of the 3D model, the first plurality of pixels as the pixels of the first video frame aligned with points of the 3D model.

6. The computer-implemented method of claim 5 , further comprising:
determining, for a point of the 3D model, a first real-world coordinate corresponding to the point; and
determining, for a pixel of the first video frame, a second real-world coordinate corresponding to the pixel; and
wherein correlating is based at least in part on a mapping of the first real-world coordinate with the second real-world coordinate.

7. The computer-implemented method of claim 5 , further comprising:
determining that a first cluster of points of the 3D model correspond to the operator;
determining that a second cluster of points of the 3D model do not correspond to the operator; and
determining the first plurality of pixels as pixels aligned with the first cluster of points but not including pixels aligned with the second cluster of points.

8. The computer-implemented method of claim 5 , further comprising:
determining that a first cluster of points of the 3D model correspond to the operator;
determining that a second cluster of points of the 3D model correspond to a second operator; and
determining the first plurality of pixels as pixels aligned with the first cluster of points and pixels aligned with the second cluster of points.

9. A system, comprising:
an imaging device aligned to include at least a portion of a location within a field of view of the imaging device; and
a computing device in communication with at least the imaging device, wherein the computing device is configured to at least:
receive, from the imaging device, a first video frame representative of the location;
determine a first plurality of pixels of the first video frame that are representative of a first foreground object positioned at the location;
generate an obfuscation mask that indicates a second plurality of pixels corresponding to the first plurality of pixels that are to be obfuscated;
generate, based on the first video frame, a second video frame, wherein the second video frame is a lower resolution video frame of the first video frame, wherein the lower resolution is such that an identity of the first foreground object cannot be determined from the second video frame;
determine a third plurality of pixels of the second video frame that do not correspond to the second plurality of pixels of the obfuscation mask;
set color values for each of the third plurality of pixels to transparent; and
generate, based at least in part on the first video frame and the second video frame, an obfuscated video data such that the object cannot be identified when the obfuscated video data is presented.

10. The system of claim 9 , wherein:
the obfuscated video data includes the first video frame and the second video frame; and
the computing device is further configured to at least:
send the obfuscated video data to a second computing device; and
wherein the first video frame and the second video frame of the obfuscated video data are concurrently presented by the second computing device.

11. The system of claim 9 , wherein the computing device is further configured to at least:
determine that the first foreground object is an object positioned at the location; and
wherein generation of the obfuscation mask is based at least in part on a determination that the first foreground object is the object.

12. The system of claim 9 , wherein the computing device is further configured to at least:
determine an event boundary corresponding to at least a portion of the first video frame; and
indicate in the obfuscation mask, a fourth plurality of pixels corresponding to a position that is outside the event boundary; and
wherein the third plurality of pixels do not include the fourth plurality of pixels.

13. The system of claim 9 , wherein determination that the first plurality of pixels of the first video frame are representative of the first foreground object positioned at the location, further comprises:
generate a three-dimensional model (“3D model”) that includes a plurality of points;
correlate pixels of the first video frame with the plurality of points of the 3D model; and
determine, based at least in part on the points of the 3D model, the first plurality of pixels as the pixels of the first video frame aligned with points of the 3D model.

14. The system of claim 13 , wherein the computing device is further configured to at least:
determine, for a point of the 3D model, a first real-world coordinate corresponding to the point; and
determine, for a first pixel of the first video frame, a second real-world coordinate corresponding to the pixel; and
wherein correlation is based at least in part on a mapping of the first real-world coordinate with the second real-world coordinate.

15. The system of claim 9 , wherein determination that the first plurality of pixels of the first video frame are representative of the first foreground object positioned at the location, further comprises:
generate a three-dimensional model (“3D

Assignees

• Amazon Tech Inc

Inventors

• Barkus Adrian Bell
• Mohanasundaram Prathiban
• Riley Joseph M
• Guan Danny
• Ahmed Waqas Syed

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/75

  involving models · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title