Neural network-based image stream modification

Patent metadata

Field	Value
Publication number	US-11830209-B2
Application number	US-202217651524-A
Country	US
Kind code	B2
Filing date	Feb 17, 2022
Priority date	May 26, 2017
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

Systems, devices, media, and methods are presented for object detection and inserting graphical elements into an image stream in response to detecting the object. The systems and methods detect an object of interest in received frames of a video stream. The systems and methods identify a bounding box for the object of interest and estimate a three-dimensional position of the object of interest based on a scale of the object of interest. The systems and methods generate one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position estimated for the object of interest. The one or more graphical elements are generated within the video stream to form a modified video stream. The systems and methods cause presentation of the modified video stream including the object of interest and the one or more graphical elements.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, an image;
detecting an object of interest within at least a region of interest in the image;
identifying a bounding box for the object of interest;
determining a distance between the bounding box and the region of interest;
comparing the distance to a threshold;
applying a nonlinear function to compute a score based on a result of comparing the distance to the threshold, the non-linear function comprising a plurality of conditions including a score of neighboring; detections, removal of neighboring; detections and an average precision;
decaying the score of the bounding box that has a higher measure of overlap with the region of interest than a second bounding box rather than suppressing the score; and
generating, based on the score of the bounding box, one or more graphical elements within the image to form a modified image.

2. The method of claim 1 , further comprising:
identifying the bounding box for the object of interest using a detector.

3. The method of claim 2 , wherein the detector is trained to identify the bounding box by reducing loss between a predicted box for a training image and a ground-truth box of the training image.

4. The method of claim 1 , wherein the image is a frame of a video.

5. The method of claim 1 , wherein the non-linear function decays scores of bounding boxes linearly based on a measure of overlap in relation to the threshold.

6. The method of claim 5 , wherein the nonlinear function applies a sudden penalty to bounding boxes for which the measure of overlap transgresses the threshold.

7. The method of claim 1 , further comprising:
causing presentation of the modified image including the object of interest and the one or more graphical elements.

8. The method of claim 1 , further comprising generating an ephemeral message comprising the modified image, the ephemeral message being deleted following a deletion trigger event comprising viewing time or viewing completion.

9. The method of claim 1 , wherein the one or more graphical elements comprise a first graphical element positioned proximate to the object of interest in the image and a second graphical element that is animated to interact with the first graphical element.

10. The method of claim 1 , further comprising estimating a three-dimensional position of the object of interest based on a scale of the object of interest, wherein the graphical elements have a size based on the scale of the object of interest and a position.

11. The method of claim 10 , wherein the three-dimensional position of the object of interest is a first position, and generating the one or more graphical elements further comprises:
tracking the object of interest within the modified image to identify a position change of the object of interest within the modified image, the position change reflecting movement from the first position to a second position; and
generating a modified position for the one or more graphical elements in response to the position change of the object of interest, the modified position corresponding to the second position of the object of interest.

12. The method of claim 1 , further comprising:
identifying one or more metadata elements corresponding to the object of interest;
generating a set of metadata tags corresponding to the one or more metadata elements identified for the object of interest; and
generating a modified bounding box by associating the set of metadata tags with the bounding box.

13. The method of claim 1 , wherein the one or more graphical elements includes an image animated to fill the bounding box, the one or more graphical elements being occluded by the object of interest in the image.

14. The method of claim 1 , wherein the one or more graphical elements modify a portion of the object of interest, further comprising:
generating the one or more graphical elements to have an element context corresponding to at least one metadata tag of a set of metadata tags, the one or more graphical elements being animated or rendered as a background behind the object of interest and comprising a character peeking out from behind the object of interest.

15. The method of claim 1 , further comprising:
applying an overlap-based weighting function to compute a measure of overlap; and
applying a continuous penalty function to compute the score in which a relatively high penalty is applied in response to the measure of overlap transgressing the threshold and in which no penalty is applied in response to the measure of overlap falling below the threshold.

16. The method of claim 1 , wherein the one or more graphical elements cause a field of view within the image to appear to be lit from an artificial position different from that of a real-world lighting source depicted in the image.

17. A system comprising:
one or more processors; and
a non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, by one or more processors, an image;
detecting an object of interest within at least a region of interest in the image;
identifying a bounding box for the object of interest;
determining a distance between the bounding box and the region of interest;
comparing the distance to a threshold;
applying a non-linear function to compute a score based on a result of comparing the distance to the threshold, the non-linear function used to compute the score comprising a plurality of conditions including a score of neighboring detections, removal of neighboring detections and an average precision;
decaying the score of the hounding box that has a higher measure of overlap with the region of interest than a second bounding box rather than suppressing the score; and
generating, based on the score of the bounding box, one or more graphical elements within the image to form a modified image.

18. The system of claim 17 , wherein the one or more graphical elements cause a field of view within the image to appear to be lit from an artificial position different from that of a real-world lighting source depicted in the image.

19. The system of claim 17 , wherein the neighboring detections are decreased to an extent the neighboring detections have a smaller likelihood of increasing a false positive rate, and wherein the average precision is measured over a range of overlap thresholds.

20. A non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:
receiving, by one or more processors, an image;
detecting an object of interest within at least a region of interest in the image;
identifying a bounding box for the object of interest;
determining a distance between the bounding box and the region of interest;
comparing the distance to a threshold;
applying a non-linear function to compute a score based on a result of comparing the distance to the threshold, the non-linear function used to compute the score comprising a plurality of conditions including a score of neighboring detections, removal of neighboring detections and an average precision;
decaying the score of the bounding box that has a higher measure of overlap with the region of interest than a second bounding box rather than suppressing the score; and
generating, based on the score of th

Assignees

• Snap Inc

Inventors

• Chen Travis
• Hare Samuel Edward
• Li Yuncheng
• Mathew Tony
• Solichin Jonathan
• Yang Jianchao
• Zhang Ning

Classifications

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title