Generating data in a messaging system for a machine learning model

Patent metadata

Field	Value
Publication number	US-11599741-B1
Application number	US-202016774796-A
Country	US
Kind code	B1
Filing date	Jan 28, 2020
Priority date	Dec 1, 2017
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Systems and methods are provided for analyzing, by a computing device, location data associated with a location of the computing device to determine that an image or video captured using a messaging application on the computing device is captured near a food-related venue or event, receiving input related to food associated with the food-related venue or event, sending the image or video and the input related to food associated with the food-related venue or event to a computing system to train a machine learning model for food detection, and updating the messaging application to comprise the trained machine learning model for food detection.

First claim

What is claimed is:

1. A method comprising:
analyzing, by a computing device, location data associated with a location of the computing device to determine that an image or video captured using a messaging application on the computing device is captured near a food-related venue or event;
receiving, by the computing device, input related to food associated with the food-related venue or event;
sending, by the computing device, the image or video and the input related to food associated with the food-related venue or event to a computing system to train a machine learning model for food detection;
updating, by the computing device, the messaging application to comprise the trained machine learning model for food detection;
detecting capture of a second image or video using the messaging application; and
determining that the second image or video comprises food using the trained machine learning model for food detection.

2. The method of claim 1 , wherein determining that the second image or video comprises food using the trained machine learning model for food detection comprises:
inputting the second image or video into the machine learning model for food detection;
generating a prediction value for the second image or video;
determining whether the prediction value exceeds a predetermined threshold value; and
determining that the second image or video comprises food based on the prediction value exceeding the predetermined threshold value.

3. The method of claim 1 , wherein analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event comprises comparing the location data to map data to determine that the location of the computing device is near the food-related venue or event.

4. The method of claim 1 , wherein the input is received in response to presented interactive features and the interactive features comprise a media overlay, a lens, or a request for a review of the food-related venue or event.

5. The method of claim 1 , wherein the input is received in response to presented interactive features and the interactive features comprise a request for an image of food from the food-related venue or event, and the image or video sent to the computing system is the requested image.

6. The method of claim 1 , wherein the computing system receives the image or video and associated input from the computing device and stores the received image or video and associated input in a database with a plurality of captured images and videos and associated input from a plurality of computing devices.

7. The method of claim 1 , wherein the trained machine learning model for food detection is further trained on a plurality of captured images and videos and input sent by a plurality of computing devices.

8. The method of claim 1 , further comprising:
detecting capture of a third image or video using the messaging application;
determining that the third image or video is captured near a food-related venue or event;
determining that the third image or video does not comprise food, using the trained machine learning model for food detection; and
presenting interactive features to incentivize capture of an image of food associated with the food-related venue or event.

9. The method of claim 8 , further comprising:
in response to the presented interactive features, receiving a fourth image; and
sending the fourth image to the computing system to update the trained machine learning model for food detection.

10. A computing device comprising:
one or more hardware processors; and
a computer-readable medium coupled with the one or more hardware processors, the computer-readable medium comprising instructions stored thereon that are executable by the one or more hardware processors to cause the computing device to perform operations comprising:
analyzing location data associated with a location of the computing device to determine that an image or video captured using a messaging application on the computing device is captured near a food-related venue or event;
receiving input related to food associated with the food-related venue or event;
sending the image or video and the input related to food associated with the food-related venue or event to a computing system to train a machine learning model for food detection;
updating the messaging application to comprise the trained machine learning model for food detection;
detecting capture of a second image or video using the messaging application; and
determining that the second image or video comprises food using the trained machine learning model for food detection.

11. The computing device of claim 10 , wherein determining that the second image or video comprises food using the trained machine learning model for food detection comprises:
inputting the second image or video into the machine learning model for food detection;
generating a prediction value for the second image or video;
determining whether the prediction value exceeds a predetermined threshold value; and
determining that the second image or video comprises food based on the prediction value exceeding the predetermined threshold value.

12. The computing device of claim 10 , wherein analyzing the location data associated with the location of the computing device to determine that the image or video is captured near the food-related venue or event comprises comparing the location data to map data to determine that the location of the computing device is near the food-related venue or event.

13. The computing device of claim 10 , wherein the input is received in response to presented interactive features and the interactive features comprise a media overlay, a lens, or a request for a review of the food-related venue or event.

14. The computing device of claim 10 , wherein the input is received in response to presented interactive features and the interactive features comprise a request for an image of food from the food-related venue or event, and the image or video sent to the computing system is the requested image.

15. The computing device of claim 10 , wherein the trained machine learning model for food detection is further trained on a plurality of captured images and videos and input sent by a plurality of computing devices.

16. The computing device of claim 10 , further comprising:
detecting capture of a third image or video using the messaging application;
determining that the third image or video is captured near a food-related venue or event;
determining that the third image or video does not comprise food, using the trained machine learning model for food detection; and
presenting interactive features to incentivize capture of an image of food associated with the food-related venue or event.

17. The computing device of claim 16 , the operations further comprising:
in response to the presented interactive features, receiving a fourth image; and
sending the fourth image to the computing system to update the trained machine learning model for food detection.

18. A non-transitory computer-readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:
analyzing location data associated with a location of the computing device to determine that an image or video captured using a messaging application on the computing device is captured near a food-related venue or event;
receiving input related to food associated wit

Assignees

• Snap Inc

Inventors

• Xue Zehao
• Ren Zhou

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/68

  Food, e.g. fruit or vegetables · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06N20/00 Primary

  Machine learning · CPC title