Systems and methods for image description generation

Patent metadata

Field	Value
Publication number	US-10360466-B2
Application number	US-201615391735-A
Country	US
Kind code	B2
Filing date	Dec 27, 2016
Priority date	Dec 27, 2016
Publication date	Jul 23, 2019
Grant date	Jul 23, 2019

Abstract

Systems, methods, and non-transitory computer-readable media can receive an image. One or more concepts depicted in the image are identified based on machine learning techniques. The one or more concepts are filtered based on filtering criteria to identify one or more selected concepts. An image description is generated comprising the one or more selected concepts.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a computing system, an image;
identifying, by the computing system, one or more concepts depicted in the image based on machine learning techniques;
filtering, by the computing system, the one or more concepts based on filtering criteria to identify one or more selected concepts;
generating, by the computing system, an image description comprising the one or more selected concepts; and
providing, by the computing system, the image description for presentation to a user.

2. The computer-implemented method of claim 1 , further comprising:
assigning each concept of the one or more concepts a confidence score indicative of a likelihood that the concept is depicted in the image.

3. The computer-implemented method of claim 2 , wherein the filtering criteria comprises one or more confidence score thresholds.

4. The computer-implemented method of claim 3 , wherein the filtering the one or more concepts comprises:
for each concept of the one or more concepts,
querying a whitelist comprising a plurality of concepts and a plurality of confidence score thresholds to determine a confidence score threshold associated with the concept; and
determining whether the confidence score assigned to the concept satisfies the confidence score threshold.

5. The computer-implemented method of claim 4 , wherein the filtering the one or more concepts comprises excluding from the one or more selected concepts any concepts of the one or more concepts that are not listed in the whitelist.

6. The computer-implemented method of claim 1 , further comprising embedding the image description in the image as alternative text.

7. The computer-implemented method of claim 1 , wherein the identifying one or more concepts further comprises identifying one or more concepts based on optical character recognition techniques.

8. The computer-implemented method of claim 1 , further comprising
receiving a request for additional information about the image.

9. The computer-implemented method of claim 8 , further comprising, responsive to receiving the request for additional about the image, presenting additional image information, wherein the additional image information comprises at least one concept of the one or more concepts that was not included in the one or more selected concepts.

10. The computer-implemented method of claim 1 , wherein the providing the image description for presentation to the user comprises providing the image description for audible presentation to the user.

11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform a method comprising:
receiving an image;
identifying one or more concepts depicted in the image based on machine learning techniques;
filtering the one or more concepts based on filtering criteria to identify one or more selected concepts;
generating an image description comprising the one or more selected concepts; and
providing the image description for presentation to a user.

12. The system of claim 11 , wherein the method further comprises:
assigning each concept of the one or more concepts a confidence score indicative of a likelihood that the concept is depicted in the image.

13. The system of claim 12 , wherein the filtering criteria comprises one or more confidence score thresholds.

14. The system of claim 13 , wherein the filtering the one or more concepts comprises:
for each concept of the one or more concepts,
querying a whitelist comprising a plurality of concepts and a plurality of confidence score thresholds to determine a confidence score threshold associated with the concept; and
determining whether the confidence score assigned to the concept satisfies the confidence score threshold.

15. The system of claim 14 , wherein the filtering the one or more concepts comprises excluding from the one or more selected concepts any concepts of the one or more concepts that are not listed in the whitelist.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
receiving an image;
identifying one or more concepts depicted in the image based on machine learning techniques;
filtering the one or more concepts based on filtering criteria to identify one or more selected concepts;
generating an image description comprising the one or more selected concepts; and
providing the image description for presentation to a user.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the method further comprises:
assigning each concept of the one or more concepts a confidence score indicative of a likelihood that the concept is depicted in the image.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the filtering criteria comprises one or more confidence score thresholds.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the filtering the one or more concepts comprises:
for each concept of the one or more concepts,
querying a whitelist comprising a plurality of concepts and a plurality of confidence score thresholds to determine a confidence score threshold associated with the concept; and
determining whether the confidence score assigned to the concept satisfies the confidence score threshold.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the filtering the one or more concepts comprises excluding from the one or more selected concepts any concepts of the one or more concepts that are not listed in the whitelist.

Assignees

• Facebook Inc

Inventors

• Wu Shaomei
• Adamic Lada Ariana
• Wieland Jeffrey C
• Farivar Omid
• Pique Corchs Hermes Germi
• King Matt
• Lavalla Brett Alden
• Paluri Balamanohar

Classifications

• G06V30/1448

  based on markings or identifiers characterising the document or the area · CPC title

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06V30/224

  of printed characters having additional code marks or containing code marks · CPC title

• G06V30/10

  Character recognition · CPC title