Modification of audio-based computer program output

Patent metadata

Field	Value
Publication number	US-10652170-B2
Application number	US-201715618842-A
Country	US
Kind code	B2
Filing date	Jun 9, 2017
Priority date	Jun 9, 2017
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

Modifying computer program output in a voice or non-text input activated environment is provided. A system can receive audio signals detected by a microphone of a device. The system can parse the audio signal to identify a computer program to invoke. The computer program can identify a dialog data structure. The system can modify the identified dialog data structure to include a content item. The system can provide the modified dialog data structure to a computing device for presentation.

First claim

What is claimed is:

1. A system to modify computer program output, comprising:
a data processing system having one or more processors and memory to:
receive, from a computing device, a digital file corresponding to a first acoustic signal carrying voice content detected by a microphone of the computing device, the first acoustic signal converted to the digital file by an analog to digital converter of the computing device;
select, responsive to the voice content of the digital file, a computer program comprising a chatbot from a plurality of computer programs comprising chatbots for execution;
identify, via the chatbot based on the voice content of the digital file, a dialog data structure comprising a placeholder field;
generate, responsive to identification of the placeholder field in the dialog data structure, a request for content in a parameterized format configured for a parametrically driven text to speech technique;
transmit the request for the content to a content selection component of the data processing system;
select, via a content selection process responsive to the request, a content item for insertion into the placeholder field of the dialog data structure, the content item in the parameterized format configured for the parametrically driven text to speech technique; and
provide, to the chatbot, the content item in the parameterized format selected via the content selection process to cause the computing device to perform the parametrically driven text to speech technique to generate a second acoustic signal corresponding to the dialog data structure modified with the content item, wherein the computing device plays the content item with an acoustic fingerprint corresponding to the chatbot.

2. The system of claim 1 , comprising the data processing system to:
determine, based on an identifier of the computing device and via a lookup in a data repository, that the computing device is authorized to access the computer program comprising the chatbot; and
select the computer program comprising the chatbot responsive to the determination that the computing device is authorized to access the chatbot.

3. The system of claim 1 , comprising:
the chatbot configured to use a natural language processing technique to identify the dialog data structure responsive to the digital file.

4. The system of claim 1 , wherein the dialog data structure comprises a tag that identifies the placeholder field and metadata of the placeholder field.

5. The system of claim 1 , comprising the data processing system to:
use a natural language processing technique to process the dialog data structure and identify a portion of the dialog data structure at which to insert the placeholder field; and
insert the placeholder field at the portion of the dialog data structure.

6. The system of claim 1 , comprising the data processing system to:
use a natural language processing technique to identify a grammar and syntax of the dialog data structure;
insert, based on the grammar and syntax of the dialog data structure, the placeholder field; and
provide, to the chatbot, the dialog data structure comprising the placeholder field.

7. The system of claim 1 , comprising the data processing system to:
generate a second dialog data structure comprising a second placeholder field;
compare the second placeholder field with the placeholder field;
determine, based on the comparison, to generate a second request for second content in the parameterized format; and
select a second content item for insertion into the second placeholder field of the second dialog data structure.

8. The system of claim 1 , comprising the data processing system to:
generate a second dialog data structure comprising a second placeholder field;
compare the second placeholder field with the placeholder field; and
determine, based on the comparison, to insert the content item selected for insertion in the placeholder into the second placeholder field of the second dialog data structure.

9. The system of claim 1 , comprising the data processing system to:
generate a second dialog data structure comprising a second placeholder field;
compare the second placeholder field with the placeholder field;
determine, based on the comparison, not to request a second content item for the second placeholder field of the second dialog data structure; and
reuse the content item selected for the placeholder field for insertion into the second placeholder field of the second dialog data structure.

10. The system of claim 1 , comprising the data processing system to:
select, via the content selection process, the content item for insertion based on profile information linked to the computing device.

11. The system of claim 1 , comprising the data processing system to:
select, via the content selection process, the content item for insertion based on a plurality of digital files corresponding to the computing device.

12. A method of modifying computer program output, comprising:
detecting, by a sensor of a computing device, a first image comprising visual content;
converting, by the computing device, the first image to a digital file corresponding to the visual content;
selecting, responsive to the visual content of the digital file, a computer program comprising a chatbot from a plurality of computer programs comprising chatbots for execution;
identifying, by the chatbot based on the visual content of the digital file, a dialog data structure comprising a placeholder field;
generating, responsive to identifying the placeholder field in the dialog data structure, a request for content in a parameterized format configured for a parametrically driven text to speech technique;
transmitting, by the chatbot, the request for the content to a content selection server;
selecting, by the content selection server responsive to the request, a content item for insertion into the placeholder field of the dialog data structure, the content item in the parameterized format configured for the parametrically driven text to speech technique; and
providing, to the chatbot via a secure network communication channel, the content item for insertion in the dialog data structure by the chatbot, the content item provided in the parameterized format selected via the content selection process to cause the computing device to perform the parametrically driven text to speech technique to generate an acoustic signal corresponding to the dialog data structure modified with the content item.

13. The method of claim 12 , comprising:
determining, based on a lookup in a data repository with an identifier of the computing device, that the computing device is authorized to access the computer program comprising the chatbot; and
selecting the computer program comprising the chatbot responsive to the determination that the computing device is authorized to access the chatbot.

14. The method of claim 12 , comprising:
using a machine learning image processing technique to identify the dialog data structure responsive to the digital file.

15. The method of claim 12 , wherein the dialog data structure comprises a tag identifying the placeholder field and metadata of the placeholder field.

16. The method of claim 12 , comprising:
using a natural language processing technique to process the dialog data structure and identify a portion of the dialog data structure at which to insert the placeholder field; and
inserting the placeholder field at the portion of the dialog data structure.

17. The method of claim 12 , comprising:
using a natu

Assignees

• Google Inc
• Google Llc

Inventors

• Eidem Laura
• Jacobson Alex

Classifications

• G10L13/08

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G06F40/211

  Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars · CPC title

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• H04L51/02 Primary

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• H04L67/306

  User profiles · CPC title