Detection of potential exfiltration of audio data from digital assistant applications

Patent metadata

Field	Value
Publication number	US-10565985-B1
Application number	US-201816131439-A
Country	US
Kind code	B1
Filing date	Sep 14, 2018
Priority date	Sep 14, 2018
Publication date	Feb 18, 2020
Grant date	Feb 18, 2020

Abstract

The present disclosure is generally related to a data processing system to detect potential exfiltration of audio data by agent applications can include a data processing system. The data processing system can identify, from an I/O record, an input received from the digital assistant application via a microphone of a client device, an output received from the agent application after the input, and a microphone status for the microphone. The data processing system can determine that the output is terminal based on the input and the output. The data processing system can identify the microphone status as in the enabled state subsequent to the input. The data processing system can determine that the agent application is unauthorized to access audio data acquired via the microphone of the client device based on determining that the output is terminal and identifying the microphone status as enabled.

First claim

What is claimed:

1. A system to detect potential exfiltration of audio data by agent applications that interface with digital assistant applications, comprising:
a data processing system having an exchange logger component, a sequence analyzer component, a microphone use detector component, an extrusion detector component, and an agent manager component;
the exchange logger component to access a log database that maintains an input/output (I/O) record between an agent application that executes on a client device and a digital assistant application that executes on the client device;
the exchange logger component to identify, from the I/O record of the log database, an input received from the digital assistant application via a microphone of the client device, an output received from the agent application that interfaces with the digital assistant application subsequent to the input, and a microphone status that indicates one of an enabled state and a disabled state of the microphone subsequent to the input;
the sequence analyzer component to determine that the output is terminal to the input based on a first set of strings included in the input and a second set of strings included in the output;
the microphone use detector component to identify the microphone status of the microphone as in the enabled state subsequent to the input;
the extrusion detector component to determine that the agent application is unauthorized to access audio data acquired via the microphone of the client device based on the determination that the output is terminal to the input and the identification of the microphone status as in the enabled state; and
the agent manager component to perform an action on at least one of the agent application and the digital assistant application, responsive to the determination that the agent application is unauthorized to access the audio data acquired via the microphone.

2. The system of claim 1 , comprising:
the sequence analyzer component to:
identify, from the second set of strings included in the output, a subset of strings from at least one of a starting index and an end index;
compare the subset of strings to a plurality of predefined keywords that correlate to one of terminal responses or anticipatory responses; and
determine that the output is terminal based on the comparison of the subset of strings to the plurality of predefined keywords.

3. The system of claim 1 , comprising:
the sequence analyzer component to:
compare the first set of strings included in the input to a plurality of predefined keywords that correlate to expectative queries that indicate a subsequent response is to be anticipatory; and
determine that the input is expectative based on the comparison of the first set of strings to the plurality of predefined keywords; and

the extrusion detector component to determine that the agent application is unauthorized to access audio data acquired via the microphone of the client device based on the determination that the output is terminal and the input is expectative.

4. The system of claim 1 , comprising:
the sequence analyzer component to:
establish an I/O exchange model to correlate sets of strings communicated between the agent application and the digital assistant application with terminal responses;
apply, to the I/O exchange model, the first set of strings included in the input and the second set of strings included in the output to obtain a confidence metric that indicates a likelihood that the output is terminal; and
determine, responsive to a determination that the confidence metric is greater than a threshold, that the output is terminal.

5. The system of claim 1 , comprising:
the exchange logger component to identify, from the I/O record of the log database, a second input received from the digital assistant application via the microphone of the client device subsequent to the output from the agent application, the second input including a null marker indicating a lack of strings parsed from an audio input signal corresponding to the second input;
the sequence analyzer to identify the null marker from the second input received from the digital assistant application;
the microphone use detector component to identify the microphone status of the microphone as in the enabled state subsequent to the output determined to be the terminal response; and
the extrusion detector component to determine that the agent application is unauthorized to access audio data acquired via the microphone of the client device based on the identification of the null marker in the second response and the identification of the microphone status as in the enabled state subsequent to the output determined to be the terminal response.

6. The system of claim 1 , comprising:
the exchange logger component to identify, from the I/O record of the log database, a plurality of inputs received from the digital assistant application via the microphone of the client device, a plurality of output received from the agent application that interfaces with the digital assistant application,
the sequence analyzer component to count, from the I/O record, a number of occurrences of a first input of the plurality of inputs directly followed by a second input of the plurality of inputs without an intermediate output of the plurality of outputs; and
the extrusion detector component to determine, based on a determination that the number of occurrences of the first input directly followed by the second input without the intermediate output is greater than a threshold number, that the agent application is unauthorized to access audio data acquired via the microphone of the client device.

7. The system of claim 1 , comprising:
the exchange logger component to identify, from the I/O record of the log database, the microphone status that indicates one of the enabled state and the disabled state of the microphone subsequent to the output for a specified time duration;
the microphone use detector component to identify the specified time duration of the microphone status in the enabled state subsequent to the output; and
the extrusion detector component to determine that the agent application is unauthorized to access the audio data acquired via the microphone of the client device based on the determination that the specified time duration of microphone status in the enabled state is greater than the threshold limit.

8. The system of claim 1 , comprising:
the exchange logger component to identify, from the I/O record of the log database, a device modality of the client device that executes the agent application and the digital assistant application; and
the sequence analyzer component to determine that the output is terminal to the input based on the device modality of the client device.

9. The system of claim 1 , comprising:
the agent manager component to perform, responsive to the determination that the agent application is unauthorized to access the audio data acquired via the microphone, the action to cause the digital assistant application to present an alert that indicates that the agent application is unauthorized to access the audio data.

10. The system of claim 1 , comprising:
the agent manager component to perform, responsive to the determination that the agent application is unauthorized to access the audio data acquired via the microphone, the action to send a notification to an agent service associated with the agent application.

11. The system of claim 1 , comprising:
the agent manager component to perform, responsive to the determination that the agent application is unauthorized to access the audio data acquired via the microphone, the action to cause t

Assignees

• Google Llc

Inventors

• Huang Yan
• Rao Nikhil

Classifications

• G06F21/6245

  Protecting personal data, e.g. for financial or medical purposes · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G06F21/554

  involving event detection and direct action · CPC title

• G06F21/556

  involving covert channels, i.e. data leakage between processes (inhibiting the analysis of circuitry or operation with measures against power attack G06F21/755) · CPC title

• G06F21/60

  Protecting data · CPC title