Detecting object proximity using touch sensitive surface sensing and ultrasonic sensing

Patent metadata

Field	Value
Publication number	US-11231815-B2
Application number	US-201916457293-A
Country	US
Kind code	B2
Filing date	Jun 28, 2019
Priority date	Jun 28, 2019
Publication date	Jan 25, 2022
Grant date	Jan 25, 2022

Abstract

Techniques enabling improved classification of touch or hover interactions of objects with a touch sensitive surface of a device are presented. A speaker of the device can emit an ultrasonic audio signal comprising a first frequency distribution. A microphone of the device can detect a reflected audio signal comprising a second frequency distribution. The audio signal can be reflected off of an object in proximity to the surface to produce the reflected audio signal. A classification component can determine movement status of the object, or classify the touch or hover interaction, in relation to the surface, based on analysis of the signals. The classification component also can classify the touch or hover interaction based on such ultrasound data and/or touch surface or other sensor data. The classification component can be trained, using machine learning, to perform classifications of touch or hover interactions of objects with the surface.

First claim

What is claimed is:

1. A method, comprising:
emitting, by a system comprising a processor, an emitted audio signal, comprising at least a first frequency, from an electronic device, wherein the electronic device comprises a touch sensitive screen;
detecting, by the system, a reflected audio signal comprising at least a second frequency, wherein the emitted audio signal is reflected off of an object in proximity to the touch sensitive screen to produce the reflected audio signal;
determining, by the system, a movement status of the object in relation to the touch sensitive screen, based at least in part on a frequency shift between the first frequency of the emitted audio signal and the second frequency of the reflected audio signal;
obtaining sensor data, from surface-related sensors of the touch sensitive screen, in response to the object being in contact or in proximity with one or more portions of the touch sensitive screen; and
classifying which type of object is interacting with the touch sensitive screen based on the movement status, the emitted audio signal, the reflected audio signal, and the sensor data.

2. The method of claim 1 , wherein classifying further comprises determining a location of the object with respect to the touch sensitive screen.

3. The method of claim 1 , wherein classifying which type of object comprises distinguishing whether the object is an ear of a user in contact or proximity with the touch sensitive screen, a finger of the user in contact or proximity with the touch sensitive screen, a stylus in contact or proximity with the touch sensitive screen, or no object in contact or proximity with the touch sensitive screen.

4. The method of claim 1 , further comprising:
receiving additional sensor data from an accelerometer, a gyroscope, or an inertial measurement unit of the electronic device,

and
wherein classifying is further based on the additional sensor data.

5. The method of claim 1 , further comprising:
determining, by the system, a responsive action to be performed by the electronic device in response to classifying; and
performing, by the system, the responsive action.

6. The method of claim 5 , wherein classifying further comprises determining whether an interaction of the object with the touch sensitive screen is an intentional touch interaction or an unintentional interaction, and wherein the determining the responsive action comprises:
in response to determining the interaction of the object with the touch sensitive screen is an intentional touch interaction, determining the responsive action, comprising at least one of processing the interaction of the object with the touch sensitive screen, transitioning the touch sensitive screen to or maintaining the touch sensitive screen in an on state, transitioning the touch sensitive screen to or maintaining the touch sensitive screen in an unlocked state, or performing, by the electronic device, an operation based at least in part on the interaction of the object with the touch sensitive screen; or
in response to determining the interaction of the object with the touch sensitive screen is an unintentional touch interaction or an unintentional hover interaction, determining the responsive action, comprising at least one of rejecting the interaction of the object with the touch sensitive screen, transitioning the touch sensitive screen to or maintaining the touch sensitive screen in an off state, transitioning the touch sensitive screen to or maintaining the touch sensitive screen in a locked state, or preventing the operation from being performed in response to the interaction of the object with the touch sensitive screen.

7. The method of claim 6 , wherein classifying further comprises determining whether the interaction is an ear touch interaction of an ear of a user in contact with the touch sensitive screen, an ear hover interaction of the ear of the user in proximity to the touch sensitive screen, a finger touch interaction of a finger of the user in contact with the touch sensitive screen, a finger hover interaction of the finger of the user in proximity to the touch sensitive screen, a stylus touch interaction of a stylus in contact with the touch sensitive screen, a stylus hover interaction of the stylus in proximity to the touch sensitive screen, or no interaction of the object with the touch sensitive screen.

8. The method of claim 1 , wherein the emitting the emitted audio signal comprises emitting the emitted audio signal comprising a first distribution of frequencies, wherein the first distribution of frequencies comprises the first frequency,
wherein the detecting the reflected audio signal comprises detecting the reflected audio signal comprising a second distribution of frequencies, wherein the second distribution of frequencies comprises the second frequency, and
wherein the determining the movement status of the object in relation to the touch sensitive screen comprises determining the movement status of the object in relation to the touch sensitive screen based at least in part on a third distribution of the frequency shift between the first distribution of frequencies of the emitted audio signal and the second distribution of frequencies of the reflected audio signal.

9. The method of claim 1 , further comprising:
training, by the system, a machine learning component of the electronic device to classify touch interactions and hover interactions of a plurality of objects with the touch sensitive screen, based at least in part on training data relating to the touch interactions and the hover interactions of the objects with the touch sensitive screen, wherein the objects comprise the object,
wherein the classifying further comprises classifying an interaction state of the object with respect to the touch sensitive screen based at least in part on the training of the machine learning component and sensor data relating to an interaction of the object with the touch sensitive screen, and
wherein the sensor data comprises at least one of the reflected audio signal or the emitted audio signal.

10. The method of claim 9 , wherein the training data comprises a set of example items of training information, and wherein the method further comprises:
applying, by the system, the set of example items of training information to the machine learning component to facilitate the training of the machine learning component, wherein the set of example items of training information indicates respective relationships between respective touch or hover interactions and at least one of respective ultrasonic sensor data of an ultrasonic sensor, respective touch sensor data of a touch sensor, respective accelerometer data of an accelerometer, respective gyroscope data of a gyroscope, or respective inertial measurement unit data of an inertial measurement unit.

11. A system, comprising:
at least one speaker configured to emit an emitted ultrasonic audio signal, comprising at least a first frequency, from an electronic device, wherein the electronic device comprises a touch sensitive screen;
at least one audio sensor configured to sense a reflected ultrasonic audio signal comprising at least a second frequency, wherein the emitted ultrasonic audio signal is reflected off of an object in proximity to the touch sensitive screen to produce the reflected ultrasonic audio signal;
a plurality of surface-related sensors for obtaining sensor data in response to the object being in contact or in proximity with one or more portions of the touch sensitive screen;
a memory that stores executable components;
a processor, operatively coupled to the memory, for executing the executable components, the executable

Assignees

• Qeexo Co

Inventors

• Munemoto Taihei
• Levine William Isaac

Classifications

• G06F3/04186

  Touch location disambiguation · CPC title

• G01S15/586

  using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets · CPC title

• G01S15/523

  for presence detection (burglar, theft or intruder alarms G08B13/00, e.g. G08B13/16) · CPC title

• G06F2203/04106

  Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection · CPC title

• G06F3/0416

  Control or interface arrangements specially adapted for digitisers · CPC title