Latency Reduction For Detached Content

Patent metadata

Field	Value
Publication number	US-2016357325-A1
Application number	US-201615174731-A
Country	US
Kind code	A1
Filing date	Jun 6, 2016
Priority date	Jun 7, 2015
Publication date	Dec 8, 2016
Grant date	—

Abstract

Systems, methods, and computer readable media to reduce latency in a graphics processing pipeline of an electronic device having a touch-sensitive input device and a display are described herein. In one embodiment, the method includes obtaining a temporal sequence of multi-touch scan data, processing the multi-touch scan data, determining an “optimal” time to deliver the processed multi-touch scan data to the relevant application(s) running on the device for processing, animating the resultant graphical changes that are determined by the application(s) based on the multi-touch input, and rendering the determined graphical changes to a frame buffer. In some embodiments, determining the “optimal” time for delivering the processed multi-touch scan data to the relevant application(s) comprises measuring the performance of the relevant application(s) in order to determine how likely it is that the necessary processing of the received multi-touch information may be completed before the conclusion of the next vertical blanking interval.

First claim

1 . A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more programmable control devices to:
obtain a temporal sequence of multi-touch scan data from a touch-sensitive input device of an electronic device having a display, wherein the temporal sequence of multi-touch scan data is obtained during a first vertical blanking interval (VBI) of the display; process the multi-touch scan data to determine at least time and location information associated with the multi-touch scan data; deliver the processed multi-touch scan data to one or more applications executing on the electronic device for application processing, wherein the delivery occurs at a determined optimal time; perform the application processing beginning at the determined optimal time; determine one or more graphical changes to be applied to the display in response to the performed application processing; render the determined one or more graphical changes to a frame buffer; and update the display with the contents of the frame buffer, wherein the determined optimal time comprises a determination of an approximation of an amount of time the one or more applications will need to perform the application processing.

2 . The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to obtain a temporal sequence of multi-touch scan data from a touch-sensitive input device of an electronic device having a display further comprise instructions to cause the one or more programmable control devices to obtain multiple multi-touch scan data samples during the first VBI of the display.

3 . The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more programmable control devices to perform the application processing beginning at the determined optimal time further comprise instructions to perform the application processing during a second VBI that occurs after that first VBI.

4 . The non-transitory program storage device of claim 3 , wherein the instructions to cause the one or more programmable control devices to determine one or more graphical changes to be applied in response to the performed application processing further comprise instructions to determine one or more graphical changes to be applied in response to the performed application processing during the second VBI.

5 . The non-transitory program storage device of claim 4 , wherein the instructions to cause the one or more programmable control devices to render the determined one or more graphical changes to a frame buffer further comprise instructions to render the determined one or more graphical changes to a frame buffer during the second VBI.

6 . The non-transitory program storage device of claim 1 , wherein the determined optimal time further comprises a determination of a delivery time wherein: the application processing; the determination of the one or more graphical changes; and the rendering of the one or more graphical changes may all occur during the first VBI.

7 . The non-transitory program storage device of claim 1 , wherein the determined optimal time further comprises a determination of a delivery time wherein: the application processing; the determination of the one or more graphical changes; and the rendering of the one or more graphical changes may all occur during a second VBI that occurs after that first VBI.

8 . The non-transitory program storage device of claim 1 , wherein the determined optimal time further comprises a time determined based, at least in part, on a determination of at least one of the following: a maximum application performance time; a maximum percentile of application performance times; a sliding window of application performance times; and output from a feedback control loop of application performance times.

9 . The non-transitory program storage device of claim 1 , wherein the determined approximation of an amount of time the one or more applications will need to perform the application processing further comprises an approximation based, at least in part, on gathered samples of past performance times of the one or more applications.

10 . A system comprising:
an touch-sensitive input device; a display; memory; and one or more programmable control devices operable to interact with the touch-sensitive input device and the memory, and to perform operations comprising:
obtaining a temporal sequence of multi-touch scan data from the touch-sensitive input device, wherein the temporal sequence of multi-touch scan data is obtained during a first vertical blanking interval (VBI) of the display;
processing the multi-touch scan data to determine at least time and location information associated with the multi-touch scan data;
delivering the processed multi-touch scan data to one or more applications for application processing, wherein the delivery occurs at a determined optimal time;
performing the application processing beginning at the determined optimal time;
determining one or more graphical changes to be applied to the display in response to the performed application processing;
rendering the determined one or more graphical changes to a frame buffer; and
updating the display with the contents of the frame buffer,
wherein the determined optimal time comprises a determination of an approximation of an amount of time the one or more applications will need to perform the application processing.

11 . The system of claim 10 , wherein the operations comprising obtaining a temporal sequence of multi-touch scan data from the touch-sensitive input device further comprise operations to obtain multiple multi-touch scan data samples during the first VBI of the display.

12 . The system of claim 10 , wherein the determined optimal time further comprises a determination of a delivery time wherein: the application processing; the determination of the one or more graphical changes; and the rendering of the one or more graphical changes may all occur during the first VBI.

13 . The system of claim 10 , wherein the determined optimal time further comprises a determination of a delivery time wherein: the application processing; the determination of the one or more graphical changes; and the rendering of the one or more graphical changes may all occur during a second VBI that occurs after that first VBI.

14 . The system of claim 10 , wherein the determined optimal time further comprises a time determined based, at least in part, on a determination of at least one of the following: a maximum application performance time; a maximum percentile of application performance times; a sliding window of application performance times; and output from a feedback control loop of application performance times.

15 . The system of claim 10 , wherein the determined approximation of an amount of time the one or more applications will need to perform the application processing further comprises an approximation based, at least in part, on gathered samples of past performance times of the one or more applications.

16 . A method comprising:
obtaining a temporal sequence of multi-touch scan data from a touch-sensitive input device of an electronic device having a display, wherein the temporal sequence of multi-touch scan data is obtained during a first vertical blanking interval (VBI) of the display; processing the multi-touch scan data to determine at least time and location information associated with the multi-touch scan data; delivering the processed multi-tou

Assignees

• Apple Inc

Inventors

• Beaver Jason C
• Nilo Bruce D
• Zhang Chendi

Classifications

• G06F2203/04104

  Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger · CPC title

• G06T1/60

  Memory management · CPC title

• G06F3/0418 Primary

  for error correction or compensation, e.g. based on parallax, calibration or alignment · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06F3/0416 Primary

  Control or interface arrangements specially adapted for digitisers · CPC title