Systems and methods for gesture handling in a three-dimensional visual scene

Patent metadata

Field	Value
Publication number	US-10459526-B2
Application number	US-201715648031-A
Country	US
Kind code	B2
Filing date	Jul 12, 2017
Priority date	Jul 12, 2017
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

In one embodiment, a method includes displaying, on a client system, a visual scene with one or more first objects and one or more second objects. The one or more second objects are associated with an augmented reality context. A first touch event handler, associated with an operating system running on the client system, may receive a set of touch events and send the set of touch events to a second touch event handler running on the client system. The second touch event handler may detect a first subset of touch events relating to the one or more second objects. The second touch event handler may process the first subset of touch events and send a second subset of touch events relating to the one or more first objects to the first touch event handler. The first touch event handler may process the second subset of touch events.

First claim

What is claimed is:

1. A method comprising:
displaying, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;
receiving, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;
sending, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;
detecting, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;
processing, by the second touch event handler, the first subset of touch events relating to the one or more second objects;
sending, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;
processing, by the first touch event handler, the second subset of touch events relating to the one or more first objects;
sending, to the client system associated with the user, instructions for updating the visual scene in response to the processing of the first subset of touch events or the processing of the second subset of touch events; and
updating, on the client system associated with the user, the visual scene for display in response to receiving the instructions.

2. The method of claim 1 , wherein processing of the second subset of touch events relating to the one or more first objects by the first touch event handler is done synchronously.

3. The method of claim 2 , wherein the first touch event handler is an OS touch event handler.

4. The method of claim 1 , wherein processing of the first subset of touch events relating to the one or more second objects by the second touch event handler is done asynchronously.

5. The method of claim 4 , wherein the second touch event handler is associated with an application configured to generate the AR context.

6. The method of claim 1 , wherein the detecting of the first subset of touch events relating to the one or more second objects is performed using a ray casting technique.

7. The method of claim 1 , wherein the one or more second objects are three-dimensional (3D) objects that are overlaid on top of a real-world scene captured by a camera of the client system.

8. The method of claim 1 , wherein the one or more first objects are system-level user interface objects.

9. The method of claim 1 , wherein the set of touch events comprises one or more of a tap gesture, a drag gesture, a touch-up gesture to scroll upwards, a touch-down gesture to scroll downwards, a pinch-to-zoom gesture, or a swipe gesture.

10. The method of claim 1 , wherein the sending of the set of touch events to the second touch event handler comprises:
injecting, by the first touch event handler, the set of touch events and information associated with each touch event in a data structure.

11. The method of claim 10 , wherein the information associated with a touch event in the data structure comprises position or location coordinates where the touch event is performed and type of gesture associated with the touch event.

12. The method of claim 10 , wherein the sending of the second subset of touch events to the first touch event handler comprises:
re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler into the data structure.

13. The method of claim 12 , wherein re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler comprises:
re-creating the second subset of touch events for processing by the first touch event handler.

14. The method of claim 13 , wherein re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler comprises: determining to re-create the second subset of touch events based on a type of the OS running on the client system.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
display, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;
receive, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;
send, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;
detect, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;
process, by the second touch event handler, the first subset of touch events relating to the one or more second objects;
send, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;
process, by the first touch event handler, the second subset of touch events relating to the one or more first objects;
send, to the client system associated with the user, instructions for updating the visual scene in response to the processing of the first subset of touch events or the processing of the second subset of touch events; and
update, on the client system associated with the user, the visual scene for display in response to receiving the instructions.

16. The media of claim 15 , wherein the first touch event handler is an OS touch event handler that is configured to process the second subset of touch events synchronously.

17. The media of claim 15 , wherein the second touch event handler is associated with an application configured to generate the AR context, and wherein the second touch event handler is configured to process the first subset of touch events asynchronously.

18. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
display, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;
receive, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;
send, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;
detect, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;
process, by the second touch event handler, the first subset of the touch events relating to the one or more second objects;
send, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;
process, by the first touch event handler, the second subset of touch events relating to the one or more first objects;
send, to the client system associated with the user, instructions fo

Assignees

• Facebook Inc

Inventors

• Gontovnik Danil
• Ng Yu Hang
• Hanchar Siarhei
• Slater Michael
• Anpilov Sergei Viktorovich

Classifications

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/00

  Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title