Gesture-based whiteboard

Patent metadata

Field	Value
Publication number	US-11736661-B2
Application number	US-202217568549-A
Country	US
Kind code	B2
Filing date	Jan 4, 2022
Priority date	Jan 4, 2021
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

A video capturing a user facing a camera is received. An image of the video is analyzed to identify a marking object in the image. The image captures the marking object held by the user in between the user and the camera. A gesture performed by the user is detected using the marking object in a space between the user and the camera. A location associated with the marking object is marked including by overlaying a highlight of a portion of content virtually shared between the user and one or more other remote participants. The overlaid highlight of the portion of the content is provided to one or more other remote participants via a collaboration medium.

First claim

What is claimed is:

1. A method, comprising:
receiving a video capturing a user;
analyzing an image of the video to identify a marking object in the image, wherein the image captures the marking object held by the user;
detecting a marking gesture performed by the user using the marking object;
marking a location associated with the marking object by overlaying a mark of a portion of content shared between the user and one or more remote participants;
providing to one or more other remote participants the overlaid mark of the portion of the shared content;
detecting a stop indicator gesture performed by the user while the marking object is detected in the marking gesture; and
in response to detecting the stop indicator gesture, preventing a subsequent movement of the marking object from being recognized as marking on the shared content.

2. The method of claim 1 , wherein the marking gesture is performed by the user using the marking object in a space between the user and a camera while the user is facing the camera.

3. The method of claim 1 , further comprising:
analyzing the video to identify a start indicator gesture prior to identifying the marking object in the image.

4. The method of claim 3 , wherein analyzing the video to identify the start indicator gesture prior includes identifying an indicator object that is different from the marking object.

5. The method of claim 1 , wherein the marking gesture is performed by the user using a first hand and the stop indicator gesture is performed by the user using a second hand different from the first hand.

6. The method of claim 1 , wherein the stop indicator gesture corresponds to a fist gesture or an open palm gesture.

7. The method of claim 1 , wherein marking the location associated with the marking object includes:
detecting movement of a camera capturing the video of the user; and
adjusting the image of the video based on the detected movement of the camera.

8. The method of claim 1 , wherein marking the location associated with the marking object includes:
detecting a background of the image of the video; and
adjusting the image of the video by masking the background of the image of the video.

9. The method of claim 1 , wherein marking the location associated with the marking object includes removing noise in the image of the video or optimized lighting conditions of the image of the video.

10. The method of claim 1 , wherein providing the overlaid mark of the portion of the shared content includes:
creating an update to reflect the marked location; and
transmitting the update to a cloud service provider, wherein the cloud service provider is configured to distribute the update to the one or more remote participants and create a collaboration record based on the update.

11. The method of claim 1 , wherein providing the overlaid mark of the portion of the shared content includes determining an edit state of a virtual shared whiteboard, and wherein the edit state of the virtual shared whiteboard includes at least a start state and a pause state.

12. The method of claim 1 , wherein a property of the marking object is determined by identifying a color value associated with the marking object.

13. The method of claim 1 , wherein a property of the marking object is determined by identifying a whiteboard action associated with the marking object, wherein the whiteboard actions is selected from options including the following: an erasing action, a highlighting action, or a selection action.

14. The method of claim 1 , wherein a property of the marking object is determined by identifying a size or shape associated with the marking object.

15. A system, comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory is configured to provide the one or more processors with instructions which when executed cause the one or more processors to:
receive a video capturing a user;
analyze an image of the video to identify a marking object in the image, wherein the image captures the marking object held by the user;
detect a marking gesture performed by the user using the marking object;
mark a location associated with the marking object by being configured to overlay a mark of a portion of content shared between the user and one or more remote participants;
provide to one or more remote participants the overlaid mark highlight of the portion of the shared content;
detect a stop indicator gesture performed by the user while the marking object is detected in the marking gesture; and
in response to detecting the stop indicator gesture, prevent a subsequent movement of the marking object from being recognized as marking on the shared content.

16. The system of claim 15 , wherein a property of the marking object is determined by identifying a color associated with the marking object.

17. The system of claim 15 , wherein a property of the marking object is determined by identifying a size or shape associated with the marking object.

18. The system of claim 15 , wherein the memory further is configured to provide the one or more processors with further instructions which when executed cause the one or more processors to: analyze the video to identify a start indicator gesture prior to identifying the marking object in the image.

19. The system of claim 18 , wherein the memory further is configured to provide the one or more processors with further instructions which when executed cause the one or more processors to: analyze the video to identify the stop indicator gesture.

20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a video capturing a user;
analyzing an image of the video to identify a marking object in the image, wherein the image captures the marking object held by the user;
detecting a marking gesture performed by the user using the marking object;
marking a location associated with the marking object by overlaying a mark of a portion of content virtually shared between the user and one or more remote participants;
providing to one or more remote participants the overlaid mark of the portion of the shared content;
detecting a stop indicator gesture performed by the user while the marking object is detected in the marking gesture; and
in response to detecting the stop indicator gesture, preventing a subsequent movement of the marking object from being recognized as marking on the shared content.

Assignees

• Servicenow Inc

Inventors

• Gandhi Bhairavi
• Upadhya Prashant
• Pal Soumen
• Dey Avishek

Classifications

• H04N7/15 Primary

  Conference systems · CPC title

• G06F3/1454

  involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title