Capturing content on writing surfaces using depth and image sensing

Patent metadata

Field	Value
Publication number	US-11348264-B1
Application number	US-202016856779-A
Country	US
Kind code	B1
Filing date	Apr 23, 2020
Priority date	Apr 23, 2020
Publication date	May 31, 2022
Grant date	May 31, 2022

Abstract

A communication system captures writing surface content in a physical space for transmittal to remote client devices participating in a communication session. During a communication session with one or more remote client devices, a communication system captures image data and depth data describing objects in a physical space of the communication system. Based on the captured data, the communication system identifies a writing surface in the physical space and captures content on the writing surface. The communication system may also identify objects occluding content on the writing surface based on the captured data and may modify image data to make an object occluding the content at least partially transparent. The communication system transmits the content to at least one of the remote client devices participating in the communication session.

First claim

What is claimed is:

1. A method comprising:
establishing a communication session between a communication system and one or more remote client devices; and
during the communication session:
capturing, by a camera of the communication system, image data of a physical space;
capturing, by a depth sensor of the communication system, depth data associated with objects in the physical space;
processing the image data and the depth data to detect one or more flat surfaces within the physical space of the communication system;
determining whether a respective surface of the one or more flat surfaces meets one or more predefined criteria;
in response to determining that the respective surface of the one or more flat surfaces meets the one or more predefined criteria, capturing additional image data that includes content written on the respective surface;
storing identifying information of a region of the physical space corresponding to the respective surface, the identifying information including a plane of the respective surface and boundaries of the plane;
receiving, from the camera of the communication system, a current image frame;
detecting, based on depth information from the depth sensor for the current image frame, an object within the boundaries of the plane and having a depth between the camera and the plane;
processing the current image frame to render the object at least partially transparent; and
transmitting the content to the one or more remote client devices for display.

2. The method of claim 1 , wherein determining whether the respective surface meets the one or more predefined criteria comprises:
evaluating image data to identify one or more features of the respective surface;
comparing the one or more features of the respective surface to one or more predefined features of writing surfaces; and
determining, based on the comparing, whether the one or more features of the respective surface substantially match the one or more predefined features of writing surfaces.

3. The method of claim 1 , wherein processing the image frame to render the object at least partially transparent comprises:
accessing one or more prior image frames that are captured before detecting the object;
identifying, from the one or more prior image frames, pixels corresponding to a region of the respective surface occluded by the object in the current image frame; and
replacing the pixels in the current image frame corresponding to the object with the pixels from the one or more prior image frames corresponding to the region of the respective surface occluded by the object.

4. The method of claim 1 , wherein processing the image frame to render the object at least partially transparent comprises:
accessing one or more prior image frames that are captured before detecting the object;
identifying, from the one or more prior image frames, pixels corresponding to a region of the respective surface occluded by the object in the current image frame; and
blending the pixels in the current image frame corresponding to the object with the pixels from the one or more prior image frames corresponding to the region of the respective surface occluded by the object.

5. The method of claim 1 , wherein transmitting the captured content further comprises:
generating a document including the captured content; and
transmitting the generated document to the one or more remote client devices.

6. The method of claim 1 , further comprising:
identifying a symbol in the captured content on the respective surface, the symbol corresponding to an action used to modify the captured content; and
performing the action.

7. The method of claim 6 , wherein the action is one or more of: bolding, highlighting, underlining, italicizing, or otherwise emphasizing at least a portion of the captured content.

8. The method of claim 6 , wherein the action is one or more of: tagging or notifying a user or transmitting content to a user.

9. The method of claim 1 , wherein the surface of interest is one of: a whiteboard, a blackboard, a canvas, a notepad, a notebook, and transparent or semi-transparent glass being used as a writable surface.

10. The method of claim 1 , wherein transmitting the captured content comprises transmitting video data including at least a portion of the respective surface to the at least one remote client device.

11. A non-transitory computer-readable storage medium, storing one or more programs configured for execution by one or more processors of a communication device with a camera and a depth sensor, the one or more programs including instructions for:
establishing a communication session between the communication system and one or more remote client devices; and
during the communication session:
capturing, by the camera of the communication system, image data of a physical space;
capturing, by the depth sensor of the communication system, depth data associated with objects in the physical space;
processing the image data and the depth data to detect one or more flat surfaces within the physical space of the communication system;
determining whether a respective surface of the one or more flat surfaces meets one or more predefined criteria;
in response to determining that the respective surface of the one or more flat surfaces meets the one or more predefined criteria, capturing additional image data that includes content written on the respective surface;
storing identifying information of a region of the physical space corresponding to the writing surface, the identifying information including a plane of the writing surface and boundaries of the plane;
receiving, from the camera of the communication system, a current image frame;
detecting, based on depth information from the depth sensor for the current image frame, an object within the boundaries of the plane and having a depth between the camera and the plane; and
processing the current image frame to render the object at least partially transparent; and
transmitting the content to the one or more remote client devices for display.

12. The computer-readable storage medium of claim 11 , wherein determining that the surface meets the one or more predefined criteria comprises:
evaluating image data to identify one or more features of the respective surface;
comparing the one or more features of the respective surface to one or more predefined features of writing surfaces; and
determining, based on the comparing, whether the one or more features of the respective surface substantially match the one or more predefined features of writing surfaces.

13. The computer-readable storage medium of claim 11 , wherein processing the image frame to render the object at least partially transparent comprises:
accessing one or more prior image frames, the prior image frame captured before detecting the object;
identifying, from the one or more prior image frames, pixels corresponding to a region of the writing surface occluded by the object in the current image frame; and
replacing the pixels in the current image frame corresponding to the object with the pixels from the one or more prior image frames corresponding to the region of the writing surface occluded by the object.

14. The computer-readable storage medium of claim 11 , wherein processing the image frame to render the object at least partially transparent comprises:
accessing one or more prior image frames, the prior image frame captured before detecting the object;
identifying, from the one or more prior image frames, pixels corresponding to a region of the writing surface occluded by the object in the curre

Assignees

• Meta Platforms Inc

Inventors

• Back Tyler
• Molina Gabriel

Classifications

• G06V30/36 Primary

  Matching; Classification · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• H04N1/00

  Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof · CPC title