Skeletal tracking for real-time virtual effects

Patent metadata

Field	Value
Publication number	US-12531959-B2
Application number	US-202418668951-A
Country	US
Kind code	B2
Filing date	May 20, 2024
Priority date	Jun 30, 2020
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

Systems and methods are presented for capturing a video in real-time by an image capture device using a skeletal pose system. The skeletal pose system identifies first pose information in the video, applies a first virtual effect to the video in response to identifying the first pose information, identifies second pose information in the video, and applies a second virtual effect to the video in response to identifying the first pose information.

First claim

What is claimed is:

1 . A method comprising:
identifying first hand location information in a video while capturing the video, the first hand location information identified at a first distance from an image capture device; in response to identifying the first hand location information, applying a first virtual effect to the video in real time, the first virtual effect causing a visual modification to the video; identifying second hand location information in the video, the second hand location information identified at a second distance from the image capture device; and in response to identifying the second hand location information and applying the first virtual effect, applying a second virtual effect to the video while capturing the video, the second virtual effect modifying a granularity of the visual modification associated with the first virtual effect.

2 . The method of claim 1 , wherein the first hand location information represents a first plurality of joint locations of a human depicted in the video.

3 . The method of claim 1 , wherein the second hand location information represents a second plurality of joint locations of a human depicted in the video.

4 . The method of claim 1 , wherein the first virtual effect is a first augmented reality content item.

5 . The method of claim 4 , wherein the first augmented reality content item is applied to the video in real-time during capture.

6 . The method of claim 1 , wherein the second virtual effect is a second augmented reality content item.

7 . The method of claim 1 , further comprising:
storing the video comprising the first virtual effect at a first time and the second virtual effect at a second time; and transmitting the video as an ephemeral message to a computing device.

8 . The method of claim 1 , wherein the identifying the first hand location information further comprises:
identifying a hand in the video; tracking a motion of the hand from a first position to a second position; and modifying a level of granularity of the first virtual effect based on the tracked motion of the hand.

9 . A system comprising:
a processor; and a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising: capturing a video in real-time using an image capture device; identifying first hand location information in a video while capturing the video, the first hand location information identified at a first distance from the image capture device; in response to identifying the first hand location information, applying a first virtual effect to the video in real time, the first virtual effect causing a visual modification to the video; identifying second hand location information in the video, the second hand location information identified at a second distance from the image capture device; and in response to identifying the second hand location information and applying the first virtual effect, applying a second virtual effect to the video while capturing the video, the second virtual effect modifying a granularity of the visual modification associated with the first virtual effect.

10 . The system of claim 9 , wherein the first hand location information represents a first plurality of joint locations of a human depicted in the video.

11 . The system of claim 9 , wherein the second hand location information represents a second plurality of joint locations of a human depicted in the video.

12 . The system of claim 9 , wherein the first virtual effect is a first augmented reality content item.

13 . The system of claim 12 , wherein the first augmented reality content item is applied to the video in real-time during capture.

14 . The system of claim 9 , wherein the second virtual effect is a second augmented reality content item.

15 . The system of claim 9 , wherein the operations further comprise:
storing the video comprising the first virtual effect at a first time and the second virtual effect at a second time; and transmitting the video as an ephemeral message to a computing device.

16 . The system of claim 9 , wherein the identifying the first hand location information further comprises:
identifying a hand in the video; tracking a motion of the hand from a first position to a second position; and modifying a level of granularity of the first virtual effect based on the tracked motion of the hand.

17 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
capturing a video in real-time using an image capture device; identifying first hand location information in a video while capturing the video, the first hand location information identified at a first distance from the image capture device; in response to identifying the first hand location information, applying a first virtual effect to the video in real time, the first virtual effect causing a visual modification to the video; identifying second hand location information in the video, the second hand location information identified at a second distance from the image capture device; and in response to identifying the second hand location information and applying the first virtual effect, applying a second virtual effect to the video while capturing the video, the second virtual effect modifying a granularity of the visual modification associated with the first virtual effect.

18 . The computer-readable storage medium of claim 17 , wherein the first hand location information represents a first plurality of joint locations of a human depicted in the video.

19 . The computer-readable storage medium of claim 17 , wherein the second hand location information represents a second plurality of joint locations of a human depicted in the video.

20 . The computer-readable storage medium of claim 17 , wherein the first virtual effect is a first augmented reality content item.

Assignees

• Snap Inc

Inventors

• Rykhliuk Olha
• Solichin Jonathan
• Voronova Olesia
• Yagodin Artem

Classifications

• H04L51/18

  Commands or executable codes · CPC title

• H04L51/10 Primary

  Multimedia information · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title