Systems and methods for video event segmentation derived from simultaneously recorded sensor data

Patent metadata

Field	Value
Publication number	US-12067780-B2
Application number	US-202217683142-A
Country	US
Kind code	B2
Filing date	Feb 28, 2022
Priority date	Feb 28, 2022
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

In one embodiment, a method includes, by an electronic device, accessing activity data containing one or more non-image-based sensor signals from a first wearable device, where the activity data corresponds to an activity a user performs during a first timeframe, accessing from a first camera device, one or more cameras of the first camera device, where the video data corresponds to the first activity of the first user during the first timeframe, segmenting the activity data based on one or more features of the one or more non-image-based sensor signals to identify one or more segments of activity data corresponding to a second timeframe, classifying the one or more segments of the video data based on the one or more identified events associated with the first activity during the second timeframe, classifying the segments of the video data based on the one or more events during the second timeframe.

First claim

What is claimed is:

1. A method comprising, by an electronic device:
accessing, from a first wearable device on a first user, one or more non-image-based sensor signals from one or more sensors of the first wearable device, wherein the one or more non-image-based sensor signals correspond to a first timeframe;
identifying a first activity of the first user during the first timeframe based on a characteristic activity signature associated with the one or more non-image-based sensor signals over the first time period;
accessing, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;
segmenting the one or more non-image-based sensor signals corresponding to the first timeframe into a plurality of segments, wherein each segment corresponds to one or more second timeframes within the first timeframe, respectively;
automatically classifying each segment, based on (1) the identified first activity and (2) the one or more non-image-based signals of that segment, to identify one or more events within the first activity of the first user during the second timeframe corresponding to the respective segment; and
classifying one or more segments of the video data based on the identified one or more events, wherein each segment of video data corresponds to the one or more segments of the one or more non-image-based sensor signals, respectively, during the second timeframe corresponding to the respective segment of sensor signals.

2. The method of claim 1 , further comprising:
aligning the one or more non-image-based sensor signals over the first time period and the video data based on one or more of timestamps, audio elements, visual elements, or sensor signal elements.

3. The method of claim 1 , further comprising:
determining the first activity from a plurality of activities based on the one or more non-image-based sensor signals.

4. The method of claim 3 , wherein the first activity is further determined based on one or more of a global positioning system (GPS) location, a date, a time, a temperature, or a previous activity.

5. The method of claim 1 , wherein the first activity is selected from a plurality of hierarchically organized activities, wherein each of the plurality of activities is associated with a respective subclass of one or more actions or one or more events, and wherein the identified one or more events within the first activity of the first user are selected from a subclass of one or more actions associated with the first activity.

6. The method of claim 1 , further comprising:
receiving a user input specifying the first activity from a plurality of activities.

7. The method of claim 1 , further comprising:
providing instructions for presenting a user interface comprising the one or more segments of the video data, wherein the user interface comprises one or more activatable elements for filtering the one or more segments of the video data based on the one or more events associated with the respective segments of the video data.

8. An electronic device comprising:
one or more displays;
one or more sensors
one or more non-transitory computer-readable storage media including instructions; and
one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:
access, from a first wearable device on a first user, one or more non-image-based sensor signals from one or more sensors of the first wearable device, wherein the one or more non-image-based sensor signals correspond to a first timeframe;
identify a first activity of the first user during the first timeframe based on a characteristic activity signature associated with the one or more non-image-based sensor signals over the first time period
access, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;

segment the one or more non-image-based sensor signals corresponding to the first timeframe into a plurality of segments, wherein each segment corresponds to one or more second timeframes within the first timeframe, respectively;
automatically classify each segment based on (1) the identified first activity and (2) the one or more non-image-based signals of that segment, to identify one or more events within the first activity of the first user during the second timeframe corresponding to the respective segment; and
classify one or more segments of the video data based on the identified one or more events, wherein each segment of video data corresponds to the one or more segments of the one or more non-image-based sensor signals, respectively, during the second timeframe corresponding to the respective segment of sensor signals.

9. The electronic device of claim 8 , wherein the processors are further configured to execute instructions to:
align the one or more non-image-based sensor signals over the first time period and the video data based on one or more of timestamps, audio elements, visual elements, or sensor signal elements.

10. The electronic device of claim 8 , wherein the processors are further configured to execute instructions to:
determine the first activity from a plurality of activities based on the one or more non-image-based sensor signals.

11. The electronic device of claim 10 , wherein the first activity is further determined based on one or more of a global positioning system (GPS) location, a date, a time, a temperature, or a previous activity.

12. The electronic device of claim 8 , wherein the first activity is selected from a plurality of hierarchically organized activities, wherein each of the plurality of activities is associated with a respective subclass of one or more actions or one or more events, and wherein the identified one or more events within the first activity of the first user are selected from a subclass of one or more actions associated with the first activity.

13. The electronic device of claim 8 , wherein the processors are further configured to execute instructions to:
receive a user input specifying the first activity from a plurality of activities.

14. The electronic device of claim 8 , wherein the processors are further configured to execute instructions to:
provide instructions for presenting a user interface comprising the one or more segments of the video data, wherein the user interface comprises one or more activatable elements for filtering the one or more segments of the video data based on the one or more events associated with the respective segments of the video data.

15. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to:
access, from a first wearable device on a first user, one or more non-image-based sensor signals from one or more sensors of the first wearable device, wherein the one or more non-image-based sensor signals correspond to a first timeframe;
identify a first activity of the first user during the first timeframe based on a characteristic activity signature associated with the one or more non-image-based sensor signals over the first time period
access, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;
segment the one or more non-image-based sensor signals correspondin

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Hajash Kathleen Sofia
• Brenner Thomas
• Harms Brian
• Kim Catherine S
• Mohammed Imran

Classifications

• G06V20/44

  Event detection · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V20/49 Primary

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title