Selecting image sensor for object classification

Patent metadata

Field	Value
Publication number	US-2026073680-A1
Application number	US-202418827175-A
Country	US
Kind code	A1
Filing date	Sep 6, 2024
Priority date	Sep 6, 2024
Publication date	Mar 12, 2026
Grant date	—

Abstract

A head-worn device system includes multiple image sensors (e.g., cameras), one or more display devices and one or more processors. The system also includes a memory storing instructions that, when executed by the one or more processors, configure the system to obtain a first image captured by a first image sensor of the device; generate, based on obtaining the first image, a respective predicted skeleton corresponding to a respective view from each of the first image sensor and the one or more second image sensors; select, based on generating the respective predicted skeletons, an image sensor from among the first image sensor and the one or more second image sensors, the selected image sensor being used for classifying the first image; and determine, based on the selected image sensor, a classification for the first image.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
obtaining a first image captured by a first image sensor of a device, the device including the first image sensor and one or more second image sensors; generating, based on obtaining the first image, a respective predicted skeleton corresponding to a respective view from each of the first image sensor and the one or more second image sensors; selecting, based on generating the respective predicted skeletons, an image sensor from among the first image sensor and the one or more second image sensors, the selected image sensor being used for classifying the first image; and determining, based on the selected image sensor, a classification for the first image.

2 . The computer-implemented method of claim 1 , wherein the first image corresponds to an object, and
wherein the classification corresponds to an identification of the object or a position of the object.

3 . The computer-implemented method of claim 2 , wherein the object is a hand, and
wherein the classification corresponds to a hand gesture.

4 . The computer-implemented method of claim 1 , wherein selecting the image sensor comprises:
determining a pinch plane for each of the respective predicted skeletons; calculating a dot product value for each of the pinch planes, such that each of the first image sensor and the one or more second image sensors has a dot product value associated therewith; and selecting, based on calculating the dot product value and from among the first image sensor and the one or more second image sensors, the image sensor having a lowest dot product value associated therewith.

5 . The computer-implemented method of claim 1 , wherein selecting the image sensor comprises:
determining, using a first neural network, an occlusion value for each of the respective predicted skeletons, such that each of the first image sensor and the one or more second image sensors has an occlusion value associated therewith; and selecting, based on determining the occlusion value and from among the first image sensor and the one or more second image sensors, the image sensor having a lowest occlusion value associated therewith.

6 . The computer-implemented method of claim 5 , wherein generating the respective predicted skeletons uses a second neural network which is separate from the first neural network.

7 . The computer-implemented method of claim 6 , wherein determining the classification uses a third neural network which is separate from the first neural network and from the second neural network.

8 . The computer-implemented method of claim 1 , further comprising:
obtaining a second image captured by the selected image sensor, wherein determining the classification for the first image is based on the second image.

9 . A system comprising:
at least one processor; at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: obtaining a first image captured by a first image sensor of a device, the device including the first image sensor and one or more second image sensors; generating, based on obtaining the first image, a respective predicted skeleton corresponding to a respective view from each of the first image sensor and the one or more second image sensors; selecting, based on generating the respective predicted skeletons, an image sensor from among the first image sensor and the one or more second image sensors, the selected image sensor being used for classifying the first image; and determining, based on the selected image sensor, a classification for the first image.

10 . The system of claim 9 , wherein the first image corresponds to an object, and
wherein the classification corresponds to an identification of the object or a position of the object.

11 . The system of claim 10 , wherein the object is a hand, and
wherein the classification corresponds to a hand gesture.

12 . The system of claim 9 , wherein selecting the image sensor comprises:
determining a pinch plane for each of the respective predicted skeletons; calculating a dot product value for each of the pinch planes, such that each of the first image sensor and the one or more second image sensors has a dot product value associated therewith; and selecting, based on calculating the dot product value and from among the first image sensor and the one or more second image sensors, the image sensor having a lowest dot product value associated therewith.

13 . The system of claim 9 , wherein selecting the image sensor comprises:
determining, using a first neural network, an occlusion value for each of the respective predicted skeletons, such that each of the first image sensor and the one or more second image sensors has an occlusion value associated therewith; and selecting, based on determining the occlusion value and from among the first image sensor and the one or more second image sensors, the image sensor having a lowest occlusion value associated therewith.

14 . The system of claim 13 , wherein generating the respective predicted skeletons uses a second neural network which is separate from the first neural network.

15 . The system of claim 14 , wherein determining the classification uses a third neural network which is separate from the first neural network and from the second neural network.

16 . The system of claim 9 , the operations further comprising:
obtaining a second image captured by the selected image sensor, wherein determining the classification for the first image is based on the second image.

17 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
obtaining a first image captured by a first image sensor of a device, the device including the first image sensor and one or more second image sensors; generating, based on obtaining the first image, a respective predicted skeleton corresponding to a respective view from each of the first image sensor and the one or more second image sensors; selecting, based on generating the respective predicted skeletons, an image sensor from among the first image sensor and the one or more second image sensors, the selected image sensor being used for classifying the first image; and determining, based on the selected image sensor, a classification for the first image.

18 . The non-transitory computer-readable storage medium of claim 17 , wherein the first image corresponds to an object, and
wherein the classification corresponds to an identification of the object or a position of the object.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein the object is a hand, and
wherein the classification corresponds to a hand gesture.

20 . The non-transitory computer-readable storage medium of claim 17 , wherein selecting the image sensor comprises:
determining a pinch plane for each of the respective predicted skeletons; calculating a dot product value for each of the pinch planes, such that each of the first image sensor and the one or more second image sensors has a dot product value associated therewith; and selecting, based on calculating the dot product value and from among the first image sensor and the one or more second image sensors, the image sensor having a lowest dot product value associated therewith.

Assignees

• Snap Inc

Inventors

• Alvarez Attila
• Faeulhammer Thomas
• Nemeth Gabor
• Retek David

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V10/10

  Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) · CPC title