Hardware encoder for stereo stitching

Patent metadata

Field	Value
Publication number	US-12294688-B2
Application number	US-202217716786-A
Country	US
Kind code	B2
Filing date	Apr 8, 2022
Priority date	May 19, 2021
Publication date	May 6, 2025
Grant date	May 6, 2025

Abstract

A system to perform stereo stitching of image frames comprises a stereo camera system and a hardware encoder. The hardware encoder receives a left image stream and a right image stream from the stereo camera system simultaneously and processes the left and right image stream to generate a single stitched encoded frame. The apparatus can also comprise a processor and a memory having instructions stored thereon, when executed by the processor, causes the processor to perform operations comprising receiving the left image stream and the right image stream, processing the left and right image streams and generating a single stitched encoded frame.

First claim

What is claimed is:

1. A system comprising:
a first camera lens comprising a first image sensor;
a second camera lens comprising a second image sensor; and
a singular hardware encoder coupled to the first camera lens and the second camera lens, the singular hardware encoder to perform operations comprising:
receiving, by the singular hardware encoder, a single buffer comprising first image data from the first camera lens and second image data from the second camera lens, the single buffer received for a single activation of stitched frame encoding;
processing, by the singular hardware encoder, the first image data and the second image data in the single buffer; and
generating, by the singular hardware encoder, a single stitched encoded frame based on the processed first image data and the processed second image data, wherein each frame in the single stitched encoded frame comprises of a corresponding first image frame from the processed first image data and a corresponding second image from the processed second image data, the single stitched encoded frame generated in response to the single activation.

2. The system of claim 1 , wherein the first camera lens and the second camera lens are part of a stereo camera system.

3. The system of claim 1 , wherein the singular hardware encoder receives input from multiple locations in memory.

4. The system of claim 3 , wherein the singular hardware encoder reads the received input simultaneously and outputs the single buffer.

5. The system of claim 1 , wherein the first image data comprises a first set of image frames.

6. The system of claim 1 , wherein the second image data comprises a second set of image frames.

7. The system of claim 3 , wherein the processing further comprises:
synchronizing a first exposure time of the first image sensor with a second exposure time of the second image sensor.

8. The system of claim 1 , wherein the operations further comprise:
transmitting the single stitched encoded frame to a client device.

9. The system of claim 8 , wherein the client device is associated with one of a user's contacts.

10. An apparatus comprising:
a singular hardware encoder to perform operations comprising:
receiving, by the singular hardware encoder, a single buffer comprising first image data from a first camera lens and second image data from a second camera lens, the single buffer received for a single activation of stitched frame encoding;
processing, by the singular hardware encoder, the first image data and the second image data in the single buffer; and
generating, by the singular hardware encoder, a single stitched encoded frame based on the processed first image data and the processed second image data, wherein each frame in the single stitched encoded frame comprises of a corresponding first image frame from the processed first image data and a corresponding second image from the processed second image data, the single stitched encoded frame generated in response to the single activation.

11. The apparatus of claim 10 , wherein the operations further comprise:
transmitting the single stitched encoded frame to a messaging server.

12. The apparatus of claim 10 , wherein the apparatus is a head-wearable apparatus comprising:
a frame coupled to a camera; and
a first temple and a second temple coupled to opposite sides of the frame, wherein a first camera lens is coupled to the first temple and a second camera lens is coupled to the second temple.

13. The apparatus of claim 12 , wherein the first camera lens receives the first image data and the second camera lens receives the second image data.

14. The apparatus of claim 10 , wherein the first camera lens comprises a first image sensor and the second camera lens comprises a second image sensor, the processing further comprising:
synchronizing a first exposure time of the first image sensor with a second exposure time of the second image sensor.

15. The apparatus of claim 10 , wherein the operations further comprise:
transmitting the single stitched encoded frame to a client device.

16. A method comprising:
receiving, by a singular hardware encoder, a single buffer comprising first image data from a first camera lens and second image data from a second camera lens, the single buffer received for a single activation of stitched frame encoding;
processing, by the singular hardware encoder, the first image data and the second image data in the single buffer; and
generating, by the singular hardware encoder, a single stitched encoded frame based on the processed first image data and the processed second image data, wherein each frame in the single stitched encoded frame comprises of a corresponding first image frame from the processed first image data and a corresponding second image from the processed second image data, the single stitched encoded frame generated in response to the single activation.

17. The method of claim 16 , wherein the singular hardware encoder receives input from multiple locations in memory.

18. The method of claim 16 , wherein the first image data comprises a first set of stereo image frames.

19. The method of claim 16 , wherein the second image data comprises a second set of stereo image frames.

20. The method of claim 17 , wherein the singular hardware encoder reads the received input simultaneously and outputs the single buffer.

Assignees

• Snap Inc

Inventors

• Ryuma Dmitry
• Zare Seisan Farid

Classifications

• H04N13/161 Primary

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/167

  Synchronising or controlling image signals · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title