Accommodating irregular timing information in streamed media data

Patent metadata

Field	Value
Publication number	US-11089386-B1
Application number	US-201916457862-A
Country	US
Kind code	B1
Filing date	Jun 28, 2019
Priority date	Jun 28, 2019
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

Techniques for accommodating irregular timing information in streamed media data are described. According to some embodiments, at least a portion of a media stream that includes a video component is received, the video component including a plurality of video frames and video timing data, and wherein the media stream includes an indication of a video frame rate. A first duration of a first video frame of the plurality of video frames is determined to be different than an expected video frame duration, wherein the expected video frame duration is based at least in part on the indication of the video frame rate. A timestamp of at least one video frame of the plurality of video frames is adjusted to change the first duration.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving at least a portion of a media stream that includes a video component and an audio component, the video component including a plurality of video frames and video timing data, the audio component including a plurality of audio frames and audio timing data, and wherein the media stream includes an indication of a video frame rate and further includes an indication of an audio frame rate;
tracking a number of times a difference between each of a plurality of audio timestamps of the plurality of audio frames and an associated expected audio timestamp exceeded a tolerance;
increasing the tolerance when the number of times exceeds a maximum number of resynchronization events between the audio component and the video component over a period of time;
determining that a first duration of a first video frame of the plurality of video frames is different than an expected video frame duration, wherein the expected video frame duration is based at least in part on the indication of the video frame rate; and
adjusting a timestamp of at least one video frame of the plurality of video frames to change the first duration.

2. The computer-implemented method of claim 1 , further comprising:
incrementing an expected video timestamp by the first duration;
using the expected video timestamp to identify a subsequent video frame of the plurality of video frames; and
processing the subsequent video frame.

3. The computer-implemented method of claim 1 , further comprising:
tracking a variation parameter indicating a difference between each of a plurality of audio timestamps of the plurality of audio frames and an associated expected audio timestamp, wherein the associated expected audio timestamp for each of the plurality of audio timestamps is based at least in part on the indication of the audio frame rate; and
resynchronizing the audio component to the video component when the variation parameter is within a range.

4. The computer-implemented method of claim 1 , further comprising decreasing the tolerance when the number of times falls below a maximum number of resynchronization events between the audio component and the video component over a period of time.

5. The computer-implemented method of claim 1 :
wherein the at least one video frame is part of a group of video frames that match a pattern of frame durations; and
wherein a second duration of a second video frame of the group of video frames is greater than the expected video frame duration.

6. The computer-implemented method of claim 1 :
wherein the at least one video frame is part of a group of video frames; and
the method further comprising adjusting each timestamp associated with a video frame of the group of video frames based at least in part on a total duration of the group of video frames.

7. The computer-implemented method of claim 1 , wherein the adjusted timestamp is stored in a buffer upstream of a video encoder of a media ingestion service of a provider network.

8. The computer-implemented method of claim 1 , wherein the adjusting the timestamp of the at least one video frame is contingent on receipt of a message that enables timestamp adjustment.

9. A system comprising:
a media distribution service implemented by a first one or more electronic devices of a provider network; and
a media ingestion service implemented by a second one or more electronic devices of the provider network, the media ingestion service including instructions that upon execution cause the media ingestion service to:
receive at least a portion of a media stream that includes a video component and an audio component, the video component including a plurality of video frames and video timing data, the audio component including a plurality of audio frames and audio timing data, and wherein the media stream includes an indication of a video frame rate and an indication of an audio frame rate;
track a number of times a difference between each of a plurality of audio timestamps of the plurality of audio frames and an associated expected audio timestamp exceeded a tolerance;
increase the tolerance when the number of times exceeds a maximum number of resynchronization events between the audio component and the video component over a period of time;
determine that a first duration of a first video frame of the plurality of video frames is different than an expected video frame duration, wherein the expected video frame duration is based at least in part on the indication of the video frame rate;
adjust a timestamp of at least one video frame of the plurality of video frames to change the first duration;
encode the at least one video frame to generate one or more encoded video frames in a compressed format; and
send the one or more encoded video frames to the media distribution service.

10. The system of claim 9 , wherein the media ingestion service includes further instructions that upon execution cause the media ingestion service to:
increment an expected video timestamp by the first duration;
use the expected video timestamp to identify a subsequent video frame of the plurality of video frames; and
process the subsequent video frame.

11. The system of claim 9
wherein the media ingestion service includes further instructions that upon execution cause the media ingestion service to:
track a variation parameter indicating a difference between each of a plurality of audio timestamps of the plurality of audio frames and an associated expected audio timestamp, wherein the associated expected audio timestamp for each of the plurality of audio timestamps is based at least in part on the indication of the audio frame rate; and
resynchronize the audio component to the video component when the variation parameter is within a range.

12. The system of claim 9 , wherein the media ingestion service includes further instructions that upon execution cause the media ingestion service to decrease the tolerance when the number of times falls below a maximum number of resynchronization events between the audio component and the video component over a period of time.

13. The system of claim 9 :
wherein the at least one video frame is part of a group of video frames that match a pattern of frame durations; and
wherein a second duration of a second video frame of the group of video frames is greater than the expected video frame duration.

14. The system of claim 9 :
wherein the at least one video frame is part of a group of video frames; and
wherein the media ingestion service includes further instructions that upon execution cause the media ingestion service to adjust each timestamp associated with a video frame of the group of video frames based at least in part on a total duration of the group of video frames.

15. The system of claim 9 , wherein the adjustment of the timestamp of the at least one video frame is contingent on receipt of a message that enables timestamp adjustment by the media ingestion service.

16. One or more non-transitory computer readable storage media having instructions which, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving at least a portion of a media stream that includes a video component and an audio component, the video component including a plurality of video frames and video timing data, the audio component including a plurality of audio frames and audio timing data, and wherein the media stream includes an indication of a video frame rate and furt

Assignees

• Amazon Tech Inc

Inventors

• Bernstein Ryan
• Hegar Ryan
• Kiykioglu Timur Darren
• Montgomery David

Classifications

• H04N21/8547 Primary

  involving timestamps for synchronizing content · CPC title

• H04N21/64322

  IP · CPC title

• H04N21/242 Primary

  Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)} · CPC title

• H04N21/218

  Source of audio or video content {, e.g. local disk arrays (details of retrieval in video databases G06F16/739)} · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title