Using reinforcement learning and personalized recommendations to generate a video stream having a predicted, personalized, and enhance-quality field-of-view

Patent metadata

Field	Value
Publication number	US-11671573-B2
Application number	US-202017120439-A
Country	US
Kind code	B2
Filing date	Dec 14, 2020
Priority date	Dec 14, 2020
Publication date	Jun 6, 2023
Grant date	Jun 6, 2023

Abstract

Embodiments of the invention are directed to a computer-implemented method that includes using a reinforcement learning (RL) system to generate a first set of displayed region candidates based on inputs received from online users while watching video. A recommendation system is used to rank the first set of displayed region candidates based on inputs received from a local user watching video. The recommendation system is further used to select a first highest ranked one of the first set of displayed region candidates. Based on the first highest ranked one of the first set of displayed region candidates, a first section of a first raw video frame is fetched that matches the first highest ranked one of the first set of displayed candidate regions, wherein the first section of the first raw video frame includes a first predicted display region of the video frame.

First claim

What is claimed is:

1. A computer-implemented method of predicting a displayed region of a video frame, the computer-implemented method comprising:
using a reinforcement learning (RL) system of a processor system to generate a first set of displayed region candidates based on inputs received at the RL system from online users while the online users are actively watching video;
using a recommendation system to rank the first set of displayed region candidates based on inputs received from a local user watching video;
using the recommendation system to select a first highest ranked one of the first set of displayed region candidates; and
based on the first highest ranked one of the first set of displayed region candidates, fetching a first section of a first raw video frame that matches the first highest ranked one of the first set of displayed candidate regions;
wherein the first section of the first raw video frame comprises a first predicted display region of the video frame.

2. The computer-implemented method of claim 1 further comprising selectively applying a video enhancement technique to only the first predicted display region of the video frame.

3. The computer-implemented method of claim 1 further comprising:
receiving from the local user an adjustment to the first predicted display region;
using the RL system of the processor system to generate a second set of displayed region candidates based on updated inputs received from the online users while watching video; and
using the recommendation system to rank the second set of displayed region candidates based on the adjustments to the first predicted display region received from the local user watching video.

4. The computer-implemented method of claim 3 further comprising using the recommendation system to rank the second set of displayed region candidates based on the adjustments to the first predicted display region received from the local user watching video.

5. The computer-implemented method of claim 4 further comprising using the recommendation system to select a second highest ranked one of the second set of displayed region candidates.

6. The computer-implemented method of claim 5 further comprising, based on the second highest ranked one of the second set of displayed region candidates, fetching a second section of a second raw video frame that matches the second highest ranked one of the second set of displayed candidate regions.

7. The computer-implemented method of claim 1 , wherein the machine learning algorithm has been trained to perform the machine learning task using a historical target environment analysis corpus comprising information from prior analyses performed by trained interpreters on other target environments.

8. A computer system comprising a processor communicatively coupled to a memory, wherein the processor performs processor operations comprising:
using a reinforcement learning (RL) system to generate a first set of displayed region candidates based on inputs received at the RL system from online users while the online users are actively watching video;
using a recommendation system to rank the first set of displayed region candidates based on inputs received from a local user watching video;
using the recommendation system to select a first highest ranked one of the first set of displayed region candidates; and
based on the first highest ranked one of the first set of displayed region candidates, fetching a first section of a first raw video frame that matches the first highest ranked one of the first set of displayed candidate regions;
wherein the first section of the first raw video frame comprises a first predicted display region of the video frame.

9. The computer system of claim 8 , wherein the processor operations further comprise selectively applying a video enhancement technique to only the first predicted display region of the video frame.

10. The computer system of claim 8 , wherein the processor operations further comprise:
receiving from the local user an adjustment to the first predicted display region; and
using the RL system of the processor system to generate a second set of displayed region candidates based on updated inputs received from the online users while watching video.

11. The computer system of claim 10 further comprising using the recommendation system to rank the second set of displayed region candidates based on the adjustments to the first predicted display region received from the local user watching video.

12. The computer system of claim 11 further comprising using the recommendation system to select a second highest ranked one of the second set of displayed region candidates.

13. The computer system of claim 12 further comprising, based on the second highest ranked one of the second set of displayed region candidates, fetching a second section of a second raw video frame that matches the second highest ranked one of the second set of displayed candidate regions.

14. The computer system of claim 13 , wherein the second section of the second raw video frame comprises a second predicted display region of the video frame.

15. A computer program product comprising a computer readable program stored on a computer readable storage medium, wherein the computer readable program, when executed on a processor system, causes the processor system to perform processor operations comprising:
using a reinforcement learning (RL) system to generate a first set of displayed region candidates based on inputs received at the RL system from online users while the online users are actively watching video;
using a recommendation system to rank the first set of displayed region candidates based on inputs received from a local user watching video;
using the recommendation system to select a first highest ranked one of the first set of displayed region candidates; and
based on the first highest ranked one of the first set of displayed region candidates, fetching a first section of a first raw video frame that matches the first highest ranked one of the first set of displayed candidate regions;
wherein the first section of the first raw video frame comprises a first predicted display region of the video frame.

16. The computer program product of claim 15 , wherein the processor operations further comprise selectively applying a video enhancement technique to only the first predicted display region of the video frame.

17. The computer program product of claim 15 , wherein the processor operations further comprise:
receiving from the local user an adjustment to the first predicted display region; and
using the RL system of the processor system to generate a second set of displayed region candidates based on updated inputs received from the online users while watching video.

18. The computer program product of claim 17 , wherein the processor operations further comprise using the recommendation system to rank the second set of displayed region candidates based on the adjustments to the first predicted display region received from the local user watching video.

19. The computer program product of claim 18 , wherein the processor operations further comprise using the recommendation system to select a second highest ranked one of the second set of displayed region candidates.

20. The computer program product of claim 19 , wherein:
the processor operations further comprise, based on the second highest ranked one of the second set of displayed region candidates, fetching a second section of a second

Assignees

• IBM

Inventors

• Wang Tao
• Zhang shi lei
• Wang Lingyun
• Wang Qing

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• H04N21/234345

  the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment · CPC title

• H04N21/23439

  for generating different versions · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N13/139

  Format conversion, e.g. of frame-rate or size · CPC title