Reinforcement learning-based techniques for training a natural media agent

Patent metadata

Field	Value
Publication number	US-12481884-B2
Application number	US-202318479486-A
Country	US
Kind code	B2
Filing date	Oct 2, 2023
Priority date	Aug 23, 2019
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

Some embodiments involve a reinforcement learning based framework for training a natural media agent to learn a rendering policy without human supervision or labeled datasets. The reinforcement learning based framework feeds the natural media agent a training dataset to implicitly learn the rendering policy by exploring a canvas and minimizing a loss function. Once trained, the natural media agent can be applied to any reference image to generate a series (or sequence) of continuous-valued primitive graphic actions, e.g., sequence of painting strokes, that when rendered by a synthetic rendering environment on a canvas, reproduce an identical or transformed version of the reference image subject to limitations of an action space and the learned rendering policy.

First claim

What is claimed is:

1 . One or more non-transitory computer-readable media storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
predicting, by applying a representation of a current working observation of a canvas in a synthetic rendering environment to a natural media agent comprising one or more deep neural networks, at least one primitive graphic action; executing, by a media rendering engine, the at least one primitive graphic action to generate an updated state of the canvas in the synthetic rendering environment; updating, by a reward generation module, an accumulated reward, accumulated over a plurality of iterations of the natural media agent, based on a difference between at least a portion of the updated state of the canvas and a current training image of a set of training images; and updating, in response to detecting a trigger, the one or more deep neural networks using the accumulated reward, wherein the trigger comprises at least one of a designated number of iterations of the natural media agent, a number of iterations of the natural media agent that increases between episodes of iterations during which the accumulated reward is updated, or a determination that a position of a media rendering instrument in the updated state of the canvas moved more than a threshold distance from a center of an ego-centric patch of the canvas.

2 . The one or more non-transitory computer-readable media of claim 1 , wherein the representation of the current working observation applied to the natural media agent comprises an encoded representation of position of a media rendering instrument on the canvas.

3 . The one or more non-transitory computer-readable media of claim 1 , wherein the representation of the current working observation applied to the natural media agent comprises a concatenated representation of corresponding ego-centric patches of the canvas and the current training image.

4 . The one or more non-transitory computer-readable media of claim 1 , wherein the accumulated reward encourages the natural media agent to predict longer strokes.

5 . The one or more non-transitory computer-readable media of claim 1 , the operations further comprising pre-training the natural media agent on data representing human behaviors prior to training the natural media agent using reinforcement learning.

6 . The one or more non-transitory computer-readable media of claim 1 , wherein curriculum leaning is used to encourage the natural media agent to find the accumulated reward.

7 . The one or more non-transitory computer-readable media of claim 1 , wherein the accumulated reward is updated over the plurality of iterations using a difficulty-based method.

8 . A method comprising:
generating, based at least on processing a representation of a current working observation of a canvas in a synthetic rendering environment using a natural media agent comprising one or more deep neural networks, a representation of at least one primitive graphic action; generating an updated state of the canvas in the synthetic rendering environment based at least on the at least one primitive graphic action; updating an accumulated reward, accumulated over a plurality of iterations of the natural media agent, based on a difference between at least a portion of the updated state of the canvas and a current training image of a set of training images; and updating, in response to detecting a trigger, the one or more deep neural networks using the accumulated reward, wherein the trigger comprises at least one of a designated number of iterations of the natural media agent, a number of iterations of the natural media agent that increases between episodes of iterations during which the accumulated reward is updated, or a determination that a position of a media rendering instrument in the updated state of the canvas moved more than a threshold distance from a center of an ego-centric patch of the canvas.

9 . The method of claim 8 , wherein the representation of the current working observation processing using the natural media agent comprises an encoded representation of position of a media rendering instrument on the canvas.

10 . The method of claim 8 , wherein the representation of the current working observation processing using the natural media agent comprises a concatenated representation of corresponding ego-centric patches of the canvas and the current training image.

11 . The method of claim 8 , wherein the accumulated reward encourages the natural media agent to generate longer strokes.

12 . The method of claim 8 , further comprising pre-training the natural media agent on data representing human behaviors prior to training the natural media agent using reinforcement learning.

13 . The method of claim 8 , wherein curriculum leaning is used to encourage the natural media agent to find the accumulated reward.

14 . The method of claim 8 , wherein the accumulated reward is updated over the plurality of iterations using a difficulty-based method.

15 . A system comprising:
a memory component; and one or more processing devices coupled to the memory component, the one or more processing devices configured to perform operations comprising:
predicting, by applying a representation of a current working observation of a canvas in a synthetic rendering environment to a natural media agent comprising one or more deep neural networks, at least one primitive graphic action;
executing the at least one primitive graphic action to generate an updated state of the canvas in the synthetic rendering environment;
updating an accumulated reward, accumulated over a plurality of iterations of the natural media agent, based on a difference between at least a portion of the updated state of the canvas and a current training image of a set of training images; and
updating, in response to detecting a trigger, the one or more deep neural networks using the accumulated reward, wherein the trigger comprises at least one of a designated number of iterations of the natural media agent, a number of iterations of the natural media agent that increases between episodes of iterations during which the accumulated reward is updated, or a determination that a position of a media rendering instrument in the updated state of the canvas moved more than a threshold distance from a center of an ego-centric patch of the canvas.

16 . The system of claim 15 , wherein curriculum leaning is used to encourage the natural media agent to find the accumulated reward.

17 . The system of claim 15 , wherein the representation of the current working observation of the canvas in the synthetic rendering environment comprises an encoded representation of position of a media rendering instrument on the canvas.

18 . The system of claim 15 , wherein the representation of the current working observation of the canvas in the synthetic rendering environment comprises a concatenated representation of corresponding ego-centric patches of the canvas and the current training image.

19 . The system of claim 15 , wherein the accumulated reward encourages the natural media agent to predict longer strokes.

20 . The system of claim 15 , wherein the operations further comprise pre-training the natural media agent on data representing human behaviors prior to training the natural media agent using reinforcement learning.

Assignees

• Adobe Inc

Inventors

• Brandt Jonathan
• Fang Chen
• Kim Byungmoon
• Jia Biao

Classifications

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G09G5/37

  Details of the operation on graphic patterns (G09G5/38 takes precedence) · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/09

  Supervised learning · CPC title