Interactive cost corrections with natural language feedback

Patent metadata

Field	Value
Publication number	US-12502789-B2
Application number	US-202218055569-A
Country	US
Kind code	B2
Filing date	Nov 15, 2022
Priority date	Nov 16, 2021
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

Approaches presented herein provide for a framework to integrate human provided feedback in natural language to update a robot planning cost or value. The natural language feedback may be modeled as a cost or value associated with completing a task assigned to the robot. This cost or value may then be added to an initial task cost or value to update one or more actions to be performed by the robot. The framework can be applied to both real work and simulated environments where the robot may receive instructions, in natural language, that either provide a goal, modify an existing goal, or provide constraints to actions to achieve an existing goal.

First claim

What is claimed is:

1 . A method, comprising:
receiving a natural language correction associated with a task for a robot in an environment; receiving an image of the environment; determining a robot state at a time corresponding to receipt of the natural language correction; determining, based at least on the natural language correction, the image, and the robot state, a correction value; determining, using the correction value, a refinement to one or more actions of the task, the refinement deactivating an original cost associated with the task; applying the refinement to the task; and reactivating the original cost after completing the refinement.

2 . The method of claim 1 , further comprising:
providing the robot state and the correction value to a motion planner prior to computing the refinement.

3 . The method of claim 1 , wherein the image is one of captured by a camera in the environment or simulated.

4 . The method of claim 1 , further comprising:
generating a cost map; applying a mask to the cost map, the mask based, at least in part, on the natural language correction; and determining, based at least in part on the cost map and the mask, a trajectory.

5 . The method of claim 1 , further comprising:
determining a correction type associated with the natural language correction; and adjusting a task value based, at least in part, on the correction type.

6 . The method of claim 5 , further comprising:
determining the correction type is a goal specification; and setting an existing goal value to zero.

7 . A system, comprising:
a processor; and memory including instructions that, when performed by the processor, cause the system to:
determine, from at least a task value and a base value, a cumulative value for performing a task within an environment;
determine, based at least on a natural language correction for the task, a correction value for an intermediate task configured to modify a goal associated with the task;
determine, from the correction value and the cumulative value, an updated value after removing the task value from the cumulative value;
modify one or more actions associated with the task based, at least in part, on the updated value; and
determine, after completion of the intermediate task, a modified cumulative value including the task value and the base value.

8 . The system of claim 7 , wherein the instructions, when performed by the processor, cause the system to further:
generate a cost map for a trajectory associated with the task; generate a mask based, at least in part, on the natural language correction; apply the mask to the cost map; and determine an updated trajectory.

9 . The system of claim 7 , wherein the correction value is determined, at least in part, using a pre-trained Contrastive Language-Image Pre-training (CLIP) model.

10 . The system of claim 9 , wherein the CLIP model includes a first train and a second train, the first train to embed the natural language correction and the second train to encode an image of the environment.

11 . The system of claim 7 , wherein the system comprises at least one of:
a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system incorporating one or more Virtual Machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Assignees

• Nvidia Corp

Inventors

• Sundaralingam Balakumar
• Sharma Pratyusha
• Paxton Christopher Jason
• Blukis Valts
• Hermans Tucker
• Fox Dieter

Classifications

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• B25J19/023

  including video camera means · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title

• G05B2219/40425

  Sensing, vision based motion planning · CPC title