Using NeRF models to facilitate operations of a UAV delivery service

Patent metadata

Field	Value
Publication number	US-12461543-B2
Application number	US-202318382806-A
Country	US
Kind code	B2
Filing date	Oct 23, 2023
Priority date	Oct 23, 2023
Publication date	Nov 4, 2025
Grant date	Nov 4, 2025

Abstract

A method of operation of an unmanned aerial vehicle (UAV) service includes acquiring aerial images of a scene at an area of interest (AOI), wherein the aerial images are acquired with a UAV of the UAV service during a flight mission of the UAV that passes over the AOI; uploading a mission log of the flight mission to a backend data system of the UAV service, the mission log including image data that includes, or is derived from, at least a portion of the aerial images; and training a neural radiance field (NeRF) model with one or more of the aerial images, wherein the NeRF model comprises a neural network, which after the training, encodes a volumetric representation of the scene capable of generating novel views of the scene different than any of the aerial images used to train the NeRF model.

First claim

What is claimed is:

1 . A method of operation of an unmanned aerial vehicle (UAV) service, the method comprising:
acquiring aerial images of a scene at an area of interest (AOI), wherein the aerial images are acquired with a UAV of the UAV service during a flight mission of the UAV that passes over the AOI; training a neural radiance field (NeRF) model on-board the UAV with one or more of the aerial images, wherein the NeRF model comprises a neural network, which after the training, encodes a volumetric representation of the scene capable of generating novel views of the scene different than any of the aerial images used to train the NeRF model; determining whether a terrain model of the AOI maintained in a backend data system of the UAV service is deemed out-of-date based upon whether the training of the NeRF model results in greater than a threshold change in the NeRF model; and uploading a mission log of the flight mission to the backend data system, the mission log including image data that includes, or is derived from, at least a portion of the aerial images.

2 . The method of claim 1 , wherein the image data included with the mission log comprises the NeRF model after the training.

3 . The method of claim 1 , further comprising:
conducting a UAV flight simulation using one or more of the novel views output from the NeRF model to test at least one of a UAV hardware or software revision under consideration for updating UAVs of the UAV service.

4 . The method of claim 3 , wherein conducting the UAV flight simulation comprises:
executing a log replay simulation that uses the mission log from the flight mission to provide first sensor stimulus to a virtual UAV within the UAV flight simulation for a first portion of the UAV flight simulation; and executing a closed loop simulation that uses the NeRF model to generate second sensor stimulus provided to the virtual UAV within the UAV flight simulation for a second portion of the UAV flight simulation.

5 . The method of claim 4 , wherein conducting the UAV flight simulation further comprises:
transitioning between the log replay simulation and the closed loop simulation based upon a geofence trigger.

6 . The method of claim 4 , wherein conducting the UAV flight simulation further comprises:
transitioning between the log replay simulation and the closed loop simulation based upon a transition between flight phases of the flight mission.

7 . The method of claim 4 , wherein conducting the UAV flight simulation further comprises:
transitioning between the log replay simulation and the closed loop simulation based upon an obstacle encounter by the virtual UAV during the UAV flight simulation.

8 . The method of claim 4 , wherein conducting the UAV flight simulation further comprises:
comparing the log replay simulation against the mission log; and transitioning between the log replay simulation and the closed loop simulation based upon reaching a threshold deviation between at least one of a heading, an attitude, a velocity, a position, or a route of the virtual UAV compared to at least one corresponding value derived from the mission log.

9 . The method of claim 1 , wherein the training comprises retraining of the NeRF model that was previously trained based upon previously acquired aerial images of the scene at the AOI, the method further comprising:
determining whether the terrain model of the AOI maintained in the backend data system is deemed out-of-date based upon whether the retraining of the NeRF model results in greater than the threshold change in the NeRF model.

10 . The method of claim 1 , wherein the training comprises retraining of the NeRF model on the UAV and wherein the uploading of the mission log further comprises:
uploading a larger set of the image data to the backend data system from the UAV with the mission log when the retraining of the NeRF model results in greater than a threshold change in the NeRF model than when the retraining results in less than the threshold change in the NeRF model.

11 . The method of claim 10 , wherein the retraining of the NeRF model is conducted on the UAV while flying over the AOI with an initial limited set of the aerial images and used to determine whether the larger set of the aerial images is acquired or saved.

12 . The method of claim 10 , wherein the threshold change in the NeRF model comprises one or more individual threshold changes or a collective threshold change in weights or biases of the neural network of the NeRF model.

13 . The method of claim 1 , further comprising:
uploading a reference NeRF model into the UAV with mission data used by the UAV to execute the flight mission; querying the reference NeRF model to derive a pose estimate associated with one of the aerial images; and geolocating the UAV while the UAV is flying based upon the pose estimation.

14 . At least one non-transitory computer-readable medium storing instructions that, when executed by one or more machines of an unmanned aerial vehicle (UAV) delivery service, will cause the one or more machines to perform operations comprising:
acquiring aerial images of a scene at an area of interest (AOI), wherein the aerial images are acquired with a UAV of the UAV delivery service during a flight mission of the UAV that passes over the AOI; uploading a mission log of the flight mission to a backend data system of the UAV delivery service, the mission log including image data that includes, or is derived from, at least a portion of the aerial images; and training a neural radiance field (NeRF) model with one or more of the aerial images, wherein the NeRF model comprises a neural network, which after the training, encodes a volumetric representation of the scene capable of generating novel views of the scene different than any of the aerial images used to train the NeRF model, wherein the training comprises retraining of the NeRF model on the UAV and wherein the uploading of the mission log further comprises:
uploading a larger set of the image data to the backend data system from the UAV with the mission log when the retraining of the NeRF model results in greater than a threshold change in the NeRF model than when the retraining results in less than the threshold change in the NeRF model.

15 . The at least one non-transitory computer-readable medium of claim 14 , wherein training the NeRF model comprises training the NeRF model on-board the UAV and the image data included with the mission log comprises the NeRF model after the training.

16 . The at least one non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:
conducting a UAV flight simulation using one or more of the novel views output from the NeRF model to test at least one of a UAV hardware or software revision under consideration for updating UAVs of the UAV delivery service.

17 . The at least one non-transitory computer-readable medium of claim 16 , wherein conducting the UAV flight simulation comprises:
executing a log replay simulation that uses the mission log from the flight mission to provide first sensor stimulus to a virtual UAV within the UAV flight simulation for a first portion of the UAV flight simulation; and executing a closed loop simulation that uses the NeRF model to generate second sensor stimulus provided to the virtual UAV within the UAV flight simulation for a second portion of the UAV flight simulation.

18 . The at least one non-transitory computer-readable medium of claim 17 , wherein conducting the UAV

Assignees

• Wing Aviation Llc

Inventors

• Abeywardena Dinuka
• Bozhkov Konstantin
• Jin Linhao
• Commun Domitille
• Fan Xinzhi
• Krafka Kyle

Classifications

• G06N3/02

  Neural networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/13

  Satellite images · CPC title