Closed loop tasking and control of heterogeneous sensor networks

Patent metadata

Field	Value
Publication number	US-12452957-B2
Application number	US-202418907392-A
Country	US
Kind code	B2
Filing date	Oct 4, 2024
Priority date	Oct 6, 2023
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

Technology is described for controlling a heterogeneous sensor node network. The method can include obtaining sensor data from a plurality of sensors which include a plurality of sensor modalities in sensor nodes. The sensor data may be combined in a joint feature space that represents multimodal input. Another operation may be detecting features from the sensor data using the joint feature space. Neighboring sensor nodes may be identified to receive the features. A further operation may be sending the features to other sensor nodes in a sensor node network.

First claim

The invention claimed is:

1. A method for controlling a sensor node network, comprising:
obtaining sensor data from a plurality of sensors which include a plurality of sensor modalities, wherein the sensors are associated with sensor nodes;
combining the sensor data in a joint feature space that represents multimodal input from the plurality of sensor modalities;
detecting features from the sensor data using the joint feature space;
identifying neighboring sensor nodes to receive the features;
sending the features to other sensor nodes in the sensor node network; and
applying low-rank tensor regression to sensor data from separate sensor data modalities to enable discovery of cross-modality contextual correlations that are features in the sensor data from the plurality of sensors.

2. The method as in claim 1 , further comprising correlating spatial position data of the features by time and modality in the joint feature space for the sensor data of a sensor node.

3. The method as in claim 1 , further comprising sending the features to other sensor nodes that are neighbors within a communication neighborhood when a feature is available.

4. The method as in claim 1 , further comprising, using an online reinforcement learning (RL) process to adapt sensor control parameters, wherein a reward function is maximized by modifying the sensor control parameters until features related to events are detected, tracked and classified.

5. The method as in claim 1 , further comprising using a non-cooperative differential game, consensus negotiation protocol and distributed messages for determining a global sensor network utility function.

6. The method as in claim 1 , further comprising applying game theory to influence a focus of RL (reinforcement learning) agents of the sensor nodes.

7. The method as in claim 1 , further comprising:
receiving a notification that a communication link has terminated on one or more sensor nodes; and
playing an assignment game via remaining communication links, wherein the one or more sensor nodes can operate their RL (reinforcement learning) agents to allow the sensor nodes to continue to maintain situation awareness.

8. The method as in claim 1 , wherein the features are relevant to events captured in the sensor data.

9. The method as in claim 1 , further comprising sensing a feature in the sensor data based on detecting a response of elements of the environment.

10. The method as in claim 1 , further comprising sensing a feature in the sensor data based on detecting a response of humans in the environment.

11. The method as in claim 1 , further comprising using game theory to decide which features received from other sensor nodes to use in order increase a reward for a RL (reinforcement learning) process.

12. The method as in claim 1 , further comprising using utility functions with Nash equilibrium for a game allocating targets to sensors involved in the game.

13. A system for controlling a sensor node network, comprising:
at least one processor;
a memory device including instructions that, when executed by the at least one processor, cause the system to:
obtain sensor data from a plurality of sensors which include a plurality of sensor modalities, wherein the sensors are associated with sensor nodes;
combine the sensor data in a joint feature space that represents multimodal input from the plurality of sensor modalities, wherein spatial position data of the features is correlated by time and modality in the joint feature space;
detect features from the sensor data using the joint feature space;
identify neighboring sensor nodes to receive the features;
send the features to other sensor nodes in the sensor node network; and
apply low-rank tensor regression to sensor data from separate sensor data modalities to enable discovery of cross-modality contextual correlations that are features in the sensor data from the sensors.

14. The system as in claim 13 , further comprising sending the features to other sensor nodes that are neighbors within a communication neighborhood when a feature is available.

15. The system as in claim 13 , further comprising, using an online reinforcement learning (RL) process with an in-situ training process to adapt sensor control parameters, wherein a reward function is maximized by modifying the sensor control parameters until features related to events are detected, tracked and classified.

16. The system as in claim 13 , further comprising using a non-cooperative differential game, consensus negotiation protocol and distributed messages for determining a global sensor network utility function.

17. The system as in claim 13 , further comprising applying game theory to influence a focus of reinforcement learning agents of the sensor nodes.

18. The system as in claim 13 , further comprising:
receiving a notification that a communication link has terminated on one or more sensor nodes; and
playing an assignment game via remaining communication links, wherein the sensor nodes can operate RL (reinforcement learning) agents to allow the sensor nodes to continue to maintain situation awareness.

19. The system as in claim 13 , wherein the features are relevant to events captured in the sensor data.

20. The system as in claim 13 , further comprising sensing a feature in the sensor data based on detecting a response of elements of the environment.

21. The system as in claim 13 , further comprising sensing a feature in the sensor data based on detecting a response of humans in the environment.

22. The system as in claim 13 , further comprising using game theory to decide which features received from other sensor nodes to use in order increase a reward for an RL (reinforcement learning) process.

23. The system as in claim 13 , further comprising using utility functions with Nash equilibrium for a game allocating targets to sensors involved in the game.

24. A non-transitory machine readable storage medium having instructions embodied thereon, the instructions when executed by one or more processors, cause the one or more processors to perform a process including:
obtaining sensor data from a plurality of sensors which include a plurality of sensor modalities, wherein the sensors are associated with sensor nodes;
combining the sensor data in a joint feature space that represents multimodal input from the plurality of sensor modalities, wherein spatial position data of the features is correlated by time and modality in the joint feature space;
detecting features from the sensor data using the joint feature space;
identifying neighboring sensor nodes to receive the features;
sending the features to other sensor nodes in a sensor node network; and
applying low-rank tensor regression to sensor data from separate sensor data modalities to enable discovery of cross-modality contextual correlations that are features in the sensor data from the sensors.

25. The non-transitory machine readable storage medium as in claim 24 , further comprising sending the features to other sensor nodes that are neighbors within a communication neighborhood when a feature is available.

26. A method for controlling a sensor node network, comprising:
obtaining sensor data from a plurality of sensors which include a plurality of sensor modalities, wherein the sensors are associated with sensor nodes;
combining the sensor data in a joint feature space that represents

Assignees

• Sarcos Corp

Inventors

• Garagic Denis
• Ravier Robert
• Galoppo Travis
• Jameson Rex
• Smith Fraser M

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• H04W4/02

  Services making use of location information · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06N20/00

  Machine learning · CPC title

• H04W4/38

  for collecting sensor information · CPC title