Time synchronized collective communication

Patent metadata

Field	Value
Publication number	US-12562994-B2
Application number	US-202217886606-A
Country	US
Kind code	B2
Filing date	Aug 12, 2022
Priority date	Aug 12, 2022
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

Systems, methods, and devices that perform computing operations are provided. In one example, a system includes a least one node, the at least one node having one or more processors, each having associated memory, a clock, a scheduler, the scheduler monitoring one or more of rates, rates of lanes, rates at which packets are sent, times, latencies of packets, topology, communication states, nodes, and packets in the system, an attribute monitor that measures counters for one or more of congestion state, line rate, and communication attributes. A packet scheduler determines a destination node based on information from the scheduler and the attribute monitor, and sends at least a portion of a packet to the destination node.

First claim

The invention claimed is:

1 . A system comprising:
a least one compute node that is interconnected with a plurality of other compute nodes, the at least one compute node having:
one or more processors enabling the at least one compute node to communicate with the plurality of other compute nodes via a collective communication flow, each of the one or more processors having associated memory;
a clock;
a scheduler, the scheduler monitoring one or more of rates, rates of lanes, rates at which packets are sent, times, latencies of packets, topology, communication states, nodes, and packets in the system;
an attribute monitor that measures counters for one or more of congestion state, line rate, and communication attributes; and
a packet scheduler that determines a destination compute node and data transmission characteristics for a packet based on information from the scheduler and the attribute monitor, and sends at least a portion of the packet to the destination compute node according to the transmission characteristics.

2 . The system of claim 1 , further comprising a data and time slot manager that manages an amount of data and a time slot for transmission of the at least the portion of the packet.

3 . The system of claim 2 , wherein the data and time slot manager coordinates collective communication in the system and defines one or more time slots for one or more destination compute nodes, and wherein the size of the at least a portion of the packet is further determined based on: (i) a time between time slots in the one or more time slots and/or (ii) a latency between the at least one node and the destination compute node.

4 . The system of claim 1 , wherein the clock is indirectly synchronized to single packet times between the at least one compute node and any other compute node in the system and the communication attributes include one or more of a switch buffer level, a number of pause frames, a number of dropped packets and an average packet size.

5 . The system of claim 1 , further comprising a buffer manager that monitors buffer utilization within the system, information about the buffer utilization also used by the packet scheduler to determine the destination compute node for the at least the portion of the packet.

6 . The system of claim 1 , wherein the system is a distributed, multi-node processing architecture.

7 . The system of claim 1 , wherein the system is part of a smart network interface card.

8 . The system of claim 1 , wherein the collective communication flow comprises at least one of all-to-all and all-gather and wherein the at least one compute node as well as the plurality of other compute nodes perform a collective operation using the collective communication flow by indirectly synchronizing to single packet times.

9 . The system of claim 1 , wherein the collective communication flow enables one or more collective operations to be performed.

10 . A system comprising:
a scheduler, the scheduler monitoring one or more of rates, rates of lanes, rates at which packets are sent, times, latencies of packets, topology, communication states, nodes, and packets; an attribute monitor that measures counters for one or more of congestion state, line rate, and communication attributes; a packet scheduler that: determines a destination compute node and data transmission characteristics for a packet based on information from the scheduler and the attribute monitor, and triggers hardware to send at least a portion of the packet to the destination compute node according to the transmission characteristics, wherein the destination compute node is one of a plurality of compute nodes that communicate via a collective communication flow, and wherein the transmission characteristics include at least one of a size of the at least a portion of the packet, a time to transmit the at least a portion of the packet, and a rate at which each portion of the packet is to be transmitted.

11 . The system of claim 10 , further comprising a time slot manager that manages an amount of data and a time slot for transmission of the at least the portion of the packet.

12 . The system of claim 10 , wherein the time slot manager defines one or more time slots for one or more destination compute nodes.

13 . The system of claim 10 , further comprising a clock which is indirectly synchronized amongst the plurality of compute nodes to single packet times.

14 . The system of claim 10 , further comprising a buffer manager that monitors buffer utilization, information about the buffer utilization also used by the packet scheduler to determine the destination compute node for the at least the portion of the packet.

15 . The system of claim 10 , wherein information regarding destination compute node availability is provided to the packet scheduler to assist with the determination of the destination compute node.

16 . The system of claim 10 , wherein the packet scheduler causes the hardware to send the portions of the packet until all portions are sent to their respective destinations.

17 . A method comprising:
monitoring one or more of rates, rates of lanes, rates at which packets are sent, times, latencies, latencies of packets, topology, communication states, nodes, and packets; measuring counters for one or more of congestion state, line rate, and communication attributes; determining a destination compute node and data transmission characteristics for a packet based on the monitored and measured information, wherein the destination compute node is one of a plurality of compute nodes that communicate via a collective communication flow, and wherein the transmission characteristics include at least one of a size of the at least a portion of the packet, a time to transmit the at least a portion of the packet, and a rate at which each portion of the packet is to be transmitted; and causing hardware to send at least a portion of a packet to the destination node according to the transmission characteristics.

18 . The method of claim 17 , further comprising monitoring buffer utilization, information about the buffer utilization also used to determine the destination compute node for the at least the portion of the packet.

19 . The method of claim 17 , further comprising managing an amount of data and a time slot for transmission of the at least the portion of the packet.

20 . The method of claim 17 , further comprising indirectly synchronizing a clock between the plurality of compute nodes to single packet times.

21 . The method of claim 17 , further comprising sending the portions of the packet until all portions are sent to their respective destinations.

22 . The method of claim 17 , further comprising, prior to sending the at least the portion of the packet to the destination compute node, defining one or more time slots for one or more destination compute nodes.

23 . A system comprising:
a data and time slot manager that manages one or more of an amount of data and a time slot for sending of at least a portion of a packet to a destination compute node, wherein the destination compute node is one of a plurality of compute nodes that communicate via a collective communication flow; and a packet scheduler, cooperating with the data and time slot manager, that determines the destination compute node and data transmission characteristics for a packet based on one or more of communications attributes and network attributes within t

Assignees

• Mellanox Technologies Ltd

Inventors

• Wertheimer Zsolt Alon
• Shabtai Omer
• Goldberg Barak
• Levi Lion
• Mey-Tal Gil
• Or Shapira Bar
• Levi Dotan David

Classifications

• H04L43/0894

  Packet rate · CPC title

• H04L43/0817

  by checking functioning · CPC title

• H04L43/0852

  Delays · CPC title

• H04L47/56 Primary

  implementing delay-aware scheduling · CPC title