10 Gbit/s Ethernet Link

The Time Tagger X provides multiple interfaces for transferring time-tag data. For standard operation, the recommended configuration is to transfer time tags to the host PC via USB 3.0, which provides a specified data transfer rate up to 90 MTags/s. For applications requiring higher sustained throughput or lower latency, the Time Tagger X also provides SFP+ and QSFP+ interfaces. These interfaces are also discussed in the FPGA Link in-depth guide, where they are employed to stream time tags to external FPGA hardware through an Ethernet-based protocol.

This guide describes how to use the SFP+ interface as a 10 Gbit/s Ethernet (10 GbE) link for streaming time tags directly from the Time Tagger X to an acquisition PC. The data stream is received and processed by the Time Tagger software.

The 10 GbE interface supplements rather than replaces the USB connection: USB remains required for device configuration and control, while the time-tag data stream is routed over the SFP+ interface. At any given time, all time-tag data is transferred over a single data interface; it is not possible to route data from some channels over USB and from others over the SFP+ interface simultaneously.

Note

The 10 GbE link is intended for applications with sustained time-tag rates that exceed what can be transferred consistently over the USB interface. For lower data rates, the USB interface is simpler and therefore recommended.

Note

10 GbE data transfer is available to all users at up to 100 MTags/s. With the High Rate hardware license, the full transfer rate of up to 300 MTags/s over SFP+ is enabled. To upgrade your license, contact sales@swabianinstruments.com. The license can then be updated using the standard procedure described in Hardware License Upgrades.

The following sections describe the setup requirements and provide step-by-step guidance for configuring and validating the connection.

Setup overview and requirements

The acquisition PC must be equipped with a dedicated 10 GbE network adapter with an SFP+ port, either an internal PCIe card or an external adapter, and connected to the Time Tagger X via two independent interfaces: USB 3.0 for device configuration and control, and the SFP+ interface for the time-tag data stream. The recommended physical setup is a direct point-to-point connection between the SFP+ port of the Time Tagger X and the 10 GbE adapter. The acquisition PC must run Time Tagger software version 2.22 or later.

The time-tag data stream uses UDP. Because the Time Tagger X generates Ethernet frames directly without relying on ARP, the IPv4 address and MAC address of the receiving network interface must be supplied explicitly when enabling the stream via the Time Tagger API. How to identify these values is described in the next section.

The achievable time-tag transfer rate depends strongly on the MTU, which defines the maximum packet size used for the UDP data stream. Standard Ethernet operation uses an MTU of 1500 bytes. Larger MTU values require support for larger-than-standard Ethernet frames, commonly referred to as jumbo frames. The Time Tagger API uses an MTU of 9000 bytes by default. For high-rate operation, the receiving network interface must therefore be configured to accept Ethernet frames large enough to carry UDP packets with this MTU, as described in the following section.

If the network path does not support an MTU of 9000 bytes, the Time Tagger MTU parameter must be reduced to the maximum value accepted by all components in the path. At the standard Ethernet MTU of 1500 bytes, the achievable transfer rate is significantly reduced because the same data must be carried by many more, smaller UDP packets; in this case, we recommend using the USB interface instead.

The following table lists hardware configurations validated for 10 GbE streaming, together with the achieved sustained transfer rates at MTU 9000 bytes. Comparable rates may be achievable with MTU values above approximately 5000 bytes.

10 GbE SFP+ interface	CPU	OS	Transfer rate (MTags/s)
Intel X520 (PCIe)	AMD Ryzen 9 9950X	Windows 11 / Ubuntu 24.04	300 / 260
Intel X710 (PCIe)	AMD Ryzen 9 9950X	Ubuntu 24.04	264
QNAP QNA-T310G1S (Thunderbolt 3)	AMD Ryzen 9 9950X	Windows 11	300
QNAP QNA-T310G1S (Thunderbolt 3)	AMD Ryzen 7 PRO 8840HS	Windows 11	300
QNAP QNA-T310G1S (Thunderbolt 3)	Intel Core i7-1360P	Windows 11	300

All measurements were performed with Time Tagger software version 2.22 using a modified version of the transfer_rate.py example found in the 6-Various-Examples folder of the Time Tagger installation, with 8 channels enabled.

Preparing the receiving network interface

Before enabling 10 GbE streaming, the receiving network interface must be configured to accept packets with an MTU of at least 9000 bytes, which matches the default MTU used by the Time Tagger API. The IPv4 address and MAC address of the receiving interface must also be identified. The exact steps depend on the operating system and on the network-adapter driver.

Windows

Identifying the 10 GbE interface

Open a PowerShell window and run:

Get-NetAdapter

The output lists all available network adapters with their name, description, and link state. Connect the cable between the Time Tagger X SFP+ port and the 10 GbE adapter; the corresponding entry will change from Disconnected to Up. Note the adapter name; it is used in the following steps.

Configuring jumbo frames / MTU

1. Open Device Manager and expand Network Adapters.

2. Right-click the 10 GbE adapter and select Properties.

3. Go to the Advanced tab and locate the Jumbo Packet (or Jumbo Frame) property.

4. Set the value to 9014 Bytes or the highest available value of at least 9000 bytes.

5. Click OK.

Finding the IPv4 address and MAC address

Open a Command Prompt and run:

ipconfig /all

Locate the entry for the adapter name identified above. The IPv4 Address and Physical Address (MAC address) fields provide the values required by the Time Tagger API.

Alternatively, open Settings ‣ Network & Internet, select the adapter, and click Properties. Both the IPv4 address and the Physical address (|MAC|) are listed there.

Linux

Identifying the 10 GbE interface

Run the following command to list all network interfaces:

ip link show

The output lists all interfaces with their current state. The 10 GbE interface connected to the Time Tagger X can be identified by its state changing from DOWN to UP when the cable is plugged in. Note the interface name (e.g. enp3s0 or eth0); it is used in all subsequent commands.

Configuring the MTU

Set the MTU to 9000 bytes, replacing <interface> with the name identified above:

sudo ip link set <interface> mtu 9000

Verify that the MTU has been applied:

ip link show <interface>

The output should contain mtu 9000. This change takes effect immediately but might not be persistent across reboots.

Finding the IPv4 address and MAC address

ip addr show <interface>

The MAC address appears on the link/ether line and the IPv4 address on the inet line. These are the values to supply to the Time Tagger API.

Enabling 10 GbE streaming

After the receiving network interface has been prepared, 10 GbE streaming is enabled via the Time Tagger API. The Time Tagger X must first be connected and initialized over USB, as in normal operation.

Call xtra_enableStreamInterfaceUDP() with the following parameters:

• pc_ip: the IPv4 address of the receiving network interface, identified in the previous section.

• pc_mac: the MAC address of the receiving network interface, identified in the previous section.

• tagger_ip: a user-chosen IPv4 address for the Time Tagger X’s SFP+ interface. This address must be on the same subnet as pc_ip. A simple choice is to take the PC’s IP address and change the last octet to an unused address in the same subnet, as shown in the example below.

• MTU: the MTU in bytes (default: 9000 bytes). Must not exceed the MTU configured on the receiving adapter and is limited to a maximum of 9500 bytes.

Replace the values in the example below with the values used in your setup.

from Swabian import TimeTagger

PC_IP     = '169.254.99.217'
PC_MAC    = '24:5E:BE:44:26:AF'
TAGGER_IP = '169.254.99.216'  # user-chosen; must be on the same subnet as PC_IP

tagger = TimeTagger.createTimeTagger()

tagger.xtra_enableStreamInterfaceUDP(pc_ip=PC_IP,
                                     pc_mac=PC_MAC,
                                     tagger_ip=TAGGER_IP,
                                     MTU=9000)

Once the UDP stream is enabled, Time Tagger measurements can be started as usual.

When 10 GbE streaming is no longer required, call xtra_disableStreamInterfaceUDP() to return the time-tag data path to the default USB interface.

tagger.xtra_disableStreamInterfaceUDP()

Performance considerations

At high data rates, each time tag is compressed to 32 bits, enabling a maximum transfer rate of 300 MTags/s over the 10 Gbit/s link. On reception, the software expands each tag to a 128-bit internal representation, producing a sustained in-memory data rate of approximately 4.8 GB/s, a substantial volume that underlines the scale of the data being processed.

The raw time-tag transfer rate given in the hardware comparison table is measured with a minimal custom measurement that accumulates the size of each received tag block without processing individual tags. Under these conditions, the achievable rate reflects the capacity of the network adapter and the host to sustain high-speed data reception.

When measurements are running, every received time tag must be processed by each active measurement object. Running multiple measurements simultaneously means their processing costs add up within each data block, and the combined cost determines the achievable throughput. The Time Tagger measurement backend is highly optimized to handle this stream efficiently, sustaining high throughput across most measurement types even when individual measurements do not reach the maximum transfer rate.

The computational cost per tag varies substantially across measurement types. The throughput figures below were obtained on an AMD Ryzen 9 9950X with an Intel X520 PCIe adapter and should be interpreted as indicative rather than as specifications; actual values depend on hardware, operating system, and measurement parameters.

FileWriter achieves the full hardware transfer rate, independently of the number of channels. Although it does not perform per-tag analysis, it must batch time tags into blocks, compress each block, and write the result to disk. Its throughput is sensitive to thread scheduling: when simultaneous multi-threading (SMT) is enabled, the data-conversion and the file-writing threads may be placed on logical cores sharing the same physical core, reducing throughput substantially down to 200-250 MTags/s. Disabling SMT recovers the full transfer rate.

Lightweight measurements such as Counter and Countrate add minimal overhead and achieve close to the full hardware transfer rate.

Correlation is considerably more expensive. For each incoming tag, the algorithm compares it against all previously received tags on the correlated channel that fall within the correlation time window, defined as binwidth × n_bins. The computational cost per tag therefore scales with the number of historical tags retained in that window. When multiple peaks of the correlation function fall within this window, all inter-peak tags must be retained and examined for each new tag, multiplying the cost accordingly. Even in a single-peak configuration, throughput is typically limited to approximately 240 MTags/s. To maximize throughput, set the time window to the minimum value that covers the region of interest and ensure that at most one peak falls within the window.

FrequencyStability can approach the full transfer rate when the averaging parameter is large (e.g. average = 1000); throughput is slightly dependent on the number of active channels. PhaseNoise performs an FFT-based analysis; throughput improves as more channels share the computation, with a practical upper limit of approximately 200 MTags/s.

HistogramLogBins must search across all bin edges for each detected stop event, a cost that grows with the number of bins and the span of the time axis; despite a multi-threaded implementation, it remains one of the most computationally intensive common measurement types.

The achievable throughput for a given setup depends on a combination of factors: the network adapter and host hardware, the operating system, the number of active channels, the number and type of simultaneous measurements, and their individual parameters. For guidance on a specific configuration, please contact support@swabianinstruments.com.