Overview

What I wanted to learn

The R6S has two PCIe 2.5GbE controllers and one native 1GbE controller. Because those ports use different hardware and driver paths, a single aggregate benchmark would not say very much.

I wanted to compare the ports individually, in both directions, and under loads long enough to reveal queue, interrupt or stability problems. The hardware remained the same NanoPi R6S; only the boot media and software stack changed.

The mainline system ran Gentoo with Linux 6.18.38. Its two 2.5GbE controllers used the in-kernel r8169 driver, while the native controller used st_gmac. The FriendlyELEC system ran Debian 13 with Linux 6.1.141, the vendor r8125 driver on both 2.5GbE controllers, and st_gmac on the native port.

Method

Systems and physical setup

The same peers, direct cables, stream counts, test order, durations, offered UDP loads and acceptance thresholds were used for both operating systems. There was no switch in the traffic path.

Systems under test

The comparison changed the operating-system stack while retaining the same R6S and external equipment.

Table 1
ItemGentoo / mainlineFriendlyELEC stockConstant
KernelLinux 6.18.38Linux 6.1.141Same NanoPi R6S
2.5GbE driverr8169Vendor r8125 9.016.01Same two PCIe controllers
Native 1GbE driverst_gmacst_gmacSame integrated controller
Traffic tooliperf3 3.21Same version and parameters
PeersAQC113/Atlantic, RTL8125B/r8169, and Intel/igbSame peer for each port
CablesThree direct 15-foot Cat6 24AWG cablesSame cable for each port

The 2.5GbE tests used four TCP streams and four 575 Mb/s UDP streams. The 1GbE tests used two TCP streams and two 460 Mb/s UDP streams.

Qualification phases ran for 300 seconds one-way and 600 seconds bidirectional. Sustained phases ran for 1,800 seconds one-way, 3,600 seconds bidirectional and 900 seconds for UDP. Long phases were divided into 600-second evidence segments, with a 600-second cooldown between the default and optimized aggregates.

Results

Optimized throughput

After the IRQ policy was applied, the two 2.5GbE ports produced nearly identical TCP results on the two operating systems. One-way traffic reached about 2.35 Gb/s, and both directions of the bidirectional tests remained close to that level.

The 2.5GbE difference was in UDP transmitted by the R6S. Stock sustained about 2.30 Gb/s, while mainline measured about 2.00 Gb/s. Receive-side UDP was close to 2.30 Gb/s on both systems.

Optimized weighted throughput

All values are Mb/s. Bidirectional and UDP cells list TX / RX from the R6S perspective.

Table 2
PortTestMainline 6.18Stock 6.1Observation
2.5G LANTCP TX2353.72353.7Equivalent
2.5G LANTCP RX2353.62353.7Equivalent
2.5G LANBidir TX / RX2349.9 / 2347.72326.7 / 2348.1Similar
2.5G LANUDP TX / RX2002.0 / 2298.32300.0 / 2300.1Higher stock TX
2.5G WANTCP TX2353.62353.7Equivalent
2.5G WANTCP RX2353.72353.7Equivalent
2.5G WANBidir TX / RX2350.4 / 2350.32342.8 / 2350.4Similar
2.5G WANUDP TX / RX1990.5 / 2298.42300.0 / 2300.0Higher stock TX
Native 1GTCP TX544.6932.8Higher stock TX
Native 1GTCP RX941.5941.5Equivalent
Native 1GBidir TX / RX529.5 / 562.4907.9 / 936.5Higher stock throughput
Native 1GUDP TX / RX788.8 / 918.3920.0 / 920.0Higher stock TX

“Equivalent” and “similar” describe practical throughput, not exact equality at every decimal place.

The native 1GbE result

The native port behaved differently from the two PCIe ports. Receive-only TCP measured 941.5 Mb/s on both systems, but transmit-only TCP measured 932.8 Mb/s on stock and 544.6 Mb/s on mainline. The bidirectional test showed the same directional difference: 907.9 / 936.5 Mb/s on stock and 529.5 / 562.4 Mb/s on mainline.

Because the mainline receive result matched stock on the same cable and peer, I think the useful place to investigate is the R6S transmit path: st_gmac, device-tree configuration, DMA or queue setup, or clocking.

This is an inference from the direction of the results, not a demonstrated root cause.

IRQ affinity

What interrupt placement changed

The optimized policy assigned the 2.5G LAN port to CPU 5, the 2.5G WAN port to CPU 6 and the native 1GbE port to CPU 4. The largest improvements were on the receive side of the two 2.5GbE ports.

Selected stock default-to-optimized changes

Weighted Mb/s before and after the IRQ policy.

Table 3
PortTestDefaultOptimizedChange
2.5G LANTCP RX2085.62353.7+12.9%
2.5G LANBidir RX1751.72348.1+34.0%
2.5G LANUDP RX1114.02300.1+106.5%
2.5G WANTCP RX2085.62353.7+12.9%
2.5G WANBidir RX1771.52350.4+32.7%
2.5G WANUDP RX1114.02300.0+106.5%

Mainline showed the same general sensitivity. On its 2.5G LAN port, bidirectional receive increased from 1,424.6 to 2,347.7 Mb/s, and UDP receive increased from 1,518.9 to 2,298.3 Mb/s.

The native port did not materially improve under the same policy. That is why I treat IRQ placement and the native-port transmit result as separate issues.

Validation notes

Failures, counters and limitations

The validator considers throughput, packet loss and counter growth. A nonzero overall status therefore does not always mean that a throughput threshold was missed.

  • The optimized stock run cleared every configured throughput and loss threshold. Its nonzero status came from eth2 rx_missed_errors +31 and small private-counter growth.
  • The optimized mainline run missed the native 1GbE transmit and bidirectional thresholds. One 2.5GbE LAN UDP-transmit segment also measured 0.558% loss against a 0.500% ceiling.
  • The first stock attempt was excluded from the clean comparison after the AQC113 peer stopped passing frames while carrier remained up. Cycling the peer interface restored traffic. I preserved that attempt, reset the peers and reran the complete comparison from the beginning.
  • The stock image required a compatibility-only change because the vendor r8125 driver exposed stale IRQ entries in sysfs. IRQ discovery was changed to ignore entries without live affinity files under /proc. Traffic settings and acceptance thresholds were not changed.
  • Both optimized runs peaked at 65.6°C, with no evidence of thermal throttling.
  • The R6S and peer evidence archives were checksum-verified and copied off the test systems.

Practical conclusion

How I am using the result

I am keeping Gentoo and mainline Linux on the box. With IRQ pinning, the two 2.5GbE ports are essentially as fast as stock for sustained TCP and receive-side UDP, so I will keep the important routing duties on those ports.

The FriendlyELEC image remains the performance reference for the native 1GbE port and for 2.5GbE UDP transmit. My next step is to compare the stock and mainline device trees and examine the st_gmac DMA, queue and clock configuration.

The main unresolved question is the “full-rate receive, reduced transmit” pattern on the native controller. The current data identifies that pattern reliably, but it does not yet explain its cause.