Using link type CSMA seems to significantly limit throughput when you increase the link delay

[jafingerhut]

I was experimeting with varying the parameters of the following example command given for using P4sim with the p4lang/tutorials repository:

./ns3 run p4-v1model-ipv4-forwarding -- \
  --pktSize=1000 --appDataRate=100Mbps --linkRate=1000Mbps \
  --switchRate=100000 --linkDelay=0.01ms --simDuration=20 --pcap=true

I noticed that the throughput was near 1000 Mbps with those parameters, but it went down significantly when increasing the linkDelay parameter, e.g. to 1ms.

Looking further, it appears that perhaps a behavior of the CSMA link type is that after it transmits a packet on the link, the link transitions to a TRANSMITTING state for the duration of the link delay, which prevents anyone from transmitting on the link until that link delay has elapsed.

That might be how some CSMA links behave, but CSMA Ethernet links are uncommon in data centers these days. Most Ethernet links are point-to-point, and senders can send a subsequent packet immediately after completing the transmission of the previous packet.

This issue is just to point out this property of the current example code for P4sim, and to ask whether because of this, you might want to consider changing the csma links to point-to-point links, which are also available in the ns-3 set of available link types?

[Mingyumaz]

• For the throughput, I need some time to have a check, it should be an issue/bug for P4sim, the decrease in throughput should not be caused only by the CSMA channel.
• The CSMA channel in ns-3 is an older model with half-duplex mode; p4sim could also use a modified p2p channel. But this p2p channel is just a simple full-duplex without the mathematical modeling (Error model & delay model). Currently, most examples use the CSMA channels.
• Actually, the ns-3 doesn't have a switch model, so neither does it have a full-duplex channel for ethernet swtich. But the ns-3 org also wants to have a ethernet swtich model, so there is a GSOC topic for Switched_Ethernet this year.

[jafingerhut]

• For the throughput, I need some time to have a check, it should be an issue/bug for P4sim, the decrease in throughput should not be caused only by the CSMA channel.

The traffic in this case is UDP, so I don't think there is any host-to-host round trip time that should cause the throughput to decrease, as there might be with TCP or other ack-based transport protocols. I searched through ns-3 code in the csma implementation, and it definitely does seem to impose this waiting time on all senders after a frame is sent, waiting until it is received before anyone can start sending a subsequent frame on the csma channel again. If I am correct in that, then this would easily cause a significant loss in throughput when one increases the link delay.

I don't see any way to avoid that other than (a) changing ns-3 csma channel implementation to avoid this property, which seems like a change the ns-3 developers are unlikely to accept, or (b) using some channel in P4sim topologies that does not have this sender limitation, which I believe the point-to-point links should not.

[Mingyumaz]

Based on our testing, using the p2p channel resolves the issue. I will continue with further adjustments and test additional examples, as most current use cases rely on the csma channel.