RDMA, iWARP and Linux

Fell down a rabbit hole today when I was looking at a data sheet for the Broadcom 5709c chipset. "RDMA over TCP (iWARP) - RDMAC 1.0 compliant". Huh?

This blog has a good overview of RDMA:

The rationale for RDMA is laid out in great detail in RFC 4297, but the basic idea is that allowing network messages to carry information about where they should be received and allowing the NIC to place the data directly in that buffer allows fundamentally better performance. [...]

With RDMA andiSCSI Extensions for (iSER, which is RFC 5046), the target can send the data in response to a read command and have it placed directly in the receive buffer on the initiator, which saves the copy and uses 3x less memory bandwidth (which is huge if the data is running at 10Gb/sec).

But there's more in that post, like this bit of drama on the LKML from 2007:

How about we just remove the RDMA stack altogether? I am not at all kidding. If you guys can't stay in your sand box and need to cause problems for the normal network stack, it's unacceptable. We were told all along the if RDMA went into the tree none of this kind of stuff would be an issue.

These are exactly the kinds of problems for which people like myself were dreading. These subsystems have no buisness using the TCP port space of the Linux software stack, absolutely none.

After TCP port reservation, what's next? It seems an at least bi-monthly event that the RDMA folks need to put their fingers into something else in the normal networking stack. No more.

I will NACK any patch that opens up sockets to eat up ports or anything stupid like that.

From Dave Miller, no less. Of course, that was four years ago (shit!). Now where do things stand?

Well, there's this announcement from IBM of a Linux iWARP driver and user library. And there's NFS over RDMA (is that the right term?) in the kernel now, including iWARP support.

I'm not sure if this'll be useful at work, but it's interesting to read about.