some quick iperf numbers for Connect-X
By joe
- 3 minutes read - 602 wordsOn this blog, we have done some testing of Connect-X using OFED 1.3.1. This is on Ubuntu 8.04, and not on RedHat. Yes, we have a mostly functional OFED 1.3.1 we distribute on JackRabbit (and soon ΔV) I wanted to see what sort of performance we could get out of a single port DDR connection between two units.
On the server, I ran
iperf -s
and on the client I ran
iperf -c 10.201.1.101
This is a baseline single threaded test of IPoIB. IPoIB is not known to be very fast.
root@jackrabbitm-2:~# iperf -c 10.201.1.101
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size: 193 KByte (default)
------------------------------------------------------------
[ 3] local 10.201.1.102 port 54571 connected with 10.201.1.101 port 5001
[ 3] 0.0-10.0 sec 5.31 GBytes 4.56 Gbits/sec
This is ok, but not great for DDR IB. This is one way traffic. Now, lets try bidirectional …
root@jackrabbitm-2:~# iperf -c 10.201.1.101 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size: 645 KByte (default)
------------------------------------------------------------
[ 5] local 10.201.1.102 port 52476 connected with 10.201.1.101 port 5001
[ 4] local 10.201.1.102 port 5001 connected with 10.201.1.101 port 53376
[wait … wait … wait ….] Hmmm…. may need to see what is going on with this config. Ok, parallel unidirectional
root@jackrabbitm-2:~# iperf -c 10.201.1.101 -P 10
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size: 193 KByte (default)
------------------------------------------------------------
[ 12] local 10.201.1.102 port 51283 connected with 10.201.1.101 port 5001
[ 3] local 10.201.1.102 port 51274 connected with 10.201.1.101 port 5001
[ 4] local 10.201.1.102 port 51275 connected with 10.201.1.101 port 5001
[ 5] local 10.201.1.102 port 51276 connected with 10.201.1.101 port 5001
[ 6] local 10.201.1.102 port 51277 connected with 10.201.1.101 port 5001
[ 7] local 10.201.1.102 port 51278 connected with 10.201.1.101 port 5001
[ 8] local 10.201.1.102 port 51279 connected with 10.201.1.101 port 5001
[ 9] local 10.201.1.102 port 51280 connected with 10.201.1.101 port 5001
[ 10] local 10.201.1.102 port 51281 connected with 10.201.1.101 port 5001
[ 11] local 10.201.1.102 port 51282 connected with 10.201.1.101 port 5001
[ 3] 0.0-10.0 sec 1.41 GBytes 1.21 Gbits/sec
[ 6] 0.0-10.0 sec 1.58 GBytes 1.35 Gbits/sec
[ 7] 0.0-10.0 sec 1.37 GBytes 1.18 Gbits/sec
[ 8] 0.0-10.0 sec 1.45 GBytes 1.24 Gbits/sec
[ 10] 0.0-10.0 sec 1.40 GBytes 1.20 Gbits/sec
[ 11] 0.0-10.0 sec 1.47 GBytes 1.26 Gbits/sec
[ 12] 0.0-10.0 sec 1.56 GBytes 1.34 Gbits/sec
[ 4] 0.0-10.0 sec 1.53 GBytes 1.32 Gbits/sec
[ 5] 0.0-10.0 sec 1.67 GBytes 1.44 Gbits/sec
[ 9] 0.0-10.0 sec 1.41 GBytes 1.21 Gbits/sec
[SUM] 0.0-10.0 sec 14.9 GBytes 12.7 Gbits/sec
Ok, looks like we are not filling up the pipe. Try with 20 threads … unidirectional, and we hit the same limits. Doesn’t prove much. Other than IPoIB is good for pinging … [update] On the other hand …
root@jackrabbitm-1:~# ib_rdma_bw 10.201.1.102
6259: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
6259: Local address: LID 0x02, QPN 0x2004b, PSN 0x30959a RKey 0x001a00 VAddr 0x00000000619000
6259: Remote address: LID 0x01, QPN 0x2004b, PSN 0x47d096, RKey 0x001d00 VAddr 0x00000000619000
6259: Bandwidth peak (#0 to #989): 1641.66 MB/sec
6259: Bandwidth average: 1641.56 MB/sec
6259: Service Demand peak (#0 to #989): 1189 cycles/KB
6259: Service Demand Avg : 1189 cycles/KB
and
root@jackrabbitm-1:~# ib_rdma_lat 10.201.1.102
local address: LID 0x02 QPN 0x4004b PSN 0xe3c106 RKey 0x2001a00 VAddr 0x00000000609001
remote address: LID 0x01 QPN 0x4004b PSN 0x19c529 RKey 0x2001d00 VAddr 0x00000000609001
Latency typical: 1.00797 usec
Latency best : 0.995968 usec
Latency worst : 5.92031 usec
Thats nice. 1.008 usec.