some quick iperf numbers for Connect-X

On this blog, we have done some testing of Connect-X using OFED 1.3.1. This is on Ubuntu 8.04, and not on RedHat. Yes, we have a mostly functional OFED 1.3.1 we distribute on JackRabbit (and soon ΔV)

I wanted to see what sort of performance we could get out of a single port DDR connection between two units.


On the server, I ran

iperf -s

and on the client I ran


iperf -c 10.201.1.101

This is a baseline single threaded test of IPoIB. IPoIB is not known to be very fast.

root@jackrabbitm-2:~# iperf -c 10.201.1.101
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size:   193 KByte (default)
------------------------------------------------------------
[  3] local 10.201.1.102 port 54571 connected with 10.201.1.101 port 5001
[  3]  0.0-10.0 sec  5.31 GBytes  4.56 Gbits/sec

This is ok, but not great for DDR IB. This is one way traffic. Now, lets try bidirectional …

root@jackrabbitm-2:~# iperf -c 10.201.1.101 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size:   645 KByte (default)
------------------------------------------------------------
[  5] local 10.201.1.102 port 52476 connected with 10.201.1.101 port 5001
[  4] local 10.201.1.102 port 5001 connected with 10.201.1.101 port 53376

[wait … wait … wait ….]

Hmmm…. may need to see what is going on with this config.

Ok, parallel unidirectional

root@jackrabbitm-2:~# iperf -c 10.201.1.101 -P 10
------------------------------------------------------------
Client connecting to 10.201.1.101, TCP port 5001
TCP window size:   193 KByte (default)
------------------------------------------------------------
[ 12] local 10.201.1.102 port 51283 connected with 10.201.1.101 port 5001
[  3] local 10.201.1.102 port 51274 connected with 10.201.1.101 port 5001
[  4] local 10.201.1.102 port 51275 connected with 10.201.1.101 port 5001
[  5] local 10.201.1.102 port 51276 connected with 10.201.1.101 port 5001
[  6] local 10.201.1.102 port 51277 connected with 10.201.1.101 port 5001
[  7] local 10.201.1.102 port 51278 connected with 10.201.1.101 port 5001
[  8] local 10.201.1.102 port 51279 connected with 10.201.1.101 port 5001
[  9] local 10.201.1.102 port 51280 connected with 10.201.1.101 port 5001
[ 10] local 10.201.1.102 port 51281 connected with 10.201.1.101 port 5001
[ 11] local 10.201.1.102 port 51282 connected with 10.201.1.101 port 5001
[  3]  0.0-10.0 sec  1.41 GBytes  1.21 Gbits/sec
[  6]  0.0-10.0 sec  1.58 GBytes  1.35 Gbits/sec
[  7]  0.0-10.0 sec  1.37 GBytes  1.18 Gbits/sec
[  8]  0.0-10.0 sec  1.45 GBytes  1.24 Gbits/sec
[ 10]  0.0-10.0 sec  1.40 GBytes  1.20 Gbits/sec
[ 11]  0.0-10.0 sec  1.47 GBytes  1.26 Gbits/sec
[ 12]  0.0-10.0 sec  1.56 GBytes  1.34 Gbits/sec
[  4]  0.0-10.0 sec  1.53 GBytes  1.32 Gbits/sec
[  5]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
[  9]  0.0-10.0 sec  1.41 GBytes  1.21 Gbits/sec
[SUM]  0.0-10.0 sec  14.9 GBytes  12.7 Gbits/sec

Ok, looks like we are not filling up the pipe. Try with 20 threads … unidirectional, and we hit the same limits.

Doesn’t prove much. Other than IPoIB is good for pinging …

[update] On the other hand …

root@jackrabbitm-1:~# ib_rdma_bw 10.201.1.102
6259: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
6259: Local address:  LID 0x02, QPN 0x2004b, PSN 0x30959a RKey 0x001a00 VAddr 0x00000000619000
6259: Remote address: LID 0x01, QPN 0x2004b, PSN 0x47d096, RKey 0x001d00 VAddr 0x00000000619000


6259: Bandwidth peak (#0 to #989): 1641.66 MB/sec
6259: Bandwidth average: 1641.56 MB/sec
6259: Service Demand peak (#0 to #989): 1189 cycles/KB
6259: Service Demand Avg  : 1189 cycles/KB

and

root@jackrabbitm-1:~# ib_rdma_lat 10.201.1.102
   local address: LID 0x02 QPN 0x4004b PSN 0xe3c106 RKey 0x2001a00 VAddr 0x00000000609001
  remote address: LID 0x01 QPN 0x4004b PSN 0x19c529 RKey 0x2001d00 VAddr 0x00000000609001
Latency typical: 1.00797 usec
Latency best   : 0.995968 usec
Latency worst  : 5.92031 usec

Thats nice. 1.008 usec.

Viewed 7397 times by 1553 viewers

Facebooktwittergoogle_plusredditpinterestlinkedinmail