I keep hearing this. Its not true though. See below.
NFS client: Scalable Informatics Delta-V (ΔV) 4 unit
NFS server: Scalable Informatics JackRabbit 4 unit.
(you can buy these units today from Scalable Informatics and its partners)
10GbE: single XFP fibre between two 10GbE NICs.
This is NOT a clustered NFS result.
root@dv4:~# mount | grep data2 10.1.3.1:/data on /data2 type nfs (rw,intr,rsize=262144,wsize=262144,tcp,addr=10.1.3.1) root@dv4:~# mpirun -np 4 ./io-bm.exe -n 32 -f /data2/test/file -r -d -v N=32 gigabytes will be written in total each thread will output 8.000 gigabytes page size ... 4096 bytes number of elements per buffer ... 2097152 number of buffers per file ... 512 Thread=3: time = 33.665s IO bandwidth = 243.337 MB/s Thread=2: time = 33.910s IO bandwidth = 241.580 MB/s Thread=1: time = 34.262s IO bandwidth = 239.101 MB/s Thread=0: time = 34.244s IO bandwidth = 239.226 MB/s Naive linear bandwidth summation = 963.244 MB/s More precise calculation of Bandwidth = 956.404 MB/s
Whats interesting about this is that this is only 3/4 of the wire speed. I think we were running into contention on the client machine (4 threads contending for a single 10GbE port).
Should be able to do 1.2 GB/s or very nearly this.