Its all in how you do the IO ...

JackRabbit 5U (JR5) 96TB unit, with 8 threads writing to the same file (each one writing to a different section of the file to reduce contention). write performance below.

[root@jr5 ~]# mpirun -np 8 ./io-bm.exe -n 128 -f /data/file -w  -s -d -v
N=128 gigabytes will be written in total
each thread will output 16.000 gigabytes
page size                     ... 4096 bytes
number of elements per buffer ... 2097152
number of buffers per file    ... 1024
[tid=1] file name = /data/file
[tid=3] file name = /data/file
[tid=4] file name = /data/file
[tid=0] file name = /data/file
[tid=5] file name = /data/file
[tid=6] file name = /data/file
[tid=7] file name = /data/file
[tid=2] file name = /data/file
Thread=1: time = 51.819s IO bandwidth = 316.179 MB/s
Thread=2: time = 51.824s IO bandwidth = 316.149 MB/s
Thread=3: time = 51.828s IO bandwidth = 316.123 MB/s
Thread=4: time = 51.832s IO bandwidth = 316.099 MB/s
Thread=5: time = 51.837s IO bandwidth = 316.065 MB/s
Thread=6: time = 51.843s IO bandwidth = 316.033 MB/s
Thread=7: time = 51.848s IO bandwidth = 315.999 MB/s
Thread=0: time = 51.848s IO bandwidth = 315.999 MB/s
Naive linear bandwidth summation = 2528.646 MB/s
More precise calculation of Bandwidth = 2527.989 MB/s

For laughs, lets write 256GB to the same file as well

[root@jr5 ~]# mpirun -np 8 ./io-bm.exe -n 256 -f /data/file -w -d  -s  -v
N=256 gigabytes will be written in total
each thread will output 32.000 gigabytes
page size                     ... 4096 bytes
number of elements per buffer ... 2097152
number of buffers per file    ... 2048
[tid=0] file name = /data/file
[tid=5] file name = /data/file
[tid=1] file name = /data/file
[tid=2] file name = /data/file
[tid=7] file name = /data/file
[tid=6] file name = /data/file
[tid=3] file name = /data/file
[tid=4] file name = /data/file
Thread=1: time = 102.272s IO bandwidth = 320.402 MB/s
Thread=2: time = 102.276s IO bandwidth = 320.389 MB/s
Thread=3: time = 102.281s IO bandwidth = 320.372 MB/s
Thread=4: time = 102.220s IO bandwidth = 320.563 MB/s
Thread=5: time = 102.297s IO bandwidth = 320.321 MB/s
Thread=6: time = 102.294s IO bandwidth = 320.330 MB/s
Thread=0: time = 102.307s IO bandwidth = 320.290 MB/s
Thread=7: time = 102.299s IO bandwidth = 320.317 MB/s
Naive linear bandwidth summation = 2562.983 MB/s
More precise calculation of Bandwidth = 2562.321 MB/s

I want to delve into this more deeply at some point.