... and 8 simultaneous buffered threads with 32 GB of files over iSCSI
By joe
- 8 minutes read - 1604 wordsroot@jrs8:/home/scalable/io-bm# mpirun -np 8 ./io-bm.exe -n 32 -f /big/file -w
[tid=0] each thread will output 4.000 gigabytes
[tid=0] using buffered IO
[tid=2] each thread will output 4.000 gigabytes
[tid=0] page size ... 4096 bytes
[tid=0] number of elements per buffer ... 2097152
[tid=5] each thread will output 4.000 gigabytes
[tid=5] using buffered IO
[tid=5] page size ... 4096 bytes
[tid=2] using buffered IO
[tid=2] page size ... 4096 bytes
[tid=3] each thread will output 4.000 gigabytes
[tid=3] using buffered IO
[tid=3] page size ... 4096 bytes
[tid=3] number of elements per buffer ... 2097152
[tid=3] number of buffers per file ... 256
[tid=4] each thread will output 4.000 gigabytes
[tid=4] using buffered IO
[tid=4] page size ... 4096 bytes
[tid=4] number of elements per buffer ... 2097152
[tid=6] each thread will output 4.000 gigabytes
[tid=6] using buffered IO
[tid=6] page size ... 4096 bytes
[tid=6] number of elements per buffer ... 2097152
[tid=6] number of buffers per file ... 256
[tid=6] Allocating memory ... 16777216 bytes
[tid=6] Done allocating memory
[tid=0] number of buffers per file ... 256
[tid=0] Allocating memory ... 16777216 bytes
[tid=0] Done allocating memory
[tid=0] storing random numbers ...
[tid=1] each thread will output 4.000 gigabytes
[tid=1] using buffered IO
[tid=1] page size ... 4096 bytes
[tid=1] number of elements per buffer ... 2097152
[tid=1] number of buffers per file ... 256
[tid=1] Allocating memory ... 16777216 bytes
[tid=1] Done allocating memory
[tid=1] storing random numbers ...
[tid=2] number of elements per buffer ... 2097152
[tid=2] number of buffers per file ... 256
[tid=2] Allocating memory ... 16777216 bytes
[tid=5] number of elements per buffer ... 2097152
[tid=5] number of buffers per file ... 256
[tid=5] Allocating memory ... 16777216 bytes
[tid=5] Done allocating memory
[tid=5] storing random numbers ...
[tid=7] each thread will output 4.000 gigabytes
[tid=7] using buffered IO
[tid=7] page size ... 4096 bytes
[tid=7] number of elements per buffer ... 2097152
[tid=7] number of buffers per file ... 256
[tid=4] number of buffers per file ... 256
[tid=4] Allocating memory ... 16777216 bytes
[tid=6] storing random numbers ...
[tid=7] Allocating memory ... 16777216 bytes
[tid=7] Done allocating memory
[tid=7] storing random numbers ...
[tid=2] Done allocating memory
[tid=2] storing random numbers ...
[tid=4] Done allocating memory
[tid=4] storing random numbers ...
[tid=3] Allocating memory ... 16777216 bytes
[tid=3] Done allocating memory
[tid=3] storing random numbers ...
[tid=0] Done storing random numbers
[tid=0] opening file=/big/file.0 for writing
[tid=0] file descriptor = 6
[tid=0] file open for file=/big/file.0 is complete
[tid=1] Done storing random numbers
[tid=1] opening file=/big/file.1 for writing
[tid=6] Done storing random numbers
[tid=6] opening file=/big/file.6 for writing
[tid=1] file descriptor = 6
[tid=5] Done storing random numbers
[tid=1] file open for file=/big/file.1 is complete
[tid=5] opening file=/big/file.5 for writing
[tid=6] file descriptor = 6
[tid=5] file descriptor = 6
[tid=6] file open for file=/big/file.6 is complete
[tid=5] file open for file=/big/file.5 is complete
[tid=7] Done storing random numbers
[tid=7] opening file=/big/file.7 for writing
[tid=7] file descriptor = 6
[tid=7] file open for file=/big/file.7 is complete
[tid=2] Done storing random numbers
[tid=2] opening file=/big/file.2 for writing
[tid=2] file descriptor = 6
[tid=2] file open for file=/big/file.2 is complete
[tid=4] Done storing random numbers
[tid=4] opening file=/big/file.4 for writing
[tid=4] file descriptor = 6
[tid=4] file open for file=/big/file.4 is complete
[tid=3] Done storing random numbers
[tid=3] opening file=/big/file.3 for writing
[tid=3] file descriptor = 6
[tid=3] file open for file=/big/file.3 is complete
Thread=1: time = 62.540s IO bandwidth = 65.494 MB/s
Thread=2: time = 62.553s IO bandwidth = 65.480 MB/s
Thread=3: time = 62.535s IO bandwidth = 65.499 MB/s
Thread=4: time = 62.583s IO bandwidth = 65.450 MB/s
Thread=5: time = 62.598s IO bandwidth = 65.434 MB/s
Thread=6: time = 62.617s IO bandwidth = 65.414 MB/s
Thread=0: time = 62.632s IO bandwidth = 65.397 MB/s
Thread=7: time = 62.630s IO bandwidth = 65.400 MB/s
Naive linear bandwidth summation = 523.568 MB/s
More precise calculation of Bandwidth = 523.180 MB/s
root@jrs8:/home/scalable/io-bm# mpirun -np 8 ./io-bm.exe -n 32 -f /big/file -r
[tid=2] each thread will output 4.000 gigabytes
[tid=2] using buffered IO
[tid=2] page size ... 4096 bytes
[tid=2] number of elements per buffer ... 2097152
[tid=2] number of buffers per file ... 256
[tid=2] Allocating memory ... 16777216 bytes
[tid=1] each thread will output 4.000 gigabytes
[tid=1] using buffered IO
[tid=2] Done allocating memory
[tid=2] storing random numbers ...
[tid=1] page size ... 4096 bytes
[tid=1] number of elements per buffer ... 2097152
[tid=3] each thread will output 4.000 gigabytes
[tid=3] using buffered IO
[tid=3] page size ... 4096 bytes
[tid=3] number of elements per buffer ... 2097152
[tid=4] each thread will output 4.000 gigabytes
[tid=4] using buffered IO
[tid=4] page size ... 4096 bytes
[tid=4] number of elements per buffer ... 2097152
[tid=4] number of buffers per file ... 256
[tid=4] Allocating memory ... 16777216 bytes
[tid=5] each thread will output 4.000 gigabytes
[tid=5] using buffered IO
[tid=5] page size ... 4096 bytes
[tid=5] number of elements per buffer ... 2097152
[tid=5] number of buffers per file ... 256
[tid=6] each thread will output 4.000 gigabytes
[tid=6] using buffered IO
[tid=6] page size ... 4096 bytes
[tid=0] each thread will output 4.000 gigabytes
[tid=0] using buffered IO
[tid=0] page size ... 4096 bytes
[tid=1] number of buffers per file ... 256
[tid=1] Allocating memory ... 16777216 bytes
[tid=3] number of buffers per file ... 256
[tid=3] Allocating memory ... 16777216 bytes
[tid=3] Done allocating memory
[tid=4] Done allocating memory
[tid=4] storing random numbers ...
[tid=7] each thread will output 4.000 gigabytes
[tid=7] using buffered IO
[tid=7] page size ... 4096 bytes
[tid=7] number of elements per buffer ... 2097152
[tid=7] number of buffers per file ... 256
[tid=7] Allocating memory ... 16777216 bytes
[tid=7] Done allocating memory
[tid=0] number of elements per buffer ... 2097152
[tid=0] number of buffers per file ... 256
[tid=0] Allocating memory ... 16777216 bytes
[tid=1] Done allocating memory
[tid=1] storing random numbers ...
[tid=3] storing random numbers ...
[tid=5] Allocating memory ... 16777216 bytes
[tid=5] Done allocating memory
[tid=5] storing random numbers ...
[tid=6] number of elements per buffer ... 2097152
[tid=6] number of buffers per file ... 256
[tid=6] Allocating memory ... 16777216 bytes
[tid=6] Done allocating memory
[tid=7] storing random numbers ...
[tid=6] storing random numbers ...
[tid=0] Done allocating memory
[tid=0] storing random numbers ...
[tid=4] Done storing random numbers
[tid=4] opening file=/big/file.4 for reading
[tid=4] file open for file=/big/file.4 is complete
[tid=2] Done storing random numbers
[tid=2] opening file=/big/file.2 for reading
[tid=2] file open for file=/big/file.2 is complete
[tid=1] Done storing random numbers
[tid=1] opening file=/big/file.1 for reading
[tid=1] file open for file=/big/file.1 is complete
[tid=7] Done storing random numbers
[tid=7] opening file=/big/file.7 for reading
[tid=7] file open for file=/big/file.7 is complete
[tid=3] Done storing random numbers
[tid=3] opening file=/big/file.3 for reading
[tid=3] file open for file=/big/file.3 is complete
[tid=5] Done storing random numbers
[tid=5] opening file=/big/file.5 for reading
[tid=5] file open for file=/big/file.5 is complete
[tid=6] Done storing random numbers
[tid=6] opening file=/big/file.6 for reading
[tid=6] file open for file=/big/file.6 is complete
[tid=0] Done storing random numbers
[tid=0] opening file=/big/file.0 for reading
[tid=0] file open for file=/big/file.0 is complete
Thread=4: time = 100.817s IO bandwidth = 40.628 MB/s
Thread=3: time = 100.994s IO bandwidth = 40.557 MB/s
Thread=2: time = 101.072s IO bandwidth = 40.526 MB/s
Thread=7: time = 101.082s IO bandwidth = 40.522 MB/s
Thread=1: time = 101.399s IO bandwidth = 40.395 MB/s
Thread=6: time = 101.475s IO bandwidth = 40.365 MB/s
Thread=5: time = 103.221s IO bandwidth = 39.682 MB/s
Thread=0: time = 103.447s IO bandwidth = 39.595 MB/s
Naive linear bandwidth summation = 322.269 MB/s
More precise calculation of Bandwidth = 316.761 MB/s
and for those cache lovers …
root@jrs8:/home/scalable/io-bm# mpirun -np 4 ./io-bm.exe -n 2 -f /big/file -r
[tid=0] each thread will output 0.500 gigabytes
[tid=0] using buffered IO
[tid=1] each thread will output 0.500 gigabytes
[tid=1] using buffered IO
[tid=0] page size ... 4096 bytes
[tid=0] number of elements per buffer ... 2097152
[tid=0] number of buffers per file ... 32
[tid=2] each thread will output 0.500 gigabytes
[tid=2] using buffered IO
[tid=2] page size ... 4096 bytes
[tid=2] number of elements per buffer ... 2097152
[tid=2] number of buffers per file ... 32
[tid=2] Allocating memory ... 16777216 bytes
[tid=2] Done allocating memory
[tid=2] storing random numbers ...
[tid=3] each thread will output 0.500 gigabytes
[tid=3] using buffered IO
[tid=3] page size ... 4096 bytes
[tid=3] number of elements per buffer ... 2097152
[tid=3] number of buffers per file ... 32
[tid=3] Allocating memory ... 16777216 bytes
[tid=3] Done allocating memory
[tid=3] storing random numbers ...
[tid=0] Allocating memory ... 16777216 bytes
[tid=0] Done allocating memory
[tid=0] storing random numbers ...
[tid=1] page size ... 4096 bytes
[tid=1] number of elements per buffer ... 2097152
[tid=1] number of buffers per file ... 32
[tid=1] Allocating memory ... 16777216 bytes
[tid=1] Done allocating memory
[tid=1] storing random numbers ...
[tid=0] Done storing random numbers
[tid=0] opening file=/big/file.0 for reading
[tid=0] file open for file=/big/file.0 is complete
[tid=2] Done storing random numbers
[tid=2] opening file=/big/file.2 for reading
[tid=2] file open for file=/big/file.2 is complete
[tid=3] Done storing random numbers
[tid=3] opening file=/big/file.3 for reading
[tid=3] file open for file=/big/file.3 is complete
[tid=1] Done storing random numbers
[tid=1] opening file=/big/file.1 for reading
[tid=1] file open for file=/big/file.1 is complete
Thread=3: time = 1.161s IO bandwidth = 440.876 MB/s
Thread=1: time = 1.179s IO bandwidth = 434.355 MB/s
Thread=0: time = 1.183s IO bandwidth = 432.923 MB/s
Thread=2: time = 1.183s IO bandwidth = 432.944 MB/s
Naive linear bandwidth summation = 1741.097 MB/s
More precise calculation of Bandwidth = 1731.691 MB/s
Ah…. cache …