root@jrs8:/home/scalable/io-bm# mpirun -np 8 ./io-bm.exe -n 32 -f /big/file -w [tid=0] each thread will output 4.000 gigabytes [tid=0] using buffered IO [tid=2] each thread will output 4.000 gigabytes [tid=0] page size ... 4096 bytes [tid=0] number of elements per buffer ... 2097152 [tid=5] each thread will output 4.000 gigabytes [tid=5] using buffered IO [tid=5] page size ... 4096 bytes [tid=2] using buffered IO [tid=2] page size ... 4096 bytes [tid=3] each thread will output 4.000 gigabytes [tid=3] using buffered IO [tid=3] page size ... 4096 bytes [tid=3] number of elements per buffer ... 2097152 [tid=3] number of buffers per file ... 256 [tid=4] each thread will output 4.000 gigabytes [tid=4] using buffered IO [tid=4] page size ... 4096 bytes [tid=4] number of elements per buffer ... 2097152 [tid=6] each thread will output 4.000 gigabytes [tid=6] using buffered IO [tid=6] page size ... 4096 bytes [tid=6] number of elements per buffer ... 2097152 [tid=6] number of buffers per file ... 256 [tid=6] Allocating memory ... 16777216 bytes [tid=6] Done allocating memory [tid=0] number of buffers per file ... 256 [tid=0] Allocating memory ... 16777216 bytes [tid=0] Done allocating memory [tid=0] storing random numbers ... [tid=1] each thread will output 4.000 gigabytes [tid=1] using buffered IO [tid=1] page size ... 4096 bytes [tid=1] number of elements per buffer ... 2097152 [tid=1] number of buffers per file ... 256 [tid=1] Allocating memory ... 16777216 bytes [tid=1] Done allocating memory [tid=1] storing random numbers ... [tid=2] number of elements per buffer ... 2097152 [tid=2] number of buffers per file ... 256 [tid=2] Allocating memory ... 16777216 bytes [tid=5] number of elements per buffer ... 2097152 [tid=5] number of buffers per file ... 256 [tid=5] Allocating memory ... 16777216 bytes [tid=5] Done allocating memory [tid=5] storing random numbers ... [tid=7] each thread will output 4.000 gigabytes [tid=7] using buffered IO [tid=7] page size ... 4096 bytes [tid=7] number of elements per buffer ... 2097152 [tid=7] number of buffers per file ... 256 [tid=4] number of buffers per file ... 256 [tid=4] Allocating memory ... 16777216 bytes [tid=6] storing random numbers ... [tid=7] Allocating memory ... 16777216 bytes [tid=7] Done allocating memory [tid=7] storing random numbers ... [tid=2] Done allocating memory [tid=2] storing random numbers ... [tid=4] Done allocating memory [tid=4] storing random numbers ... [tid=3] Allocating memory ... 16777216 bytes [tid=3] Done allocating memory [tid=3] storing random numbers ... [tid=0] Done storing random numbers [tid=0] opening file=/big/file.0 for writing [tid=0] file descriptor = 6 [tid=0] file open for file=/big/file.0 is complete [tid=1] Done storing random numbers [tid=1] opening file=/big/file.1 for writing [tid=6] Done storing random numbers [tid=6] opening file=/big/file.6 for writing [tid=1] file descriptor = 6 [tid=5] Done storing random numbers [tid=1] file open for file=/big/file.1 is complete [tid=5] opening file=/big/file.5 for writing [tid=6] file descriptor = 6 [tid=5] file descriptor = 6 [tid=6] file open for file=/big/file.6 is complete [tid=5] file open for file=/big/file.5 is complete [tid=7] Done storing random numbers [tid=7] opening file=/big/file.7 for writing [tid=7] file descriptor = 6 [tid=7] file open for file=/big/file.7 is complete [tid=2] Done storing random numbers [tid=2] opening file=/big/file.2 for writing [tid=2] file descriptor = 6 [tid=2] file open for file=/big/file.2 is complete [tid=4] Done storing random numbers [tid=4] opening file=/big/file.4 for writing [tid=4] file descriptor = 6 [tid=4] file open for file=/big/file.4 is complete [tid=3] Done storing random numbers [tid=3] opening file=/big/file.3 for writing [tid=3] file descriptor = 6 [tid=3] file open for file=/big/file.3 is complete Thread=1: time = 62.540s IO bandwidth = 65.494 MB/s Thread=2: time = 62.553s IO bandwidth = 65.480 MB/s Thread=3: time = 62.535s IO bandwidth = 65.499 MB/s Thread=4: time = 62.583s IO bandwidth = 65.450 MB/s Thread=5: time = 62.598s IO bandwidth = 65.434 MB/s Thread=6: time = 62.617s IO bandwidth = 65.414 MB/s Thread=0: time = 62.632s IO bandwidth = 65.397 MB/s Thread=7: time = 62.630s IO bandwidth = 65.400 MB/s Naive linear bandwidth summation = 523.568 MB/s More precise calculation of Bandwidth = 523.180 MB/s root@jrs8:/home/scalable/io-bm# mpirun -np 8 ./io-bm.exe -n 32 -f /big/file -r [tid=2] each thread will output 4.000 gigabytes [tid=2] using buffered IO [tid=2] page size ... 4096 bytes [tid=2] number of elements per buffer ... 2097152 [tid=2] number of buffers per file ... 256 [tid=2] Allocating memory ... 16777216 bytes [tid=1] each thread will output 4.000 gigabytes [tid=1] using buffered IO [tid=2] Done allocating memory [tid=2] storing random numbers ... [tid=1] page size ... 4096 bytes [tid=1] number of elements per buffer ... 2097152 [tid=3] each thread will output 4.000 gigabytes [tid=3] using buffered IO [tid=3] page size ... 4096 bytes [tid=3] number of elements per buffer ... 2097152 [tid=4] each thread will output 4.000 gigabytes [tid=4] using buffered IO [tid=4] page size ... 4096 bytes [tid=4] number of elements per buffer ... 2097152 [tid=4] number of buffers per file ... 256 [tid=4] Allocating memory ... 16777216 bytes [tid=5] each thread will output 4.000 gigabytes [tid=5] using buffered IO [tid=5] page size ... 4096 bytes [tid=5] number of elements per buffer ... 2097152 [tid=5] number of buffers per file ... 256 [tid=6] each thread will output 4.000 gigabytes [tid=6] using buffered IO [tid=6] page size ... 4096 bytes [tid=0] each thread will output 4.000 gigabytes [tid=0] using buffered IO [tid=0] page size ... 4096 bytes [tid=1] number of buffers per file ... 256 [tid=1] Allocating memory ... 16777216 bytes [tid=3] number of buffers per file ... 256 [tid=3] Allocating memory ... 16777216 bytes [tid=3] Done allocating memory [tid=4] Done allocating memory [tid=4] storing random numbers ... [tid=7] each thread will output 4.000 gigabytes [tid=7] using buffered IO [tid=7] page size ... 4096 bytes [tid=7] number of elements per buffer ... 2097152 [tid=7] number of buffers per file ... 256 [tid=7] Allocating memory ... 16777216 bytes [tid=7] Done allocating memory [tid=0] number of elements per buffer ... 2097152 [tid=0] number of buffers per file ... 256 [tid=0] Allocating memory ... 16777216 bytes [tid=1] Done allocating memory [tid=1] storing random numbers ... [tid=3] storing random numbers ... [tid=5] Allocating memory ... 16777216 bytes [tid=5] Done allocating memory [tid=5] storing random numbers ... [tid=6] number of elements per buffer ... 2097152 [tid=6] number of buffers per file ... 256 [tid=6] Allocating memory ... 16777216 bytes [tid=6] Done allocating memory [tid=7] storing random numbers ... [tid=6] storing random numbers ... [tid=0] Done allocating memory [tid=0] storing random numbers ... [tid=4] Done storing random numbers [tid=4] opening file=/big/file.4 for reading [tid=4] file open for file=/big/file.4 is complete [tid=2] Done storing random numbers [tid=2] opening file=/big/file.2 for reading [tid=2] file open for file=/big/file.2 is complete [tid=1] Done storing random numbers [tid=1] opening file=/big/file.1 for reading [tid=1] file open for file=/big/file.1 is complete [tid=7] Done storing random numbers [tid=7] opening file=/big/file.7 for reading [tid=7] file open for file=/big/file.7 is complete [tid=3] Done storing random numbers [tid=3] opening file=/big/file.3 for reading [tid=3] file open for file=/big/file.3 is complete [tid=5] Done storing random numbers [tid=5] opening file=/big/file.5 for reading [tid=5] file open for file=/big/file.5 is complete [tid=6] Done storing random numbers [tid=6] opening file=/big/file.6 for reading [tid=6] file open for file=/big/file.6 is complete [tid=0] Done storing random numbers [tid=0] opening file=/big/file.0 for reading [tid=0] file open for file=/big/file.0 is complete Thread=4: time = 100.817s IO bandwidth = 40.628 MB/s Thread=3: time = 100.994s IO bandwidth = 40.557 MB/s Thread=2: time = 101.072s IO bandwidth = 40.526 MB/s Thread=7: time = 101.082s IO bandwidth = 40.522 MB/s Thread=1: time = 101.399s IO bandwidth = 40.395 MB/s Thread=6: time = 101.475s IO bandwidth = 40.365 MB/s Thread=5: time = 103.221s IO bandwidth = 39.682 MB/s Thread=0: time = 103.447s IO bandwidth = 39.595 MB/s Naive linear bandwidth summation = 322.269 MB/s More precise calculation of Bandwidth = 316.761 MB/s
and for those cache lovers …
root@jrs8:/home/scalable/io-bm# mpirun -np 4 ./io-bm.exe -n 2 -f /big/file -r [tid=0] each thread will output 0.500 gigabytes [tid=0] using buffered IO [tid=1] each thread will output 0.500 gigabytes [tid=1] using buffered IO [tid=0] page size ... 4096 bytes [tid=0] number of elements per buffer ... 2097152 [tid=0] number of buffers per file ... 32 [tid=2] each thread will output 0.500 gigabytes [tid=2] using buffered IO [tid=2] page size ... 4096 bytes [tid=2] number of elements per buffer ... 2097152 [tid=2] number of buffers per file ... 32 [tid=2] Allocating memory ... 16777216 bytes [tid=2] Done allocating memory [tid=2] storing random numbers ... [tid=3] each thread will output 0.500 gigabytes [tid=3] using buffered IO [tid=3] page size ... 4096 bytes [tid=3] number of elements per buffer ... 2097152 [tid=3] number of buffers per file ... 32 [tid=3] Allocating memory ... 16777216 bytes [tid=3] Done allocating memory [tid=3] storing random numbers ... [tid=0] Allocating memory ... 16777216 bytes [tid=0] Done allocating memory [tid=0] storing random numbers ... [tid=1] page size ... 4096 bytes [tid=1] number of elements per buffer ... 2097152 [tid=1] number of buffers per file ... 32 [tid=1] Allocating memory ... 16777216 bytes [tid=1] Done allocating memory [tid=1] storing random numbers ... [tid=0] Done storing random numbers [tid=0] opening file=/big/file.0 for reading [tid=0] file open for file=/big/file.0 is complete [tid=2] Done storing random numbers [tid=2] opening file=/big/file.2 for reading [tid=2] file open for file=/big/file.2 is complete [tid=3] Done storing random numbers [tid=3] opening file=/big/file.3 for reading [tid=3] file open for file=/big/file.3 is complete [tid=1] Done storing random numbers [tid=1] opening file=/big/file.1 for reading [tid=1] file open for file=/big/file.1 is complete Thread=3: time = 1.161s IO bandwidth = 440.876 MB/s Thread=1: time = 1.179s IO bandwidth = 434.355 MB/s Thread=0: time = 1.183s IO bandwidth = 432.923 MB/s Thread=2: time = 1.183s IO bandwidth = 432.944 MB/s Naive linear bandwidth summation = 1741.097 MB/s More precise calculation of Bandwidth = 1731.691 MB/s
Ah…. cache …
Viewed 3780 times by 668 viewers
[...] I hauled out something I wrote a while ago, for precisely these purposes (I’ll get it onto our external Mercurial repository soon). Push [...]