pbzip2, how do I love thee? Let me count the ways ...

dstat output on a 10GB pbzip2 compressed file being uncompressed … with pbzip2.

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 49   3  42   5   0   0|  35M   95M|3168B 6978B|   0    28k|1575  1198
 52   4  39   5   0   0|  39M  188M|2508B 5230B|   0    80k|1769  1475
 51   4  40   5   0   0|  19M  206M|4686B 9390B|   0     0 |2396  2240
 42   4  48   5   0   0|  31M  158M|3054B 5360B|   0    16k|1820  2025
 50   5  40   5   0   0|  37M  115M|2640B 5360B|   0   104k|1731  1564
 38   4  50   8   0   0|  24M  105M|3102B 6270B|   0     0 |1639  1477 ^C

Run … don’t walk … to get pbzip2. Replace plain old bzip2 with it. Finally … to be able to use all my cores for compression … and decompression …

top - 18:15:33 up 35 days,  4:52,  7 users,  load average: 6.33, 3.49, 1.52
Tasks: 515 total,   2 running, 513 sleeping,   0 stopped,   0 zombie
Cpu0  : 49.5%us,  7.6%sy,  0.0%ni,  0.0%id, 41.0%wa,  0.0%hi,  1.9%si,  0.0%st
Cpu1  : 35.8%us, 13.2%sy,  0.0%ni, 50.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 46.7%us,  3.8%sy,  0.0%ni, 49.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 49.5%us,  2.9%sy,  0.0%ni, 47.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 52.1%us,  1.0%sy,  0.0%ni, 46.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  : 50.0%us,  2.3%sy,  0.0%ni, 47.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 50.0%us,  1.1%sy,  0.0%ni, 48.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 53.7%us,  1.1%sy,  0.0%ni, 45.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16470552k total, 16356996k used,   113556k free,       48k buffers
Swap: 19824104k total,    20700k used, 19803404k free, 14293356k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16273 landman   20   0  216m 9168  960 S   54  0.1   1:58.64 pbunzip2
16274 landman   20   0  216m 9168  960 R   52  0.1   1:53.44 pbunzip2
16277 landman   20   0  216m 9168  960 S   52  0.1   1:58.78 pbunzip2
16271 landman   20   0  216m 9168  960 S   51  0.1   1:56.79 pbunzip2
16276 landman   20   0  216m 9168  960 S   51  0.1   1:55.49 pbunzip2
16270 landman   20   0  216m 9168  960 S   49  0.1   1:58.22 pbunzip2
16275 landman   20   0  216m 9168  960 S   44  0.1   1:56.68 pbunzip2
16272 landman   20   0  216m 9168  960 S   38  0.1   1:55.12 pbunzip2
16278 landman   20   0  216m 9168  960 S   11  0.1   0:19.33 pbunzip2

Seriously … this is good stuff. Jeff Gilchrist has done good work here.