More JackRabbit 96TB OpenSolaris system

By joe

July 17, 2009 - 2 minutes read - 334 words

even more JackRabbit OpenSolaris 2009.06 goodness.

landman@pgda-100:~$ ssh scalable@192.168.1.74
Password:
Last login: Fri Jul 17 13:32:57 2009
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
scalable@jr5-96TB:~$ su
Password:
scalable@jr5-96TB:~# zpool create tank c7t1d0 c10t2d0 c11t3d0
scalable@jr5-96TB:~# zpool status tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          c7t1d0    ONLINE       0     0     0
          c10t2d0   ONLINE       0     0     0
          c11t3d0   ONLINE       0     0     0
errors: No known data errors
scalable@jr5-96TB:~# df -h /tank
Filesystem             size   used  avail capacity  Mounted on
tank                    70T    19K    70T     1%    /tank

70TB of ZFS on JR5-96Tn. [updated] to answer the questions on pricing, yes, this is less than $1 USD/usable-GB for this system. And for some simple speed pr0n

/usr/bin/time dd if=/dev/zero of=/tank/big.file ...
1024+0 records in
1024+0 records out
real       14.8
user        0.0
sys        14.0

16GB in 14.8 seconds. 1.1GB/s Unfortunately, this is cached. So the next result shouldn’t surprise you.

scalable@jr5-96TB:~# /usr/bin/time dd if=/tank/big.file of=/dev/null ...
1024+0 records in
1024+0 records out
real        4.2
user        0.0
sys         4.2

4.2 s to read 16GB. Or 3.8 GB/s out of cache. Lets try 128 GB, and see what happens.

scalable@jr5-96TB:~# /usr/bin/time dd if=/dev/zero of=/tank/big.file ...
8192+0 records in
8192+0 records out
real     2:23.3
user        0.0
sys      2:07.6

Excellent. This is 143.3 seconds, for 890 MB/s way outside cache. Lets try the read version of this …

scalable@jr5-96TB:~# /usr/bin/time dd if=/tank/big.file of=/dev/null ...
8192+0 records in
8192+0 records out
real     2:17.5
user        0.0
sys      1:31.8

This is 137.5 seconds to read 128 GB. Reads at 931 MB/s. You can see the Linux results here on this exact same hardware in a previous post or two, if you need comparisons. Why OpenSolaris 2009.06 and not Solaris? We can’t put benchmark numbers out about the latter, but can with the former. Why not NexentaStor? We have to get a supported ethernet card into the machine for that. The igb driver appears to be too new for their 2.0 release. I’ll bug them about this.