The other day I was testing a 24 drive RAID with Seagate Barracuda ES 750Gig drives on a Areca 1680 SAS card. Rsyncing terabytes of data from other systems was perfect. So an order was put in to the vendor for the same RAID units. The vendor said that the Barracuda ES drives where end-of-life. They where being replaced with Barracuda ES.2 750's. I figured how much could a .2 really make a difference. I'm here to tell you A LOT!
When the units came in every piece of hardware was the same except for the newer model of the same class hard drives we have. The RAID's initialized fine. Then we started to rsync data to them. After a ~20 hours of syncing the drives started giving the error "Time Out Error" in the RAID card's logs. The card would be come unresponsive after a while. Sometimes a drive or two would drop out completely and the RAID would become degraded.
After searching and searching on this problem I came across this entry in the Areca FAQ. I also saw this in the 3ware knowledge base. Both state problems with the Seagate Barracuda ES.2 drives and a certain firmware. There was a new firmware out to fix the problem. The drives I had did not have the newer firmware so after flashing the drives with the newer firmware everything was fixed. There is a good thread about this on Seagate's site here. From the thread Seagate support does not sound very helpful with this issue.
If your using Seagate Barracuda ES.2 family drives (NS) models ST31000340NS, ST3750330NS, ST3500320NS, or ST3250310NS with firmware SN04 or 05 with a raid card and you are experiencing problems you might need to update to the AN05 firmware.
This problem really speaks to only taking the exact system you tested and never accepting any minor changes without re-testing. That was my bad.
When faced with buying a 2Gig HBA (Host Bus Adapter) like a fiber adapter or 4Gig host bus adapter you might consider buying the 4Gig HBA. Even when hooked into a 2Gig fiber switch and a 2Gig disk you will get more throughput with the 4Gig adapter. The processor on the 4Gig adapter is better made for higher bandwidths and is usually backwards compatable to 2Gig. It's a lot more expensive for a 4G compared to a 2G but at least you can use it if you upgrade to full 4G network. I've heard this also goes for 100/Mbit vs 1000Mbit network cards. Buy the 1000Mbit cards if you are going to use them on the 100Mbit network. You get better speeds because the card was made to process packets at a higher rate. You might not see much of an increase but you can be sure that network will be running at it's fastest.
Which reminds me of a question someone asked me sometime back. It was: What are the best network cards to buy for servers. Without a doubt it's the Intel server based network cards. They are known as "Intel Pro 1000" series right now. Make sure it's the server based ones not the desktop ones. The drivers cover tons of os's and were written by Intel themselves. The driver supports checksum offloading, jumbo frames, and many other great things. The BSD and Linux drivers are fantastic.
The Areca line of SATA RAID cards seem to be the best I've found. All parity calculations are done on the cards dedicated processor and it does not used the systems processor in anyway. Other cheaper cards will do this like the Rocket Raid or Broadcom cards. You may get higher throughput but at the expense of high system processor usage. Areca card processors are also very fast compared to other SATA RAID cards that use a dedicated processor on card like 3ware. I found 3ware cards to be much slower in write speeds (like twice as slow) compared to the Areca line of cards.
The speeds I've seen with bonnie++ on the Areca 1160 (16 port) with 8 Seagate ST3400832AS and 8 Seagate ST3160023AS with native command queuing (NCQ) turned off is 254MB/s writes and 388MB/s reads. For some reason if you don't turn off NCQ the your read speeds go down the drain. We are not sure why this is but I think it's a problem with the Areca cards and have reported it to Areca. The ST3160023AS don't support NCQ but the ST3400832AS's do. Not sure if mixing NCQ and non-NCQ drives on the same controller are doing it but if you leave it on read speeds go down by 60% which is nuts.
The Areca card is a 64bit PCI-X 133/MHZ RAID card. It is in a 64bit PCI-X 133/MHZ slot. The machine is a dual 3.06 Xeon with 2G of ram. In a RAID 6 setup bonnie++ was run with the following line:
sync;bonnie++ -d /disk1/ -s:80g -mAreca1160 -r2048 -x1 -u root -f -b -n 2:10000000:1000000:64 >>Areca1160
80G was used so we would saturate all the ram in the system and get a better reading of the cards actual speed. Usually 40 times the systems ram is good. You can limit the systems memory by using a kernel option at boot. So the test is shorter and your sure you can saturate the RAM. Don't for get to turn the swap off (command is just "swapoff") also! The entry to limit the memory is below. The example was done with CentOS 4.1 and a GRUB boot menu:
title CentOS (2.6.9-11.106.unsupportedsmp) Raid Test Memory Limit 64M root (hd0,0) kernel /boot/vmlinuz-2.6.9-11.106.unsupportedsmp ro root=LABEL=/ mem=64M initrd /boot/initrd-2.6.9-11.106.unsupportedsmp.img
