We are using an ext3 filesystem with defaults mount on top of LVM + MD
raid 6. The raid-6 is on 12 disks (hence it is 10 disks for data, 2 for
parity). 6 of those disks are through the mobo controller, the others
are through the 3ware.

I hoped I would get something like 1 GB/sec sequential write on 10 disks
instead I see MUCH lower performances

U comp.arch.storage kkkk <kkkk at (no spam) bbbb.com> prica:
We are using an ext3 filesystem with defaults mount on top of LVM + MD
raid 6. The raid-6 is on 12 disks (hence it is 10 disks for data, 2 for
parity). 6 of those disks are through the mobo controller, the others
are through the 3ware.

First of all, you've got best of breed 3Ware 9650SE controller which has the
best RAID6 of all SATA RAID controllers... And you're mixing it with onboard
controller to produce the software RAID6?! WHY?!!!

Second, 12 drives for RAID6 is suboptimal...

Go with 8 or 16 drives and attach them directly to 3Ware...

If you really want, concatenate them via LVM, but I won't suggest
it...

Other thing, RAID6 has double distributed parity (like RAID5 does)... So,
every drive is for data, and every drive is used for parity...

I hoped I would get something like 1 GB/sec sequential write on 10 disks
instead I see MUCH lower performances

With your configuration - very unlikely! :/

Had we found a cheap no-raid 16 drives SATA controller for PCI-Express
we would have bought it. If you know of any, please tell me.

We are a research entity and the funding comes at unknown times. We
decided to build the system so that any component can be replaced with
any other similar component available in shops at any time, e.g. the
9650SE can be replaced with multiple controllers in the future. If the
3ware breaks in the future, at that time there might not be a compatible
controller in production. (We cannot buy from ebay!)

With the current setup, in any emergency, the disks can be connected via
any controller or even USB, and the MD linux raid will still work and we
will be able to get data out. Furthermore we trust visible, open,
old/tested, linux MD code more than any embedded RAID code which nobody
knows except 3ware. What if there was a bug in 9650SE code? It was a
recent controller when we bought it, and we would have found out only
later, maybe years later after setting up our array.
Also, we were already proficient with linux MD.

Anyway since linux MD raid never occupies more than 35% CPU (of a single
core!) in any test, I don't think it is the bottleneck. But this is part
of my question.

Second, 12 drives for RAID6 is suboptimal...

WHY??

Go with 8 or 16 drives and attach them directly to 3Ware...

We have already lots of data and virtual machines loaded in there. Even
if it was possible to attach all to 3ware controller (actually it might
indeed be possible, since it is a 16ML [we have 24 drives on the
machine]), we wouldn't have used the RAID from 3ware for the reasons
explained above.

With MD raid it shouldn't make a difference unless you say that the
larger cache in the 3ware speeds up the operation. This is again part of
my question: the cache seems to have a dramatic effect which I do not
completely understand for sequential I/O. It must be something related
to the bus overhead or the context switching of the CPU (for serving the
interrupts) but I would like a confirmation. Also consider that with 8
cores and a PCI express bus, both overheads should have been negligible.
Anyway the cache from the drives should be enough to minimize this
overhead (I mean for the MOBO drives) so I would not expect a tremendous
speedup from using the 3ware cache for all the drives (I mean still with
MD).

I hoped I would get something like 1 GB/sec sequential write on 10 disks
instead I see MUCH lower performances

With your configuration - very unlikely! :/

What performance would you expect from 3ware raid-6 12-disks with ext3
(defaults mount) sequential dd write?

Hi there,
we have a big "storage" computer: dual Xeon E5345 at (no spam) 2.33GHz (8 cores
total) with lots of disks partly connected to a 3ware 9650SE controller
and partly to the SATA/SAS controllers in the mobo.
The hard disks are Western Digital WD7500AYYS 750GB

We are using an ext3 filesystem with defaults mount on top of LVM + MD
raid 6. The raid-6 is on 12 disks (hence it is 10 disks for data, 2 for
parity). 6 of those disks are through the mobo controller, the others
are through the 3ware.

If you have so many disks connected, did you consider having at least
one as a hot spare?

Secondly, did you consider raid 10 as an alternative?

Any writes that don't cover
a complete stripe will need reads from several of the disks,

Anyway since linux MD raid never occupies more than 35% CPU (of a single
core!) in any test, I don't think it is the bottleneck. But this is part
of my question.

Calypso seems especially ignorant when talking about optimal RAID group
sizes. Perhaps he's confusing RAID-5/6 with RAID-3 - but even then he'd
be wrong, since what you really want with RAID-3 is for the total *data*
content (excluding parity) of a stripe to be a convenient value, meaning
that you tend to favor group sizes like 5 or 9 (not counting any spares
that may be present). And given that you've got both processing power
and probably system/memory bus bandwidth to burn, there's no reason why
a software RAID-6 implementation shouldn't perform fairly competitively
with a hardware one.

U comp.arch.storage Bill Todd <billtodd at (no spam) metrocast.net> prica:
Calypso seems especially ignorant when talking about optimal RAID group
sizes. Perhaps he's confusing RAID-5/6 with RAID-3 - but even then he'd
be wrong, since what you really want with RAID-3 is for the total *data*
content (excluding parity) of a stripe to be a convenient value, meaning
that you tend to favor group sizes like 5 or 9 (not counting any spares
that may be present). And given that you've got both processing power
and probably system/memory bus bandwidth to burn, there's no reason why
a software RAID-6 implementation shouldn't perform fairly competitively
with a hardware one.

RAID3 implementation doesn't exist on 3Ware controllers... So far, I've seen
RAID3 only on some storage arrays (EMC to tell you the truth only, can check
the others), and the old SCSI RAID controller - AMI MegaRAID Enterprise
1300 that I once had...

RAID3 is very similar to RAID5, but hasn't got distributed parity drive,
instead it has got dedicated parity drive, and, yes, it's used for special
purposes only where big sequential read/write speed is needed... But, like I
said already, 3Ware doesn't support it...

Seems like I was partially right with 8 or 16 drives as a optimal number of
drives... Seems like that for RAID6 it's optimal to have 6, 10 or 18 drives
(4+2, 8+2, 16+2)... Here's a nice text from EMC guy (look at Stripe size of
a LUN):

http://clariionblogs.blogspot.com/

I haven't tested RAID6 on 9650SE, but have tested RAID5 on 9650SE (older
generation on PCI-X), and IIRC got around 250MB/s write from 15x160GB
Hitachi 7200rpm SATA drives... So, with this 9650SE I expect at least around
350MB/s from 16 drives (today's SATA)...

What filesystem and operating system? This is important...

I assume you mean "first write of a sequential file"..?
(overwrites as you see are much faster)

Consider that bandwidth is not what
you'll be worried about, it's more to RAID6 write penalty that cache memory
annulates (it's 6 IOPS per write)...

"6 IOPS per write"? Could you explain this?

I still don't understand. Why 4+2, 8+2, 16+2 should be more optimal?
Please note that one raid chunk is NOT one block long (512 bytes). In
facts on my raid-6 it is 64KB long, so the stripes are 64*10=640KB long.
What's wrong with that? Why should that be less performing than 512KB
long? Please note that ext3 has blocksize 4K, so there are 160 and 128
ext3 blocks in one stripe respectively in the two configurations. I
don't see why 128 blocks should be significantly better than 160 blocks..!?

David Schwartz wrote:
It is the bottleneck, it's just not a CPU bottleneck, it's an I/O
bottleneck.

With an 8x PCI-e bus there should be space for 2 GB/sec transfer...

What filesystem and operating system? This is important...

Windows XP, NTFS...

I suspected that. I suspect NTFS is much faster than ext3, it will
probably be like XFS in Linux. (and also more unsafe e.g. in case of
power losses, just like XFS) Speed depends among other things on how
paranoid is the journal behaviour.

It is the bottleneck, it's just not a CPU bottleneck, it's an I/O
bottleneck.

The problem is simply the number of I/Os the system has to
issue. With a 12 disk RAID 6 array implemented in software, a write of
a single byte (admittedly the worst case) will require 10 reads
followed by 12 writes that cannot be started until all 10 reads
complete. Each of these operations has to be started and completed by
the MD driver.