Seems like I was partially right with 8 or 16 drives as a optimal number of
drives... Seems like that for RAID6 it's optimal to have 6, 10 or 18 drives
(4+2, 8+2, 16+2)... Here's a nice text from EMC guy (look at Stripe size of
a LUN):

http://clariionblogs.blogspot.com/

I did spend a few minutes in Google trying to find detailed
information about md's RAID-6 implementation but got nowhere.
...
There is a lot more information about linux raid5 than raid6.

What performance would you expect from 3ware raid-6 12-disks with ext3
(defaults mount) sequential dd write?

I haven't tested RAID6 on 9650SE, but have tested RAID5 on 9650SE (older
generation on PCI-X), and IIRC got around 250MB/s write from 15x160GB
Hitachi 7200rpm SATA drives... So, with this 9650SE I expect at least around
350MB/s from 16 drives (today's SATA)...

Consider that bandwidth is not what
you'll be worried about, it's more to RAID6 write penalty that cache memory
annulates (it's 6 IOPS per write)...

U comp.arch.storage kkkk <kkkk at (no spam) bbbb.com> prica:
I haven't tested RAID6 on 9650SE, but have tested RAID5 on 9650SE (older
generation on PCI-X), and IIRC got around 250MB/s write from 15x160GB
Hitachi 7200rpm SATA drives... So, with this 9650SE I expect at least around
350MB/s from 16 drives (today's SATA)...

What filesystem and operating system? This is important...

Windows XP, NTFS...

NTFS unsafe in case of power loss?

You missed something, we're not talking about FAT here (which is faster than NTFS)...

U comp.arch.storage Bill Todd <billtodd at (no spam) metrocast.net> prica:
Calypso seems especially ignorant when talking about optimal RAID group
sizes. Perhaps he's confusing RAID-5/6 with RAID-3 - but even then he'd
be wrong, since what you really want with RAID-3 is for the total *data*
content (excluding parity) of a stripe to be a convenient value, meaning
that you tend to favor group sizes like 5 or 9 (not counting any spares
that may be present). And given that you've got both processing power
and probably system/memory bus bandwidth to burn, there's no reason why
a software RAID-6 implementation shouldn't perform fairly competitively
with a hardware one.

RAID3 implementation doesn't exist on 3Ware controllers...

RAID3 is very similar to RAID5

Seems like I was partially right with 8 or 16 drives as a optimal number of
drives

(4+2, 8+2, 16+2)... Here's a nice text from EMC guy (look at Stripe size of
a LUN):

http://clariionblogs.blogspot.com/

David Brown wrote:

I did spend a few minutes in Google trying to find detailed
information about md's RAID-6 implementation but got nowhere.
...
There is a lot more information about linux raid5 than raid6.

You mean on *Linux MD* raid5? That could be good. Where?

Raid-6 algorithms are practically equivalent to raid-5, except parity
computation obviously .

Why do you think dd stays at 100% CPU? (with disks/3ware caches enabled)
Shouldn't that be 0%?

Do you think the CPU is high due to a memory-copy operation? If it was
that, I suppose dd from /dev/zero to /dev/null should go at 200MB/sec
instead it goes at 1.1GB/sec (with 100%CPU occupation indeed, 65% of
which is in kernel mode). That would mean that the number of copies
performed by dd while copying to the ext3-raid is 5 times greater than
that for copying from /dev/zero to /dev/null . Hmmm... a bit difficult
to believe. there must be other stuff performed in the ext3 case so to
hog the CPU. Is the ext3 code running whithin the dd process when dd writes?

In my case dd pushes 5 seconds of data before the disks start writing
(dirty_writeback_centiseces = 500). dd stays always at least 5 seconds
ahead of the writes. This should fill all stripes completely causing no
reads. I even tried to raise the dirty_writeback_centisecs with no
measurable performance benefit.

Where is this 5secs of data stored? Is it at the ext3 layer or at the
LVM layer (I doubt this one, also I notice there is no LVM kernel thread
runing) or at the MD layer?

Why do you think dd stays at 100% CPU? (with disks/3ware caches enabled)
Shouldn't that be 0%?

Do you think the CPU is high due to a memory-copy operation? If it was
that, I suppose dd from /dev/zero to /dev/null should go at 200MB/sec
instead it goes at 1.1GB/sec (with 100%CPU occupation indeed, 65% of
which is in kernel mode). That would mean that the number of copies
performed by dd while copying to the ext3-raid is 5 times greater than
that for copying from /dev/zero to /dev/null . Hmmm... a bit difficult
to believe. there must be other stuff performed in the ext3 case so to
hog the CPU. Is the ext3 code running whithin the dd process when dd writes?

Hmm probably not because kjournald had significant CPU occupation. What
is the role of the journal during file overwrites?

Yes. And when you're making statements that depend upon the details,
it's really a good thing to have understood them first.

8+2 drives are 10... 16+2 drives are 18... 8 drives are optimal...

You clearly didn't understand why that's incorrect the first time I
explained it. Perhaps you should just keep reading my previous response
until the light dawns rather than continue to babble on incompetently.

Fine, but, if everything is aligned with Base2, then why go around it?

What matters is request *alignment*, not whether the size of the data in a
full stripe is a power of 2. This usually means request alignment with
respect to the size of a single disk's data stripe segment, so that the
minimum number of disks is required to participate in any access request.
Since access request sizes are often a power of 2 this means that the size
of a single disk's stripe segment should usually be a power of 2, but says
nothing about how many disks should be in the stripe.

Well, yes: you still don't know what you're talking about, and appear
to be very resistant to becoming better-educated about it. Do try to
change both before you respond again.

Interestingly (and contrary to most of the articles on the Web that
credit Ouchi with the first description of RAID-5 - let alone the more
prevalent articles that seem to believe that all the RAID the concepts
originated at Berkeley much later) Ouchi's algorithm appears to have
been RAID-4 (see last sentence of first paragraph quoted above). The
small-write approach described above, however, applies equally to
RAID-5, so clearly existed well before Patterson described it. The IBM
patent cited above for a RAID-5-style algorithm was filed on 06/12/1986
(with no indication of how long IBM had been working on the concept
before then) and contains none of the names of the Berkeley RAID team -
but IBM was working with/sponsoring them at around that time so that's
likely where the Berkeley group picked it up.

What matters is request *alignment*, not whether the size of the data in a
full stripe is a power of 2. This usually means request alignment with
respect to the size of a single disk's data stripe segment, so that the
minimum number of disks is required to participate in any access request.
Since access request sizes are often a power of 2 this means that the size
of a single disk's stripe segment should usually be a power of 2, but says
nothing about how many disks should be in the stripe.

So basically, if I say that stripe segment is 64kb, it means that when I
write 512kb of data and have 12 drives, I simply use 8 drives at a time, and
the rest of 4 drives are not used (forget about parity drives now)?

What happens if I try to write 1kb data in a 64kb stripe segment using 4kb
blocks in NTFS (let's do this as an example)?

to make you angry...

So basically, if I say that stripe segment is 64kb, it means that when I
write 512kb of data and have 12 drives, I simply use 8 drives at a time, and
the rest of 4 drives are not used (forget about parity drives now)?

In a conventional RAID-5 you can allocate and write to space at the same
granularity you can to a disk (currently a single 512-byte sector). The
sectors are numbered consecutively within each stripe segment and
continuing across each stripe (leaving out the parity segment) and then
continue on to the next stripe. So if you read or write a 512 KB
request (aligned to the start of a stripe segment, rather than just
aligned to an arbitrary single sector boundary) it will indeed hit 8
logically consecutive 64 KB stripe segments.

In a 12-drive array such a 512 KB write will be optimized such that
instead of reading each modified segment, creating the XOR with the new
data, and then reading/XORing with/updating the relevant parity segment
it will just write the 8 modified segments, read in the 3 unmodified
data segments remaining in the stripe, create the full-stripe data XOR,
and update the parity segment (this is the normal optimization applied
when about half or more of the segments in a stripe are modified). If
not all the modified segments fall in the same stripe it will update the
two affected stripes separately (applying that optimization to one of
them if applicable).

In that example it would be more efficient to use a far larger stripe
segment. If, for example, you used a 512 KB stripe segment and the
access was aligned to that granularity you'd only need to read in the
old data and parity, XOR the old data with the new data, XOR the result
with the old parity, and write out the new data and the new parity: 4
disk accesses, and though each would be 512 KB instead of 64 KB this
would less than double the duration of each access resulting in less
than 2/3 the total array overhead that the 12 accesses in the optimized
(and aligned) initial case above required. In both situations unaligned
accesses increase the overhead, but for comparable lack of alignment the
large stripe segment still usually comes out ahead (and is at least as
efficient at servicing small requests, as explained below).

Not really: I actually prefer answering reasonable questions to
correcting persistent misinformation. Unfortunately, there seems to be
a lot more of the latter than of the former ever since "Generation Me"
came around (self-discipline doesn't seem to be their forte).

U comp.arch.storage Bill Todd <billtodd at (no spam) metrocast.net> prica:
So basically, if I say that stripe segment is 64kb, it means that when I
write 512kb of data and have 12 drives, I simply use 8 drives at a time, and
the rest of 4 drives are not used (forget about parity drives now)?

In a conventional RAID-5 you can allocate and write to space at the same
granularity you can to a disk (currently a single 512-byte sector). The
sectors are numbered consecutively within each stripe segment and
continuing across each stripe (leaving out the parity segment) and then
continue on to the next stripe. So if you read or write a 512 KB
request (aligned to the start of a stripe segment, rather than just
aligned to an arbitrary single sector boundary) it will indeed hit 8
logically consecutive 64 KB stripe segments.

So if you align cache page size with stripe size, you can benefit from it or
no?

2kb stripe segment size... If you dump cache, you basically write to all
drives at once, right? But this situation can slow down everything since
you've got how many IOPS per one write operation (>?

In that example it would be more efficient to use a far larger stripe
segment. If, for example, you used a 512 KB stripe segment and the
access was aligned to that granularity you'd only need to read in the
old data and parity, XOR the old data with the new data, XOR the result
with the old parity, and write out the new data and the new parity: 4
disk accesses, and though each would be 512 KB instead of 64 KB this
would less than double the duration of each access resulting in less
than 2/3 the total array overhead that the 12 accesses in the optimized
(and aligned) initial case above required. In both situations unaligned
accesses increase the overhead, but for comparable lack of alignment the
large stripe segment still usually comes out ahead (and is at least as
efficient at servicing small requests, as explained below).

Thinking..... So, you need to optimize the cache of a RAID controller to
gather changed data so that it could be written in one dump (utilize all
actuators at once)?

So if you align cache page size with stripe size, you can benefit from it or
no?

If there's no concurrency at all (i.e., no request is submitted until the
previous one completes) you're best off using full-stripe writes (and for
that matter RAID-3 rather than RAID-5) - because there's no way you can
get more than a single disk's worth of IOPS out of the array and you
should ensure that nothing causes it to be even worse than that.

If there's a lot of concurrency, you're best off minimizing the resources
that each operation uses, which usually means large stripe segment sizes -
so large that even a single segment will tend to be larger than most
requests (thus the entire stripe will be *much* larger than almost any
request): that way, each read typically is satisfied by one disk access
and each write by 4 disk accesses spread across 2 disks (3 if the old data
still happens to be in cache), so N reads or N/2 writes can proceed in
parallel with virtually no worse latency for reads and only about twice
the latency for writes as would happen with full-stripe RAID-3 accesses.

So you wind up with N times the potential read throughput with minimal
latency penalty and N/4 times the write throughput without dramatic
latency penalty - compared with the *no-load* full-stripe case. But if
having only effectively one disk's worth of IOPS for full-stripe accesses
would have resulted in significant request queuing delays, you may have a
lot *better* latency (as well as throughput) for both reads and writes
when you minimize the number of disks involved in each one by using a
large stripe segment size.