What we would like to do is to even out those spurts of actual disk
writing to be more consistent. The application takes a relatively
small portion of the 100 ms per image to process, so it should have
quite a bit of time to write out an image to disk. But it seems to
just sit there at times, content to let the disk cache build up. Does
anyone know where we might look to tune something or change some
setting that would get us more consistent disk writing? Thanks!

Have you experimented with the different I/O schedulers?

On 2009-07-29, Jonathan <berr... at (no spam) gmail.com> wrote:

What we would like to do is to even out those spurts of actual disk
writing to be more consistent.  The application takes a relatively
small portion of the 100 ms per image to process, so it should have
quite a bit of time to write out an image to disk.  But it seems to
just sit there at times, content to let the disk cache build up.  Does
anyone know where we might look to tune something or change some
setting that would get us more consistent disk writing?  Thanks!

Have you experimented with the different I/O schedulers?

You may need to experiment, but I think the deadline I/O scheduler might
be what you need.

On Wed, 29 Jul 2009, Jonathan wrote:

[...300KB images at 10 Hz...]

What we are seeing is that in most cases, it takes 1-3 ms to "write"
each image file.  Since this is obviously much faster than it is
actually possible to write 300 kB of data, it is clear it is making
good use of RAM to cache the file before actually writing it to the CF
disk.  However, sometimes it will take much longer to complete the
image write call, from a few hundred ms to a few seconds.  Using
things like vmstat, iostat, and /proc/meminfo, we have been able to
see that Linux is writing out the buffered data to disk in spurts.
When it starts to get "behind" it will block the application and wait
for some data to be written out to disk before continuing.

What we would like to do is to even out those spurts of actual disk
writing to be more consistent.  The application takes a relatively
small portion of the 100 ms per image to process, so it should have
quite a bit of time to write out an image to disk.  But it seems to
just sit there at times, content to let the disk cache build up.  Does
anyone know where we might look to tune something or change some
setting that would get us more consistent disk writing?  Thanks!

As you are writing much more than the cache on your embedded device can
hold why use the cache at all?  I it were real small chunks you would
win something because of write combining but in your case I don't see any
benefit at all.

So I would use open(,O_SYNC), create another thread in your application,
pass images to it via some sort of queue and let it do the writing.  You
need the queue because the CF disk itself could sometimes stop to do some
housekeeping but otherwise you get a constant write rate.

Vitus

What we are seeing is that in most cases, it takes 1-3 ms to "write"
each image file. Since this is obviously much faster than it is
actually possible to write 300 kB of data, it is clear it is making
good use of RAM to cache the file before actually writing it to the CF
disk. However, sometimes it will take much longer to complete the
image write call, from a few hundred ms to a few seconds. Using
things like vmstat, iostat, and /proc/meminfo, we have been able to
see that Linux is writing out the buffered data to disk in spurts.
When it starts to get "behind" it will block the application and wait
for some data to be written out to disk before continuing.

What we would like to do is to even out those spurts of actual disk
writing to be more consistent. The application takes a relatively
small portion of the 100 ms per image to process, so it should have
quite a bit of time to write out an image to disk. But it seems to
just sit there at times, content to let the disk cache build up. Does
anyone know where we might look to tune something or change some
setting that would get us more consistent disk writing? Thanks!

You should divide image recording and writing the data to the disk into
separate threads.

Better way would be to have separated processes

And you really want to play with shared memory, process synchronizing and so
on. Uhhhh...

Vladimir Jovic wrote:

Linux is not really that good with threads.

Since when?

Better way would be to have separated processes

And you really want to play with shared memory, process synchronizing and so
on. Uhhhh...

Juergen Beisert wrote:
Vladimir Jovic wrote:

[cut]

Linux is not really that good with threads.

Since when?

My experience on linux threads is limited, therefore the problem with
threads is most likely on my side

Better way would be to have separated processes

And you really want to play with shared memory, process synchronizing and
so on. Uhhhh...


Exactly. I refuse to play with mutexes, semaphores, threads
synchronisation, data accesses, etc.

Read this why I made that suggestion :
http://en.wikipedia.org/wiki/Unix_philosophy

On Aug 5, 10:29 am, Juergen Beisert <jbeis... at (no spam) netscape.net> wrote:

You should divide image recording and writing the data to the disk
into separate threads.

Better way would be to have separated processes

And you really want to play with shared memory, process synchronizing and
so on. Uhhhh...

It should not be too hard to invent your own, but don't most realtime
frameworks gives you shared memory fifo mechanisms to solve this
problem?

Linux is not really that good with threads. B

cs_posting at (no spam) hotmail.com wrote:
On Aug 5, 10:29 am, Juergen Beisert <jbeis... at (no spam) netscape.net> wrote:

You should divide image recording and writing the data to the disk
into separate threads.
Better way would be to have separated processes
And you really want to play with shared memory, process synchronizing and
so on. Uhhhh...
It should not be too hard to invent your own, but don't most realtime
frameworks gives you shared memory fifo mechanisms to solve this
problem?

Sure. But if you decouple the realtime part from the not-realtime part in
such way, it doesn't matter if your are using two threads or two processes
to do the work. Hmmm, but using two threads do not need a context switch,
so they might be faster on embedded systems with low computing power (and
no cache flush like ARM architecture for example).

Vladimir Jovic wrote:

Read this why I made that suggestion :
http://en.wikipedia.org/wiki/Unix_philosophy

Its okay to write regular programs in this manner. But don't do so, if you
try to write some program that also has to meet special timing specs. How
do you want to predict the behaviour of your program if it can block when
it tries to write some data into a pipe? Or into a file on a disk? Or on
the screen? How long will this pause take? You can't predict it. So, you
can't write programs with strict timing requirements in this way.

Juergen Beisert wrote:
cs_posting at (no spam) hotmail.com wrote:
On Aug 5, 10:29 am, Juergen Beisert <jbeis... at (no spam) netscape.net> wrote:

You should divide image recording and writing the data to the disk
into separate threads.
Better way would be to have separated processes
And you really want to play with shared memory, process synchronizing
and so on. Uhhhh...
It should not be too hard to invent your own, but don't most realtime
frameworks gives you shared memory fifo mechanisms to solve this
problem?

Sure. But if you decouple the realtime part from the not-realtime part in
such way, it doesn't matter if your are using two threads or two
processes to do the work. Hmmm, but using two threads do not need a
context switch, so they might be faster on embedded systems with low
computing power (and no cache flush like ARM architecture for example).

I have read somewhere (long time ago, now have no idea where) that
context switching in linux is almost instant, with neglectful
performances loss. Therefore your remark about context switching doesn't
hold

Juergen Beisert wrote:
Vladimir Jovic wrote:

[...]

Read this why I made that suggestion :
http://en.wikipedia.org/wiki/Unix_philosophy

Its okay to write regular programs in this manner. But don't do so, if
you try to write some program that also has to meet special timing specs.
How do you want to predict the behaviour of your program if it can block
when it tries to write some data into a pipe? Or into a file on a disk?
Or on the screen? How long will this pause take? You can't predict it.
So, you can't write programs with strict timing requirements in this way.


Before writing to a pipe, use select() with a timeout to see if it is
possible. If the pipe is full for some short time, the process can
either sleep for some time and retry, or it can report an error. It
doesn't have to be blocked.

As threads, processes can have RT priority as well.

btw in the same way the process would be blocked, your thread would be
blocked as well. If it happens, I would call it a bad design, or a bug.