 |
|
| .NET DotNet Forum Index » Visual C# Forum » runaway thread count and asynchronous sockets |
|
Page 1 of 1 |
|
| Author |
Message |
| Matthew Groch |
 Posted: Fri Oct 01, 2004 9:58 am |
|
|
|
Guest
|
Hi all,
I've got a server that handles a relatively high number of concurrent
transactions (on the magnitude of 1000's per second). Client
applications establish socket connections with the server. Data is
sent and received over these connections using the asynchronous model.
The server is currently in beta testing. Sporadically over the course
of the day, I'll observe the thread count on the process (via perfmon)
start climbing. (Let's say that on average, a 'normal' thread count is
around 80). But when it climbs, well.. Today, for example, I saw it
peak at around 355 threads.
Now, I don't explicitly spawn these threads in my code. I'm assuming
they're being spawned by the CLR to handle the high volume of async
socket sends. The count stays high for some (arbitrary?) period of
time and then, automagically, comes back down to normal levels again.
When the thread count is high, the system is obviously taxed and
performance goes down. I've written code specifically so that I keep a
flag associated with a socket session such that a session will not
initiate an async send if one is currently in progress, if that's
worth mentioning at all.
I'm hoping to hear from _anyone_ that might have encountered a similar
situation!! I'm not exactly sure what to do at this point. I'd love
any help and am particularly looking for responses like "Hey, have you
thought about this?" or "Hmm, maybe you're misunderstanding some
fundamental concept in async socket comm", or best, "Yeah, that sounds
similar to something we ran into last year; here's the strategy we
used to get around it.."
Thanks in advance- |
|
|
| Back to top |
|
|
|
| Richard |
| Posted: Fri Oct 01, 2004 1:23 pm |
|
|
|
Guest
|
FYI - I have a very similiar scenario to yours; I'm running a C# service on
a 2 HT Zeon processor box. I have a bunch of custom perfmon counters in code
and I know that ThreadPool worker thread counts stay right around 100 for me;
I dunno what IO completion thread counts are as I haven't had any issues with
them. Be aware that the system often needs a 'shadow thread' for every
thread you spin off so high thread counts in and of themselves are not always
an issue...
Quote: Data is
sent and received over these connections using the asynchronous model.
How exactly are you using asynch? Fire and forget or BeginReceive and
sleep? In my case my service receives 3 seperate pieces of data serially
before acting on the data, so I do 3 BeginReceive/sleep's in a row with my IO
completion handler setting the IO completion event each time. In each case I
sleep on IO completion and/or a service command event. This is working very
well for me.
Also be aware that your IO completion handler should catch BOTH
SocketException and generic Exception --> if the socket is closed by a worker
thread within your code {aka if you shutdown your program or service and some
other worker thread closes the socket} the IO completion callback will fire
with a generic Exception and a message that reads "socket disposed" aka:
// Called by ThreadPool IO completion thread
void ReceiveIOCompletionCallback(IAsyncResult ar)
{
try
{
// wait for IO to complete then signal BeginReceive caller thread
((Socket)ar.AsyncState).EndReceive(ar);
((ManualResetEvent)ar.AsyncWaitHandle).Set();
}
catch (SocketException ex)
{
// get here if remote client closes socket or other expected error...
}
catch (Exception ex)
{
// get here if another of your threads disposes socket
}
}
Quote: 'normal' thread count is
around 80). But when it climbs, well.. Today, for example, I saw it
peak at around 355 threads.
Now, I don't explicitly spawn these threads in my code. I'm assuming
they're being spawned by the CLR to handle the high volume of async
socket sends. The count stays high for some (arbitrary?) period of
time and then, automagically, comes back down to normal levels again.
If you are seeing runaway thread counts it could be several things. Are
they IO completion threads or are they worker threads?
If they are IO completion threads then you might be ok. In the IO
completion callback code above an IO completion thread is active and it
counts against the process as an active thread - the thread sleeps until the
IO completes but it will count as an active thread. Your high thread counts
might just mean that you have many IO requests pending and that there are
many IO completion threads sleeping while IO is in progress.
-OR- You could be getting a socket error or partial IO completion and many
IO completion threads are active to service the completed operations. If you
are doing partial read/write operations then you could see high thread counts
if operations are partially completing. In my stuff I don't allow partial IO
completion - I begin a write/read operation and sleep until the IO fully
completes.
On the other hand if your high thread counts are due to your worker threads
then invariably you have an issue with the logic related to spinning off a
thread - some condition is triggering your logic to spin off many threads -
you'll probably want to fix that...
Quote: The count stays high for some (arbitrary?) period of
time and then, automagically, comes back down to normal levels again.
When the thread count is high, the system is obviously taxed and
performance goes down.
How long until the automagic brings thread counts back down? A few seconds
or something along the lines of a socket timeout?
Quote: I've written code specifically so that I keep a
flag associated with a socket session such that a session will not
initiate an async send if one is currently in progress, if that's
worth mentioning at all.
Why only one at a time? One great advantage of asynch IO is having many
things happening at the same time. You can queue up several reads and
writes, they are guaranteed to be serviced in the order in which they are
queued...
In my stuff I have coded a 'waitable semaphore' as a throttling mechanism.
All socket activity is throttled by the configurable maximum number of
threads the semaphore will permit. At different points in code threads wait
for IO completion, a semaphore reference, and/or a service command or any
arbitrary combination of those events...
--Richard |
|
|
| Back to top |
|
|
|
|
|
All times are GMT - 5 Hours
The time now is Sun Nov 29, 2009 2:41 am
|
|