 |
|
| Linux Forum Index » Linux Development - System » thread memory size... |
|
Page 2 of 4 Goto page Previous 1, 2, 3, 4 Next |
|
| Author |
Message |
| Jan Helgesen... |
Posted: Tue Oct 06, 2009 8:03 am |
|
|
|
Guest
|
Rainer Weikusat wrote:
Quote:
That's your merciful assumption :->. I am fairly convinced that this
is part of the stern wave of this
http://www.barrelfish.org/
You lost me now, but thats ok. Because I found an article stating how
threads and kernel memory work. Linux don't have its own threads
structure its only got something called a task_structure, which is
treated equally to processes by the scheduler. The only difference is
that the task structure is part of an existing process, so the task
structure does not need to contain too much data, as all the rest is
properly set up by the process structures. That mean a thread in linux
is quite lightweight and it also explains why the new thread
implementation in Linux can start 200K threads in 2 seconds. Which I am
happy about, now I just need to figure out how to configure Linux to be
able to run that many threads So now I can start testing this kind of
massive parallelism on a Linux machine, of course not with 1 million
threads, yet. But I can have some fun with it.
(It interesting how I was only asking how much kernel memory linux
requires per thread. And as any person discussing on Usenet, I will
argue my case. But for some reason this discussion has shifted into a
completely new theme, which is some sort of bashing against what you
don't like or believe in. I have no problems with what you believe in or
like. I only ask that you answer my questions in a serious matter,
instead of trying to belittle things that you don't agree with. The
world was once flat...
(I am not saying I am Columbus nor Copernicus, I am just saying you dont
know everything and should therefore give ideas some reasonable amount
respect until you do understand it.))
regards
Jan |
|
|
| Back to top |
|
|
|
| Rainer Weikusat... |
Posted: Tue Oct 06, 2009 8:18 am |
|
|
|
Guest
|
Jan Helgesen <spam at (no spam) nospam.com> writes:
Quote: Rainer Weikusat wrote:
That's your merciful assumption :->. I am fairly convinced that this
is part of the stern wave of this
http://www.barrelfish.org/
You lost me now, but thats ok.
The time coincedence is striking.
Quote: Because I found an article stating how threads and kernel memory
work. Linux don't have its own threads structure its only got
something called a task_structure, which is treated equally to
processes by the scheduler. The only difference is that the task
structure is part of an existing process, so the task structure does
not need to contain too much data, as all the rest is properly set
up by the process structures.
There is no 'process structure'. The kernel scheduler deals with
'tasks', as represtented by a struct task_struct and these 'tasks' may
share all kinds of resources, eg open files tables, address spaces,
filesystem namespace etc, see clone( for details. Depending on what
is and what isn't shared, 'tasks' can be used to provide both POSIX
threads and UNIX(*) processes.
[...]
Quote: (It interesting how I was only asking how much kernel memory linux
requires per thread.
And I had answered this question, hadn't you immediatly started to try
to lecture me about 'proper terms' based on your (almost complete)
lack of knowledge in this area. You still haven't found either the
amount of kernel memory used by a thread (which is mostly the
associated kernel stack, default 8K, configurable 4K) nor the problem
with your other limit: the virtual address space. Each thread needs a
separate part of that for its stack.
Quote: And as any person discussing on Usenet, I will
argue my case. But for some reason this discussion has shifted into a
completely new theme, which is some sort of bashing against what you
don't like or believe in.
This is called 'argueing against something', eg your absolutely
ridicolous idea of the 256-way-SMP-problem or your misconception about
inter-thread communication (which is usually done in exactll the way
you desribed as 'new idea people would resent for irrational
reasons'). You are, of course, not even trying to provide counter
arguments (and given that you were simply wrong in both the cases I
mentioned, that would be somewhat difficult), instead you chose to
bash people (me, in this case) based on my style of writing.
Quote: I have no problems with what you believe in
or like. I only ask that you answer my questions in a serious matter,
instead of trying to belittle things that you don't agree with. The
world was once flat...
Your world is still flat. BTW, please stop this 'posted and mailed'
nonsense. |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Tue Oct 06, 2009 10:10 am |
|
|
|
Guest
|
Ok, so if you think I have misunderstood it all and I barely have any
knowledge about anything, I will give you a chance to explain,
preferably in different wording than you already have, what you think I
am misunderstanding so completely.
regards
Jan |
|
|
| Back to top |
|
|
|
| Rainer Weikusat... |
Posted: Tue Oct 06, 2009 10:14 am |
|
|
|
Guest
|
Jan Helgesen <spam at (no spam) nospam.com> writes:
Quote: Ok, so if you think I have misunderstood it all and I barely have any
knowledge about anything, I will give you a chance to explain,
preferably in different wording than you already have, what you think
I am misunderstanding so completely.
Since you have again sent me an e-mail from your fake address and have
(apparently) completely ignored the other parts of my posting, namely,
the technical explanations, as well, why would I repeat myself? You
aren't listening, anyway. |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Tue Oct 06, 2009 10:25 am |
|
|
|
Guest
|
Rainer Weikusat wrote:
Quote: Jan Helgesen <spam at (no spam) nospam.com> writes:
Ok, so if you think I have misunderstood it all and I barely have any
knowledge about anything, I will give you a chance to explain,
preferably in different wording than you already have, what you think
I am misunderstanding so completely.
Since you have again sent me an e-mail from your fake address and have
(apparently) completely ignored the other parts of my posting, namely,
the technical explanations, as well, why would I repeat myself? You
aren't listening, anyway.
That's just an pitiful excuse. If you have a compelling argument then
spit it out. Don't be a jerk and try to pretend you know something when
you don't want to share it. I am listening if you can be bothered to say
something without shouting, screaming and mocking people. If not then I
can only remain with my current conclusion of your arguments.
regards
Jan |
|
|
| Back to top |
|
|
|
| Chris Friesen... |
Posted: Tue Oct 06, 2009 2:16 pm |
|
|
|
Guest
|
On 10/06/2009 10:25 AM, Jan Helgesen wrote:
Quote: Rainer Weikusat wrote:
Jan Helgesen <spam at (no spam) nospam.com> writes:
Ok, so if you think I have misunderstood it all and I barely have any
knowledge about anything, I will give you a chance to explain,
preferably in different wording than you already have, what you think
I am misunderstanding so completely.
Since you have again sent me an e-mail from your fake address and have
(apparently) completely ignored the other parts of my posting, namely,
the technical explanations, as well, why would I repeat myself? You
aren't listening, anyway.
That's just an pitiful excuse. If you have a compelling argument then
spit it out.
He has. Go back and read the information in this thread. The answer to
your question (as well as several corrections to your misconceptions and
the introduction of several issues which you likely haven't considered)
has already been covered.
Chris |
|
|
| Back to top |
|
|
|
| David Schwartz... |
Posted: Tue Oct 06, 2009 7:16 pm |
|
|
|
Guest
|
On Oct 6, 5:32 pm, Jan Helgesen <s... at (no spam) nospam.com> wrote:
Quote: 3) Shared vm has a high performance cost.
what cost is that, and the high performance cost is compared to what?
Anything that manipulates the vm has to be synchronized. Consider, for
example, opening a file. Consider mapping a new region of memory.
Also, since they have to be in the same SMP region, they typically
have to all be talking to a single kernel. So many kernel operations
will wind up imposing some kind of synchronization cost.
The whole point of message passing for scalability is that you don't
have to use a shared vm. You can distribute your large operation over
multiple memory domains. Shared vm among thousands of cores make
absolutely no sense at all for the foreseeable future.
DS |
|
|
| Back to top |
|
|
|
| David Schwartz... |
Posted: Tue Oct 06, 2009 10:04 pm |
|
|
|
Guest
|
On Oct 7, 12:24 am, Jan Helgesen <s... at (no spam) nospam.com> wrote:
Quote: The whole point of message passing for scalability is that you don't
have to use a shared vm. You can distribute your large operation over
multiple memory domains. Shared vm among thousands of cores make
absolutely no sense at all for the foreseeable future.
I am having problems understanding what you are saying here.
Apparently.
Quote: First off, when you say vm, you mean virtual memory, right?
Wikipedia defines virtual memory as:
Yes.
Quote: "Virtual memory is a computer system technique which gives an
application program the impression that it has contiguous working memory
(an address space), while in fact it may be physically fragmented and
may even overflow on to disk storage."
So virtual memory is not a process or programs complete address space,
Umm, yes it is. The process or program's complete address space is its
vm. (Perhaps you're not familiar with this usage, but it's very
common.)
Quote: it is the technique that the kernel uses to let the process believe it
has got a contiguous address space, agreed?
That's one of the things it does.
Quote: So the question is then, when you say "shared vm", do you mean
1) shared memory (as in the IPC technique called System V Shared Memory)
No. In that case, each process has its own view of the memory. The vm
is not shared, the pages are shared.
Quote: 2) the processes complete address space
I will get back to answering the other part of the post after you have
given me an answer.
The term 'vm' means the process' view of memory -- memory as seen by
the process. Each process has its own vm. Threads have a single shared
vm.
DS |
|
|
| Back to top |
|
|
|
| David Schwartz... |
Posted: Tue Oct 06, 2009 11:06 pm |
|
|
|
Guest
|
On Oct 7, 1:36 am, Jan Helgesen <s... at (no spam) nospam.com> wrote:
Quote: Ok, so here is the confusion. Do you have any proof to back up that
claim. I have been doing computers for 20 years and all my books on
operating systems and programming defines vm as a kernel memory
management technique to create the illusion of contiguous memory to a
process. Which is not another word for the process address space, shared
or non-shared.
Yes, it is. The "illusion of contiguous memory" *IS* the process'
address space.
Quote: The processes address space is the memory available to the program, i.e.
process or threads. By definition that memory is shared among all
threads, otherwise it would not be a thread, it would be different
processes. Virtual memory on the other hand is as I have already stated,
a kernel memory management technique.
Huh? It is precisely virtual memory that makes it possible for threads
to share vm and process' not to. It is the kernel's memory management
technique that makes it possible for each process to have its own view
of memory -- its 'vm'.
Quote: These are the agreed upon definitions in the computer science world. but
you seems to have defined it differently.
Google "shared vm" and take a look at the first dozen hits or so.
DS |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Wed Oct 07, 2009 1:24 am |
|
|
|
Guest
|
David Schwartz wrote:
Quote: On Oct 6, 5:32 pm, Jan Helgesen <s... at (no spam) nospam.com> wrote:
3) Shared vm has a high performance cost.
what cost is that, and the high performance cost is compared to what?
Anything that manipulates the vm has to be synchronized. Consider, for
example, opening a file. Consider mapping a new region of memory.
Also, since they have to be in the same SMP region, they typically
have to all be talking to a single kernel. So many kernel operations
will wind up imposing some kind of synchronization cost.
The whole point of message passing for scalability is that you don't
have to use a shared vm. You can distribute your large operation over
multiple memory domains. Shared vm among thousands of cores make
absolutely no sense at all for the foreseeable future.
I am having problems understanding what you are saying here.
First off, when you say vm, you mean virtual memory, right?
Wikipedia defines virtual memory as:
"Virtual memory is a computer system technique which gives an
application program the impression that it has contiguous working memory
(an address space), while in fact it may be physically fragmented and
may even overflow on to disk storage."
So virtual memory is not a process or programs complete address space,
it is the technique that the kernel uses to let the process believe it
has got a contiguous address space, agreed?
So the question is then, when you say "shared vm", do you mean
1) shared memory (as in the IPC technique called System V Shared Memory)
2) the processes complete address space
I will get back to answering the other part of the post after you have
given me an answer.
regards
Jan |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Wed Oct 07, 2009 2:36 am |
|
|
|
Guest
|
David Schwartz wrote:
An answer that was uncalled for.
You have the wrong definitions for the terms you are using and hence are
using the terms incorrectly, as explanation below.
Quote: First off, when you say vm, you mean virtual memory, right?
Wikipedia defines virtual memory as:
Yes.
ok so far so good.
Quote:
"Virtual memory is a computer system technique which gives an
application program the impression that it has contiguous working memory
(an address space), while in fact it may be physically fragmented and
may even overflow on to disk storage."
So virtual memory is not a process or programs complete address space,
Umm, yes it is. The process or program's complete address space is its
vm. (Perhaps you're not familiar with this usage, but it's very
common.)
Ok, so here is the confusion. Do you have any proof to back up that
claim. I have been doing computers for 20 years and all my books on
operating systems and programming defines vm as a kernel memory
management technique to create the illusion of contiguous memory to a
process. Which is not another word for the process address space, shared
or non-shared.
The processes address space is the memory available to the program, i.e.
process or threads. By definition that memory is shared among all
threads, otherwise it would not be a thread, it would be different
processes. Virtual memory on the other hand is as I have already stated,
a kernel memory management technique.
These are the agreed upon definitions in the computer science world. but
you seems to have defined it differently.
regards
Jan |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Wed Oct 07, 2009 11:40 pm |
|
|
|
Guest
|
David Schwartz wrote:
Quote: So he seems to have the same misunderstanding of what virtual memory
actually is, as you do.
Perhaps everyone but you has this misunderstanding.
Until you can come up with any references from credible source that
supports your statement, I will stick to Tannenbaum and other have
defined it to be. And according to their definitions, your understanding
is wrong..
regards
Jan |
|
|
| Back to top |
|
|
|
| Jan Helgesen... |
Posted: Wed Oct 07, 2009 11:44 pm |
|
|
|
Guest
|
David Schwartz wrote:
Quote:
It makes perfect sense to use message passing within a single vm. What
makes no sense, however, is to design a system that uses message
passing for scalability using threads which share their vm.
As I have said it allows threads to access data without needing to share
it and lock it. If you don't have any more compelling arguments than
repeating what you are saying. There is no point in continuing the
discussion.
regards
Jan |
|
|
| Back to top |
|
|
|
| David Schwartz... |
Posted: Thu Oct 08, 2009 12:10 pm |
|
|
|
Guest
|
On Oct 7, 10:44 pm, Jan Helgesen <s... at (no spam) nospam.com> wrote:
Quote: David Schwartz wrote:
It makes perfect sense to use message passing within a single vm. What
makes no sense, however, is to design a system that uses message
passing for scalability using threads which share their vm.
As I have said it allows threads to access data without needing to share
it and lock it.
This is a serious misunderstanding on your part. They *do* need to
share it and lock it. The reason for the message passing is precisely
to allow the threads to easily share and lock data.
Quote: If you don't have any more compelling arguments than
repeating what you are saying. There is no point in continuing the
discussion.
It would help if you showed some indication that you understood what I
was saying.
DS |
|
|
| Back to top |
|
|
|
| David Schwartz... |
Posted: Thu Oct 08, 2009 11:03 pm |
|
|
|
Guest
|
On Oct 9, 12:02 am, Jan Helgesen <s... at (no spam) nospam.com> wrote:
Quote: David Schwartz wrote:
Huh? Now you've lost me. We were talking about architectures that use
message passing for scalability.
That is true. But your claim was that one always needs locking. which I
explained one does not always need.
You are still confused. "Lock-free" is a misnomer. There are still
locks, they are just not locks that block. For example, the primitive
lock-free algorithms are made out of contain atomic 'compare and
replace' operations or atomic 'swap' operations. These operations
internally contain locks, for example, they lock cache lines.
But I appreciate your attempt at catching me in a minor "gotcha" so as
to waste time and avoid the major issues.
Quote: There would always have to be. The scalability advantage of message
passing does not come from the message passing itself being somehow
incredibly efficient. In fact, in general message passing is less
efficient than other types of IPC.
I have have never said that the implementation of the message passing
makes it more efficient. I have said that the scalability advantage
comes from the fact that one can parallelise the program without the
need for locking.
But this is totally and utterly false. You need *precisely* the same
locking.
Quote: Hence it makes it more efficient in the fact that more
work can be done in parallel, possibly massively parallel. That an
advantage that is diminishing by the day, with sequential programming
practice of today.
This is a misunderstanding you keep repeating and repeating. It is
false. Message passing requires precisely the same locking as every
other form of IPC. The advantage of message passing has *nothing* to
do with the fact that message passing is somehow incredibly efficient
or doesn't require locks or concurrency that other forms of IPC
require.
The advantage of message passing, from a scalability and performance
standpoint, is one thing and one thing only -- you do not have to
share address space.
Quote: I still see no evidence you have understood anything I'm saying.
dont be rude, you dont seem to understand what I am saying either so
there is no need for that kind attitude, it does not help the discussion.
I understand what you're saying, it's simply completely incorrect. And
when I try to point out your misunderstandings, you try to play
"gotcha" on minor and irrelevant issues while you insist on missing
the forest.
The forest is this -- message passing is inefficient. The reason
people use it for performance and scalability is because if you limit
yourself to message passing only, you can get the concurrency of
threads without the inherent performance penalties of threads.
Quote: You seem to completely misunderstand the entire point of going to the
trouble of using message passing. You erroneously think this somehow
makes design and implementation easier, when in fact it makes design
and implementation harder. (Why? Because you can always make any IPC
look like message passing. But when you commit to using message
passing *FOR* *SCALABILITY* you lose the ability to use other forms of
IPC such as shared memory.)
The possibility of scalability does not come in one form only. Shared
memory is one answer, but it has its limitations. Message passing is
another, which addressed many of the limitations of shared memory, with
respect to concurrency. Other than that see my answers above.
Right, the advantage of message passing is that it doesn't have the
limitations of shared memory. So it would be an absolutely mind-
bogglingly stupid thing to do to build a message passing system on top
of shared memory threads. You would suffer all the disadvantages of
threads and get absolutely *none* of the benefits. *Nobody* with any
brains would do that unless performance and scalability were the
lowest priorities. If you're talking about scaling to thousands of
cores and using message passing, you would *NEVER* do it with threads.
DS |
|
|
| Back to top |
|
|
|
|
|
All times are GMT - 5 Hours
The time now is Tue Nov 24, 2009 8:04 pm
|
|