| |
 |
|
|
Science Forum Index » Math - Symbolic Forum » benchmarking CAS
Page 1 of 3 Goto page 1, 2, 3 Next
|
| Author |
Message |
| rjf |
 Posted: Sat Mar 29, 2008 8:55 am |
|
|
|
Guest
|
In discussions with fans of Maxima, a lisp-based computer algebra
system, concerning how to make it run more efficiently, a number of
issues have been raised that can speed up individual operations. I
was thinking about how to make the testing more meaningful.
In the long term, there are other issues at play that will affect
speed, other than what is revealed by timing a single command.
For example, "professional" lisp systems must be prepared to run
indefinitely, and therefore do things like compacting storage. This
generally means that addresses of cells are not permanent, and
addresses cannot be used as proxies for (say) hashcodes or total
orderings.
On the other hand, Maple, Mathematica, ... may not have such issues:
I doubt that they are used for production 24/7 computing, say for
airline reservation systems; most users run them for a few CPU minutes
at worst. Since they do not have to run indefinitely, and/or have a
much simpler memory model for most of their data, they can use
addresses as proxies for various things. (In particular, Mathematica
uses page locations as a proxy for the age of an object -- X has to be
recomputed if X is older than any of the objects on which it depends).
But if they were run for months, what would happen? (And do we care?)
Note that it is also possible for a "less-professional" lisp system to
have a simple garbage collector that does not move data: this works
for a while (and maybe indefinitely) -- so long as memory
fragmentation does not emerge as a problem. This makes the author of
every brand-new naive lisp system proud. for a little while.
So a good timing result where the whole computation takes a few
minutes is not such a great measure, at least if it is supposed to be
a predictor of longer computations. Running some 10-second test suite
predicts how long it takes to run that 10-second test suite.
One could, I suppose, also have some measure of "memory dispersion"
before cutting off such a benchmark; additionally, one could create a
benchmark that (a) took a long time and (b) created/deleted/created/
huge data objects.
Perhaps running every benchmark repeatedly, say 2000 times, might be
interesting. The first 1000 times one could delete any leftover
results, and the second 1000 times, the results should be retained,
clogging memory. Programs which retain data in hash tables for fast
recomputation, etc, must be considered within the context of truth in
testing: what would a memory-intense sequence of computations do if
they were not just repetitions of already remembered results.
Comments?
RJF |
|
|
| Back to top |
|
| Vladimir Bondarenko |
| Posted: Sat Mar 29, 2008 10:50 am |
|
|
|
Guest
|
The mean time to defect_type_j for a set of j might be a telling
paramenter at the choice of the proper CAS.
In particular, the mean time to crash.
For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.
Best wishes,
Vladimir Bondarenko
VM and GEMM architect
Co-founder, CEO, Mathematical Director
http://www.cybertester.com/ Cyber Tester, LLC
http://maple.bug-list.org/ Maple Bugs Encyclopaedia
http://www.CAS-testing.org/ CAS Testing
-----------------------------------------------------------------
"We must understand that technologies
like these are the way of the future."
-----------------------------------------------------------------
On Mar 29, 11:55 am, rjf <fate...@gmail.com> wrote:
Quote: In discussions with fans of Maxima, a lisp-based computer algebra
system, concerning how to make it run more efficiently, a number of
issues have been raised that can speed up individual operations. I
was thinking about how to make the testing more meaningful.
In the long term, there are other issues at play that will affect
speed, other than what is revealed by timing a single command.
For example, "professional" lisp systems must be prepared to run
indefinitely, and therefore do things like compacting storage. This
generally means that addresses of cells are not permanent, and
addresses cannot be used as proxies for (say) hashcodes or total
orderings.
On the other hand, Maple, Mathematica, ... may not have such issues:
I doubt that they are used for production 24/7 computing, say for
airline reservation systems; most users run them for a few CPU minutes
at worst. Since they do not have to run indefinitely, and/or have a
much simpler memory model for most of their data, they can use
addresses as proxies for various things. (In particular, Mathematica
uses page locations as a proxy for the age of an object -- X has to be
recomputed if X is older than any of the objects on which it depends).
But if they were run for months, what would happen? (And do we care?)
Note that it is also possible for a "less-professional" lisp system to
have a simple garbage collector that does not move data: this works
for a while (and maybe indefinitely) -- so long as memory
fragmentation does not emerge as a problem. This makes the author of
every brand-new naive lisp system proud. for a little while.
So a good timing result where the whole computation takes a few
minutes is not such a great measure, at least if it is supposed to be
a predictor of longer computations. Running some 10-second test suite
predicts how long it takes to run that 10-second test suite.
One could, I suppose, also have some measure of "memory dispersion"
before cutting off such a benchmark; additionally, one could create a
benchmark that (a) took a long time and (b) created/deleted/created/
huge data objects.
Perhaps running every benchmark repeatedly, say 2000 times, might be
interesting. The first 1000 times one could delete any leftover
results, and the second 1000 times, the results should be retained,
clogging memory. Programs which retain data in hash tables for fast
recomputation, etc, must be considered within the context of truth in
testing: what would a memory-intense sequence of computations do if
they were not just repetitions of already remembered results.
Comments?
RJF |
|
|
| Back to top |
|
| SzH |
| Posted: Sat Mar 29, 2008 11:48 am |
|
|
|
Guest
|
On Mar 29, 9:50 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:
Quote: For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.
You mean that you are able to run the Mathematica 6 kernel
continuously, performing random symbolic computations (like
integration), for 24 hours, without any problems? That is better than
what I expected. How much memory does your machine have? |
|
|
| Back to top |
|
| Roman Pearce |
| Posted: Sat Mar 29, 2008 3:28 pm |
|
|
|
Guest
|
On Mar 29, 12:55 pm, rjf <fate...@gmail.com> wrote:
Quote: On the other hand, Maple, Mathematica, ... may not have such issues:
....
But if they were run for months, what would happen? (And do we care?)
In Maple memory is allocated but not returned to the operating system.
The time for garbage collection grows, but eventually memory use and
performance stabilizes. Fragmentation is an issue when large
objects
of various sizes are repeatedly allocated.
For example, I recently tried the following computation:
f := expand((1+x+y+z+t)^30):
g := f+1:
p := expand(f*g):
divide(p,f,'q');
bytes used=991591348816, alloc=1649501176, time=29025.53
Because the polynomials are large, Maple uses interpreted routines
that run with garbage collection. The time was not very good, but
bytesalloc is basically the size of the product, so it looks like
the garbage collector will keep the system up. I have seen and
done other computations that took days and weeks. If something
is going to take more than a month, it would probably make sense
to write it in C. |
|
|
| Back to top |
|
| Vladimir Bondarenko |
| Posted: Sat Mar 29, 2008 8:38 pm |
|
|
|
Guest
|
SzH> You mean that you are able to run the Mathematica 6
SzH> kernel continuously, performing random symbolic computations
SzH> (like integration),
Well, the inputs the VM machine feeds into the MathKernel are not
totally random :)
Some of them are brute-force ones, but a tangible fraction of them
comes from some conditions the VM machine calculates trying to crash
the MathKernel.
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system
of the above-listed.
We observed some specific runs when Derive 6.1 had been operational
for about 3 weeks (!) of continuous calculations the VM machine forced
it to perform.)
SzH> for 24 hours, without any problems?
Kind of this, in the average.
SzH> That is better than what I expected.
:)
It was worse in Mathematica 5.2. But we feel that in Mathematica 6
some real improvements are introduced.
SzH> How much memory does your machine have?
RAM is 4 Gb, and total free HDD size of about 500 Gb.
On Mar 29, 2:48 pm, SzH <szhor...@gmail.com> wrote:
Quote: On Mar 29, 9:50 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:
For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.
You mean that you are able to run the Mathematica 6 kernel
continuously, performing random symbolic computations (like
integration), for 24 hours, without any problems? That is better than
what I expected. How much memory does your machine have? |
|
|
| Back to top |
|
| Vladimir Bondarenko |
| Posted: Sat Mar 29, 2008 11:49 pm |
|
|
|
Guest
|
Hello Nasser,
Thanks for your idea.
Technically, we could do this. Anyways, now we have a small amount of
resource we can dedicate to a concrete system; but we here hope that
the things could look better reasonably soon, hopefully, this year.
As you understand, if we here dedicate something to SAGE, we subtract
this from the above list I quoted.
Could you please tell why do you feel that testing SAGE could be
really useful? Useful for whom?
By the way, maybe you know, approximately, how many persons use SAGE?
Occasionally? On a constant basis?
Cheers,
Vladimir
On Mar 29, 11:57 pm, "Nasser Abbasi" <n...@12000.org> wrote:
Quote: Hello Vladimir;
"Vladimir Bondarenko" <v...@cybertester.com> wrote in message
news:373f7f01-6a15-49c0-83c3-7ecef59e19a5@p25g2000hsf.googlegroups.com...
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system of the
above-listed."
|
|
|
| Back to top |
|
| SzH |
| Posted: Sun Mar 30, 2008 12:56 am |
|
|
|
Guest
|
On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
AFAIK Sage is primarily a browser-based notebook interface to other
packages such as Maxima and SciPy. So it is those other packages that
would benefit from testing and not SAGE itself. |
|
|
| Back to top |
|
| Nasser Abbasi |
| Posted: Sun Mar 30, 2008 1:57 am |
|
|
|
Guest
|
Hello Vladimir;
"Vladimir Bondarenko" <vb@cybertester.com> wrote in message
news:373f7f01-6a15-49c0-83c3-7ecef59e19a5@p25g2000hsf.googlegroups.com...
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system of the
above-listed."
It would be interesting if your VM machine can add SAGE to the systems it
tests. SAGE is described here:
http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experimentation
I played with SAGE a little, but I find its interface on windows, having to
use a browser, a bit awkward to say the least. I have not tried it text
based interface.
Nasser |
|
|
| Back to top |
|
| SzH |
| Posted: Sun Mar 30, 2008 3:00 am |
|
|
|
Guest
|
On Mar 30, 2:14 pm, Jaap Spies <j.sp...@hccnet.nl> wrote:
Quote: SzH wrote:
On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
It would be interesting if your VM machine can add SAGE to the systems it
tests. SAGE is described here:
http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
I played with SAGE a little, but I find its interface on windows, having to
use a browser, a bit awkward to say the least. I have not tried it text
based interface.
AFAIK Sage is primarily a browser-based notebook interface to other
packages such as Maxima and SciPy. So it is those other packages that
would benefit from testing and not SAGE itself.
This is certainly not true! Sage has its own code base and is much, much
more then only a notebook interface.
I am not a SAGE user, but the sagemath.org website created the
impression that SAGE is just some kind of front end to other
packages. I wanted to learn more about this system, but there is no
"About SAGE" link on the front page and the FAQ is not very
informative.
So does SAGE have its own symbolic computation engine? Can it e.g.
compute integrals (one thing that Vladimir has been testing)?
Quote: Seehttp://www.sagemath.org/
Browse through the code:http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/s...
I don't think that the right way to introduce SAGE to a newbie is to
ask him to browse the system's source code ...
SAGE might be a useful program, but the current website is somehow not
convincing (and not informative) enough to persuade me to attempt a
500 MB download of something that will only run on a linux system
running inside an emulator ... Just compare it with e.g. the orders
of magnitude better Yacas website to see what I mean. |
|
|
| Back to top |
|
| mabshoff |
| Posted: Sun Mar 30, 2008 3:43 am |
|
|
|
Guest
|
On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
Quote: Hello Vladimir;
"Vladimir Bondarenko" <v...@cybertester.com> wrote in message
news:373f7f01-6a15-49c0-83c3-7ecef59e19a5@p25g2000hsf.googlegroups.com...
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system of the
above-listed."
Vladimir,
could you speculate what crashes the various systems, i.e. memory
leaks, random bugs and so on?
Hi Nasser,
looking at what the VM tests Sage farms out most of those tasks
[integration, differentiation, ODE] to Maxima, which is already
tested. We are currently in the process of implementing symbolic
caculus in Sage itself and will hopefully in the next year move
differentiation, limits and integration and so on away from Maxima
into the Sage core. This is motivated by the need for performance and
the fact that we want to dump any lisp based code from the core of
Sage in the long term.
Cheers,
Michael |
|
|
| Back to top |
|
| mabshoff |
| Posted: Sun Mar 30, 2008 3:49 am |
|
|
|
Guest
|
On Mar 30, 3:00 pm, SzH <szhor...@gmail.com> wrote:
Quote: On Mar 30, 2:14 pm, Jaap Spies <j.sp...@hccnet.nl> wrote:
SzH wrote:
On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
It would be interesting if your VM machine can add SAGE to the systems it
tests. SAGE is described here:
http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi....
I played with SAGE a little, but I find its interface on windows, having to
use a browser, a bit awkward to say the least. I have not tried it text
based interface.
AFAIK Sage is primarily a browser-based notebook interface to other
packages such as Maxima and SciPy. So it is those other packages that
would benefit from testing and not SAGE itself.
This is certainly not true! Sage has its own code base and is much, much
more then only a notebook interface.
I am not a SAGE user, but the sagemath.org website created the
impression that SAGE is just some kind of front end to other
packages. I wanted to learn more about this system, but there is no
"About SAGE" link on the front page and the FAQ is not very
informative.
So does SAGE have its own symbolic computation engine? Can it e.g.
compute integrals (one thing that Vladimir has been testing)?
No, it currently doesn't have its own engine, but some of us are in
the process of writing one.
Quote: Seehttp://www.sagemath.org/
Browse through the code:http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/s...
I don't think that the right way to introduce SAGE to a newbie is to
ask him to browse the system's source code ...
Yeah, not very useful for some newbie. But I guess that link posted by
Jaap was more directed at "Sage is a web frontend".
Quote: SAGE might be a useful program, but the current website is somehow not
convincing (and not informative) enough to persuade me to attempt a
The website could certainly be improved. For an incomplete list of
what Sage can do check out
http://www.sagemath.org/doc/html/ref/index.html
Quote: 500 MB download of something that will only run on a linux system
running inside an emulator ...
It runs on various flavors of Linux and also on OSX. Microsoft
Reseatch is paying for a native Windows port and Solaris is getting
close to running. There are many other worthwhile platforms left after
that in my book.
Quote: Just compare it with e.g. the orders
of magnitude better Yacas website to see what I mean.
Cheers,
Michael |
|
|
| Back to top |
|
| Vladimir Bondarenko |
| Posted: Sun Mar 30, 2008 4:19 am |
|
|
|
Guest
|
Michael Abshoff writes
MA> could you speculate what crashes the various systems,
MA> i.e. memory leaks, random bugs and so on?
This would require some extra work, some extra resource we lack now.
I am very reluctant to waste words.
MA> random bugs
What do you mean by "random bugs"? Couldn't you please formulate?
MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
yeah, to an extent our resource affords.
MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
Why sure, I meant precisely this!
MA> We are currently in the process of implementing symbolic
MA> calculus in Sage itself and will hopefully in the next year
MA> move differentiation, limits and integration and so on away
MA> from Maxima into the Sage core.
Oh, this is a different kettle of fish altogether!
This, along with the drive you seem to push SAGE, will be a good
incentive for Cyber Tester to test SAGE with the VM machine.
Cheers,
Vladimir Bondarenko
VM and GEMM architect
Co-founder, CEO, Mathematical Director
http://www.cybertester.com/ Cyber Tester, LLC
http://maple.bug-list.org/ Maple Bugs Encyclopaedia
http://www.CAS-testing.org/ CAS Testing
-----------------------------------------------------------------
"We must understand that technologies
like these are the way of the future."
-----------------------------------------------------------------
On Mar 30, 6:43 am, mabshoff <mabsh...@googlemail.com> wrote:
Quote: On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
Hello Vladimir;
"Vladimir Bondarenko" <v...@cybertester.com> wrote in message
news:373f7f01-6a15-49c0-83c3-7ecef59e19a5@p25g2000hsf.googlegroups.com...
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system of the
above-listed."
Vladimir,
could you speculate what crashes the various systems, i.e. memory
leaks, random bugs and so on?
It would be interesting if your VM machine can add SAGE to the systems it
tests. SAGE is described here:
http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
I played with SAGE a little, but I find its interface on windows, having to
use a browser, a bit awkward to say the least. I have not tried it text
based interface.
Nasser
Hi Nasser,
looking at what the VM tests Sage farms out most of those tasks
[integration, differentiation, ODE] to Maxima, which is already
tested. We are currently in the process of implementing symbolic
caculus in Sage itself and will hopefully in the next year move
differentiation, limits and integration and so on away from Maxima
into the Sage core. This is motivated by the need for performance and
the fact that we want to dump any lisp based code from the core of
Sage in the long term.
Cheers,
Michael |
|
|
| Back to top |
|
| mabshoff |
| Posted: Sun Mar 30, 2008 4:46 am |
|
|
|
Guest
|
On Mar 30, 4:19 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:
Quote: Michael Abshoff writes
Hi Vladimir
Quote: MA> could you speculate what crashes the various systems,
MA> i.e. memory leaks, random bugs and so on?
This would require some extra work, some extra resource we lack now.
I am very reluctant to waste words.
Sure. I considered the possibility that you did collect the kind of
failures you saw.
Quote: MA> random bugs
What do you mean by "random bugs"? Couldn't you please formulate?
Does the same sequence of input always leads reproducibly to the same
internal error/segfault at the same point during the computation or
does the same sequence of input fail at different, unpredictable
points or not at all?
Detecting that the process just ended up eating all the available RAM
while doing the same computation over and over again [i.e. indicating
memory leaks] should also be straight forward compared to either
straight segfaults or "internal error ... exiting" type situations
where the CAS does detect some issue and exits somewhat gracefully.
Quote: MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
yeah, to an extent our resource affords.
MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
Why sure, I meant precisely this!
MA> We are currently in the process of implementing symbolic
MA> calculus in Sage itself and will hopefully in the next year
MA> move differentiation, limits and integration and so on away
MA> from Maxima into the Sage core.
Oh, this is a different kettle of fish altogether!
This, along with the drive you seem to push SAGE, will be a good
incentive for Cyber Tester to test SAGE with the VM machine.
Yeah, automated testing would be excellent, once the code is getting
merged and can do more than trivial things. The Sage developers as a
whole are fanatical about testing in general and I for example will
drop anything to hunt down a reproducible memory leak. We have a rule
that fixes for bugs are only merged if accompanied by a test that
verifies that the bug in question was fixed. Unlike some of the other
CASs you test we welcome bugs reports and our bug tracker is open to
the public since we have nothing to hide :)
Sage's main strength are certainly not in the calculus are yet since
the original developers cared a lot about number theory and calculus
was initially tacked on to be able to use Sage in Calculus classes. We
are now slowly getting more developers from the areas that do care
about efficient symbolic calculus [physics for example] and
consequently things will improve in that area.
Quote: Cheers,
Vladimir Bondarenko
Cheers,
Michael
|
|
|
| Back to top |
|
| mabshoff |
| Posted: Sun Mar 30, 2008 5:13 am |
|
|
|
Guest
|
On Mar 30, 4:47 pm, caveman <dev.n...@dev.null.com> wrote:
Quote: mabshoff wrote:
It runs on various flavors of Linux and also on OSX. Microsoft
Reseatch is paying for a native Windows port and Solaris is getting
close to running. There are many other worthwhile platforms left after
that in my book.
Hi,
Quote: What is your problem with Solaris?
It compiles in 32 bit mode with small modifications, but the test
suite doesn't pass yet. One big issue is finding a self hosted lisp
that compiles with either gcc or Sun Forte's compiler suite [sbcl and
cmucl are not self hosted, but sbcl does work for my purposes],
another one is in certain parts of libSingular.
Quote: I was at one point working with
William on trying to do some work with Solaris and have spent quite a
bit of time attempting compilation. The problem seems to be the large
number of external packages you use, all of which are written by
different people using very different approaches.
Well, that is not that much of a problem since I have been fixing
[Solaris specific] bugs in all those packages and been pushing them
back up stream. There are certainly small problems left here and
there, but we can handle those. It is mostly all about smoothly
merging fixes back into our packages as well as keeping those build
issues from new patches under control. I don't think it will surprise
anybody from the Open Source world that most code these days usually
is developed and works on Linux while it can be hard to get it working
on Solaris. Ten years ago it was very much a reversed situation.
Quote: Not sure what your latest status is, but if you need hardware, rather
than man-hours, I can probably help with access via SSH to one or more
multi-processor Suns. I know at one time you were limited by hardware
availability. If that is still so, I can help.
Thanks. William did hire me full time to work on the Solaris port for
2008 and I am certain we will get it done in 32/64 bit on Solaris 9/10/
Indiana on x86/x86-64/Sparc. This is a huge test matrix and the
differences between Solaris 9 and 10 are rather large. I am switching
the 64 bit build of Sage to the Sun Forte compiler since it causes way
fewer problems in 64 bit mode compared to gcc. So if you have some
more test machines with say Indiana on Sparc I would be more than
happy to take you up on that offer. Even build and test feedback would
be nice. Since this is getting OT we should take this offlist from now
on or go over to gg:sage-devel.
Cheers,
Michael |
|
|
| Back to top |
|
| rjf |
| Posted: Sun Mar 30, 2008 5:48 am |
|
|
|
Guest
|
On Mar 30, 5:46 am, "Nasser Abbasi" <n...@12000.org> wrote:
.......
Quote:
Well, since Sage is, I think, is the first open source GPL'ed CAS software?
and so it would be useful to see how its quality stack up to commercial
software.
How is it first?
Maxima? Axiom? Jacal? |
|
|
| Back to top |
|
| |
Page 1 of 3 Goto page 1, 2, 3 Next
All times are GMT - 5 Hours
The time now is Thu Jul 24, 2008 4:56 pm
|
|
