Main Page | Report this Page
 
   
Science Forum Index  »  Math - Symbolic Forum  »  benchmarking CAS
Page 3 of 3    Goto page Previous  1, 2, 3
Author Message
Guest
Posted: Sun Mar 30, 2008 1:10 pm
Quote:
looking at what the VM tests Sage farms out most of those tasks
[integration, differentiation, ODE] to Maxima, which is already
tested. We are currently in the process of implementing symbolic
caculus in Sage itself and will hopefully in the next year move
differentiation, limits and integration and so on away from Maxima
into the Sage core. This is motivated by the need for performance and
the fact that we want to dump any lisp based code from the core of
Sage in the long term.


If you want to move away from lisp, what about using the C++ Giac
library instead of rewriting and testing again all the calculus code.
Giac already implements the mrv algorithm for limits, the Risch (pure
transcendental) algorithm for limits, as well as few methods for
definite integration.
Giac was 3rd at the Trophees du Libre that Sage won last year.
http://www-fourier.ujf-grenoble.fr/~parisse/giac.html
SzH
Posted: Sun Mar 30, 2008 9:25 pm
Guest
On Mar 30, 10:05 pm, SzH <szhor...@gmail.com> wrote:
Quote:
Two things to watch out for if you start experimenting yourself (I
have /not/ tested these thoroughly, so they might not be repeatable):

1. Each input should be put in a separate input cell. If the
Expand[...] and the x=. are in the same input cell, Mathematica does
not release memory and the system starts swapping ...

2. It appears that if Share[] is executed after Expand[...], but /
before/ x=., then the timing value does not increase.

These two are not correct. Something else must have caused them.
semiopen
Posted: Mon Mar 31, 2008 6:50 am
Guest
On Mar 30, 2:38 am, Vladimir Bondarenko <v...@cybertester.com> wrote:
Quote:
SzH>  You mean that you are able to run the Mathematica 6
SzH>  kernel continuously, performing random symbolic computations
SzH>  (like integration),

Well, the inputs the VM machine feeds into the MathKernel are not
totally random :)

Some of them are brute-force ones, but a tangible fraction of them
comes from some conditions the VM machine calculates trying to crash
the MathKernel.

(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.

As of today, Derive 6.1 is the most stable computer algebra system
of the above-listed.

We observed some specific runs when Derive 6.1 had been operational
for about 3 weeks (!) of continuous calculations the VM machine forced
it to perform.)

This is another reason that it is so sad that Texas Instruments
stopped supporting Derive. Have you looked at TI's replacement for
Derive? I am not optimistic, but since they had Derive's source code
maybe they were able to preserve some of Derive's nice features.



Quote:
SzH>  for 24 hours, without any problems?

Kind of this, in the average.

SzH>  That is better than what I expected.

:)

It was worse in Mathematica 5.2. But we feel that in Mathematica 6
some real improvements are introduced.

SzH>  How much memory does your machine have?

RAM is 4 Gb, and total free HDD size of about 500 Gb.

On Mar 29, 2:48 pm, SzH <szhor...@gmail.com> wrote:



On Mar 29, 9:50 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:

For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.

You mean that you are able to run the Mathematica 6 kernel
continuously, performing random symbolic computations (like
integration), for 24 hours, without any problems?  That is better than
what I expected.  How much memory does your machine have?- Hide quoted text -

- Show quoted text -
Vladimir Bondarenko
Posted: Mon Mar 31, 2008 7:42 am
Guest
SO> This is another reason that it is so sad that Texas
SO> Instruments stopped supporting Derive.

I've been very close (or at time inside) the Derive's witchcraft
kitchen and know more than I can say.

But I can state safely that Derive has a big potential for growth.

If one fine day Texas Instruments would sell it, I'd probably could
have a try to persuade Albert Rich to continue pushing Derive. Well
there is not exactly I would like to say, but you can guess more.

And maybe one fine day I can tell more about all this thrilling
story.

SO> Have you looked at TI's replacement for Derive?

In the autumn of 2006 I enjoyed a privilege to be selected one
of its beta testers

................................................................

http://groups.google.com/group/sci.math.symbolic/msg/9672efcab9159d9f

http://www.fedex.com/Tracking?language=english&cntry_code=&tracknumbers=790636569170

http://www.cas-testing.org/index.php?list=3

................................................................

Instead of testing it manually, we forced the VM machine to test
it. :)

With some unique results, as usually.

It is now for the first time that I mentioned this fact publicly.

SO> I am not optimistic, but since they had Derive's source
SO> code maybe they were able to preserve some of Derive's
SO> nice features.

Unfortunately, it's better that you are not optimistic... there
are several reasons for this...

What's good, you and me still can use old good Derive 6.1.

On Mar 31, 9:50 am, semiopen <former_schiz...@hotmail.com> wrote:
Quote:
On Mar 30, 2:38 am, Vladimir Bondarenko <v...@cybertester.com> wrote:





SzH>  You mean that you are able to run the Mathematica 6
SzH>  kernel continuously, performing random symbolic computations
SzH>  (like integration),

Well, the inputs the VM machine feeds into the MathKernel are not
totally random :)

Some of them are brute-force ones, but a tangible fraction of them
comes from some conditions the VM machine calculates trying to crash
the MathKernel.

(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.

As of today, Derive 6.1 is the most stable computer algebra system
of the above-listed.

We observed some specific runs when Derive 6.1 had been operational
for about 3 weeks (!) of continuous calculations the VM machine forced
it to perform.)

This is another reason that it is so sad that Texas Instruments
stopped supporting Derive. Have you looked at TI's replacement for
Derive? I am not optimistic, but since they had Derive's source code
maybe they were able to preserve some of Derive's nice features.



SzH>  for 24 hours, without any problems?

Kind of this, in the average.

SzH>  That is better than what I expected.

:)

It was worse in Mathematica 5.2. But we feel that in Mathematica 6
some real improvements are introduced.

SzH>  How much memory does your machine have?

RAM is 4 Gb, and total free HDD size of about 500 Gb.

On Mar 29, 2:48 pm, SzH <szhor...@gmail.com> wrote:

On Mar 29, 9:50 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:

For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.

You mean that you are able to run the Mathematica 6 kernel
continuously, performing random symbolic computations (like
integration), for 24 hours, without any problems?  That is better than
what I expected.  How much memory does your machine have?- Hide quoted text -

- Show quoted text -- Hide quoted text -

- Show quoted text -- Hide quoted text -

- Show quoted text -
Roman Pearce
Posted: Mon Mar 31, 2008 10:14 am
Guest
rjf wrote:
Quote:
Thomas Richard actually provided info regarding Maple 11 behavior in
returning memory.

We need to clear this up. I assume that if you're running programs
for days or weeks then you will have objects in memory that you need
to keep, so you can't use the restart command. This would not be
true for something like a web-based integrator that simply passes
independent problems to the system, but it would be true for a
complex program that maintains an internal state, like an airline
reservation program. Without restarting the kernel, Maple will
not return memory to the OS and the time for garbage collection
will grow (up to a limit).

It looks like Mathematica does the same thing:
SzH wrote:
Quote:
In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[2]= {61.86, Null}
...
In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[8]= {95.484, Null}

I would like to say that this is not an entirely unreasonable
design, and that more advanced garbage collection techniques
could possibly reduce the overhead. The difference in time
is largely because the system has more memory allocated, which
means that the garbage collector has to scan more memory each
time a GC occurs. Scanning memory is not free. Computers have
finite memory bandwidth, on the order of a few GB/s. If you are
repeatedly scanning even 200MB, you will notice it. And you will
be hit twice because garbage collection flushes the L2/L3 cache.

Regarding your question:
Quote:
can Maple 11 get into a situation where its memory usage
is so scattered that it runs at "disk speed" even though
the actual data in use is not so voluminous.

I can think of three situations where this could happen:
1) You have large objects in memory that are not being used.
The garbage collector still has to look at them.
2) You temporarily used a lot of memory. Modern operating systems
swap unused pages to disk, so even with GB allocated (because
memory is not released), the slowdown will be temporary if the
internal memory structures coalesce free space (they do).
You don't scan memory marked as "free".
3) There is a bug in the garbage collector


Also, replying to mabshoff now, I would really like to know what
you meant here:

mabshoff wrote:
Quote:
And the vast majority of benchmarks do not take long term effects
or even memory consumption into account. The most famous example I can
come up with are Gröbner Basis computations comparing the Buchberger
algorithm with F4 or F5.
Jaap Spies
Posted: Mon Mar 31, 2008 11:24 am
Guest
rjf wrote:

Quote:

If we don't raise questions when people continue to compare CAS on
essentially irrelevant grounds, we are in the position of essentially
allowing others who read this newsgroup from accepting these comments
as valid.

On Dec 6, 2005 rjf wrote a comparison in sci.math.symbolic:

Quote:
Elephants are interesting and useful. Feathers are interesting
and useful.
Elephants with feathers are a curiosity. Is SAGE an elephant
with feathers?



See: http://sci.tech-archive.net/Archive/sci.math.symbolic/2005-12/msg00096.html

This is almost two and a half year ago. Time to reconsider!?

Next release of Sage will be sage-3.0, due in a week or two. Sage is
very much alive. Not as big a an elephant, but growing. And even without
feathers the future looks light and bright.

I invite you, Richard Fateman and others to look at Sage with an open mind.

Jaap
rjf
Posted: Mon Mar 31, 2008 5:17 pm
Guest
On Mar 31, 1:14 pm, Roman Pearce <rpear...@gmail.com> wrote:
Quote:
rjf wrote:
Thomas Richard actually provided info regarding Maple 11 behavior in
returning memory.

We need to clear this up. I assume that if you're running programs
for days or weeks then you will have objects in memory that you need
to keep, so you can't use the restart command. This would not be
true for something like a web-based integrator that simply passes
independent problems to the system, but it would be true for a
complex program that maintains an internal state, like an airline
reservation program. Without restarting the kernel, Maple will
not return memory to the OS and the time for garbage collection
will grow (up to a limit).

It looks like Mathematica does the same thing:

SzH wrote:
In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[2]= {61.86, Null}
...
In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[8]= {95.484, Null}

I would like to say that this is not an entirely unreasonable
design, and that more advanced garbage collection techniques
could possibly reduce the overhead. The difference in time
is largely because the system has more memory allocated, which
means that the garbage collector has to scan more memory each
time a GC occurs. Scanning memory is not free. Computers have
finite memory bandwidth, on the order of a few GB/s. If you are
repeatedly scanning even 200MB, you will notice it. And you will
be hit twice because garbage collection flushes the L2/L3 cache.

Regarding your question:

can Maple 11 get into a situation where its memory usage
is so scattered that it runs at "disk speed" even though
the actual data in use is not so voluminous.

I can think of three situations where this could happen:
1) You have large objects in memory that are not being used.
The garbage collector still has to look at them.
2) You temporarily used a lot of memory. Modern operating systems
swap unused pages to disk, so even with GB allocated (because
memory is not released), the slowdown will be temporary if the
internal memory structures coalesce free space (they do).
You don't scan memory marked as "free".
3) There is a bug in the garbage collector

Also, replying to mabshoff now, I would really like to know what
you meant here:

mabshoff wrote:
And the vast majority of benchmarks do not take long term effects
or even memory consumption into account. The most famous example I can
come up with are Gröbner Basis computations comparing the Buchberger
algorithm with F4 or F5.

Many of the Maple problems cited here are solved by generation
scavenging garbage collectors in modern lisps.
I don't quite understand the "coalesce" comment above. If only
adjacent memory locations are coalesced, that doesn't prevent
fragmentation.
And probably Mathematica also has these problems. At some point I
suspect that the shortcuts taken by these systems in their memory
allocation and associated hacks will come back and bite them (e.g.
when problems run at disk speed); whereas certain more sophisticated
(and perhaps initially slower?) systems will compact their memory and
continue to run at RAM speed.
Whether these problems occur often, sometimes, or never --- presumably
depends on the nature of the computations attempted.
RJF
SzH
Posted: Tue Apr 01, 2008 4:01 am
Guest
On Mar 31, 10:14 pm, Roman Pearce <rpear...@gmail.com> wrote:
Quote:
SzH wrote:
In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[2]= {61.86, Null}
...
In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[8]= {95.484, Null}

I would like to say that this is not an entirely unreasonable
design, and that more advanced garbage collection techniques
could possibly reduce the overhead. The difference in time
is largely because the system has more memory allocated, which
means that the garbage collector has to scan more memory each
time a GC occurs. Scanning memory is not free. Computers have
finite memory bandwidth, on the order of a few GB/s. If you are
repeatedly scanning even 200MB, you will notice it. And you will
be hit twice because garbage collection flushes the L2/L3 cache.

I didn't mean that this was a bug, I just found it a bit surprising.
I must say that (not being a programmer) I have no idea how garbage
collection algorithms work. But this page led me believe that
clearing the value of x (x =.) will restore the Mathematica kernel to
its original state (save for the existence of 'x' as a symbol with no
OwnValues):

http://reference.wolfram.com/mathematica/note/SomeNotesOnInternalImplementation.html

"Every piece of memory used by Mathematica maintains a count of how
many times it is referenced. Memory is automatically freed when this
count reaches zero."

If it works like this then why would a large amount of memory need to
be scanned explicitly?

Again, I am not a programmer, so sorry if this is a stupid question.
I read about a similar technique in Bruce Eckel's book: "Thinking in C+
+". From what I have learned there, I do not see why doing the same
calculation a second time would be slower than it was the first time.
Daniel Lichtblau
Posted: Tue Apr 01, 2008 5:11 am
Guest
On Apr 1, 9:01 am, SzH <szhor...@gmail.com> wrote:
Quote:
On Mar 31, 10:14 pm, Roman Pearce <rpear...@gmail.com> wrote:



SzH wrote:
In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[2]= {61.86, Null}
...
In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[8]= {95.484, Null}

I would like to say that this is not an entirely unreasonable
design, and that more advanced garbage collection techniques
could possibly reduce the overhead. The difference in time
is largely because the system has more memory allocated, which
means that the garbage collector has to scan more memory each
time a GC occurs. Scanning memory is not free. Computers have
finite memory bandwidth, on the order of a few GB/s. If you are
repeatedly scanning even 200MB, you will notice it. And you will
be hit twice because garbage collection flushes the L2/L3 cache.

I didn't mean that this was a bug, I just found it a bit surprising.
I must say that (not being a programmer) I have no idea how garbage
collection algorithms work. But this page led me believe that
clearing the value of x (x =.) will restore the Mathematica kernel to
its original state (save for the existence of 'x' as a symbol with no
OwnValues):

http://reference.wolfram.com/mathematica/note/SomeNotesOnInternalImpl...

"Every piece of memory used by Mathematica maintains a count of how
many times it is referenced. Memory is automatically freed when this
count reaches zero."

If it works like this then why would a large amount of memory need to
be scanned explicitly?

Again, I am not a programmer, so sorry if this is a stupid question.
I read about a similar technique in Bruce Eckel's book: "Thinking in C+
+". From what I have learned there, I do not see why doing the same
calculation a second time would be slower than it was the first time.

I also do not know the reason for this. I can only surmise that it is
an effect of memory fragmentation. My own anecdotal experience with
this type of timing rise on repeated computations is that it tends to
stabilize after a few iterations. Which I guess is mildly encouraging.

Daniel Lichtblau
Wolfram Research
mabshoff
Posted: Mon Apr 28, 2008 12:29 am
Guest
On Mar 31, 10:14 pm, Roman Pearce <rpear...@gmail.com> wrote:

Hi Roman,

<SNIP>

Sorry, I don't follow these groups too closely, but I happened to run
across your question while I was looking for some quote.

Quote:
Also, replying to mabshoff now, I would really like to know what
you meant here:

mabshoff wrote:
And the vast majority of benchmarks do not take long term effects
or even memory consumption into account. The most famous example I can
come up with are Gröbner Basis computations comparing the Buchberger
algorithm with F4 or F5.



I meant the following: If you look at Allan Steel's Gröbner Basis
Timings Page [1] you will see him compare his excellent F4
implementation in Magma with various "classical" Bucherberger
implementations in Singular and M2. But while Magma wipes the floor
with Singular2-0-5 and Macaulay 2 0.9.2 timing wise [note: these are
benchmarks from 2004, so those were "current" releases] if you look at
the amount of memory consumed by Magma and the competition you will
quickly see that Magma's F4 consumes considerably more memory than
either Singular or M2. So an uninformed onlooker will conclude that
Magma is king of the hill [it is when enough hardware, i.e. RAM is
available], but the story changes considerably when you consider some
PhD candidate hunting for cycles and RAM to get his/her computations
done. It is likely that the person will push the available resources
and chances are that you can deal with 10 times the required time much
more easily on a budget than with let's say five time the required
RAM.

I have been asked to do various GB computations where I had trouble to
fit the end result, i.e. the GBasis in 16GB RAM [in fact I never got
to the end, the problem was just too huge - at least with the current
algorithms] , so very few systems on this planet would be able to
compute the same GBasis with F4/F5 assuming I had actually finished.
So while I consider F4/F5 a huge achievement and also the
implementation of it in FGb as well as Magma superb coding it should
not lead people to believe that we should measure progress in
benchmarking CAS problems by time alone, but also consider RAM
consumption.

Cheers,

Michael




[1]: http://magma.maths.usyd.edu.au/users/allan/gb/
 
Page 3 of 3    Goto page Previous  1, 2, 3   All times are GMT - 5 Hours
The time now is Sat Sep 06, 2008 11:14 pm