I need 4 heavily randomized, entropic files. Max size of 64kb (I am
converting it to 0's and 1's in a text file, so it will be 512kb
then... which is the maximum of the tool)

These files should be unable to be further compressed by any existing
compression utility.

I have just successfully done one test, but I used a central portion
of the Calgary Corpus file since it was to large as a whole. I used
7zip to compress, and obtained admittedly only a few bits of
compression in several different locations after accounting for a
command section, and identification, and only if I quadrupled the
estimated file size to increase the 'number of bits saved'.

This is bad sampling and I admit it. Therefore I am asking for help
generating the admittedly necessary test files.

4 should give me a baseline to see if I am being successful or not.

The million one is to large, my software cant try it. Nice slam on me,
I have since learned a lil more, post more information.

This page is not available.

on the second file source.

The million one is to large, my software cant try it.

Nice slam on me,

sorry I was dismayed my rl name was used on that report, and felt very
attacked.

I just dont want any tainted samples I guess. I am very concerned to
keep this as legit as I can... improper sampling can be taken the
wrong way imo. By me, by my system, and by everyone.

if I ran every 16 bit string possibility in order, and hooked them up,
and then parsed it at 4 bits and ran it at 4 bits, does this stand for
a random file, or would it seem to ordered?

of it, about 10% can be compressed, but results vary with how much,
and typically it is ineffectual to even try to attend to the
remainder. I literally just cost efficiently whip out the 10% as I
can... but tests will see, so dont act like I am preaching it

Well the numbers I am getting from small files are to unbelievable
atm... thats why I wanted a medium sized (since thats the max of the
tool atm) 'uncompressible' file.

On Feb 17, 3:36 pm, Einstein <michae...@gmail.com> wrote:

Well the numbers I am getting from small files are to unbelievable
atm... thats why I wanted a medium sized (since thats the max of the
tool atm) 'uncompressible' file.

Run your process on the following small file:

784951623

That's 9 random digits. Here are those digits as a single number in
binary, if you prefer to work only in binary:

101110110010010110100101000111

If you are able to compress the above to something smaller than the 30
bits it takes to represent them, and successfully decompress the
compressed data back into the source, congratulations, you are
mentally ill.

To satisfy the "RAD" people, there was a indeed context in generating
the above, but telling you wouldn't help at all because the
information needed to make sense of the context equals the size of the
output.

On Feb 18, 9:51 am, Jim Leonard <MobyGa...@gmail.com> wrote:



On Feb 17, 3:36 pm, Einstein <michae...@gmail.com> wrote:

Well the numbers I am getting from small files are to unbelievable
atm... thats why I wanted a medium sized (since thats the max of the
tool atm) 'uncompressible' file.

Run your process on the following small file:

784951623

That's 9 random digits. Here are those digits as a single number in
binary, if you prefer to work only in binary:

101110110010010110100101000111

If you are able to compress the above to something smaller than the 30
bits it takes to represent them, and successfully decompress the
compressed data back into the source, congratulations, you are
mentally ill.

To satisfy the "RAD" people, there was a indeed context in generating
the above, but telling you wouldn't help at all because the
information needed to make sense of the context equals the size of the
output.

111011000001110000101100100111001010110010001100011011000100110011001100
is actual bits for the 9 characters.

1110
1100
0001
1100
0010
1100
1001
1100
1010
1100
1000
1100
0110
1100
0100
1100
1100
1100

Dictionary

1 = 1100
0 = all else is normal

0,1100,1,0,0001,1,0,0010,1,0,1001,1,0,1010,1,0,1000,1,0,0110,1,0,0100,1,1,1

Total size = 50 bits

Original size = 72 bits

ofc it would not be 'that easy', but still having nearly 1/3 of the
space to work with, oh jeez, you really tried.

On Feb 16, 7:47 am, Einstein <michae...@gmail.com> wrote:



I need 4 heavily randomized, entropic files. Max size of 64kb (I am
converting it to 0's and 1's in a text file, so it will be 512kb
then... which is the maximum of the tool)

These files should be unable to be further compressed by any existing
compression utility.

I have just successfully done one test, but I used a central portion
of the Calgary Corpus file since it was to large as a whole. I used
7zip to compress, and obtained admittedly only a few bits of
compression in several different locations after accounting for a
command section, and identification, and only if I quadrupled the
estimated file size to increase the 'number of bits saved'.

This is bad sampling and I admit it. Therefore I am asking for help
generating the admittedly necessary test files.

4 should give me a baseline to see if I am being successful or not.

I don't think it is going to be worth the time and effort you plan to
put into it, but its your choice :-)

Try this file:
http://marknelson.us/attachments/million-digit-challenge/AMillionRand...

More information on this here:
http://marknelson.us/2006/06/20/million-digit-challenge

And you can try this file:
http://www.geocities.com/patchnpuki/other/original.gz

More information on this here:
http://www.geocities.com/patchnpuki/other/compression.htm

You can create files of your required size from these. These are both
fairly random files and should not be easily compressible. I think
someone did manage to compress the million-digit file long ago by a
few bytes but I can't recall the details. So, the files are not
perfect but should be good enough to try out your ideas.

Best of luck,
Sachin Garg [India]www.sachingarg.com|www.c10n.info

On Feb 15, 10:33 pm, Sachin Garg <schn...@gmail.com> wrote:



On Feb 16, 7:47 am, Einstein <michae...@gmail.com> wrote:

I need 4 heavily randomized, entropic files. Max size of 64kb (I am
converting it to 0's and 1's in a text file, so it will be 512kb
then... which is the maximum of the tool)

These files should be unable to be further compressed by any existing
compression utility.

I have just successfully done one test, but I used a central portion
of the Calgary Corpus file since it was to large as a whole. I used
7zip to compress, and obtained admittedly only a few bits of
compression in several different locations after accounting for a
command section, and identification, and only if I quadrupled the
estimated file size to increase the 'number of bits saved'.

This is bad sampling and I admit it. Therefore I am asking for help
generating the admittedly necessary test files.

4 should give me a baseline to see if I am being successful or not.

I don't think it is going to be worth the time and effort you plan to
put into it, but its your choice :-)

Try this file:
http://marknelson.us/attachments/million-digit-challenge/AMillionRand...

More information on this here:
http://marknelson.us/2006/06/20/million-digit-challenge

And you can try this file:
http://www.geocities.com/patchnpuki/other/original.gz

More information on this here:
http://www.geocities.com/patchnpuki/other/compression.htm

You can create files of your required size from these. These are both
fairly random files and should not be easily compressible. I think
someone did manage to compress the million-digit file long ago by a
few bytes but I can't recall the details. So, the files are not
perfect but should be good enough to try out your ideas.

Best of luck,
Sachin Garg [India]www.sachingarg.com|www.c10n.info

If you write the million random digits as a table with 20000 rows and
50 columns, then add up the columns, then each of the 50 sums will be
an even number. I discovered this a couple of years ago.

The original data was not quite random due to biases in the hardware
generator, so they took the data (20000 punched cards with 50 digits
each) and added adjacent pairs of cards to generate the published
data. Seehttp://www.rand.org/pubs/monograph_reports/MR1418/index.html

If you could guess one original card and you knew the order of the
published cards, then you could recover all the biased data by
subtracting adjacent pairs and compress it slightly. Unfortunately,
the cards were shuffled or mixed up first. The document does not
describe this, but I know they are not in the original order because
if they were, you could alternately add and subtract rows and they
would sum to 0 (mod 10). But they do not. Also, there is a weak
correlation between adjacent cards and also to the second card in
either direction. There should not be a correlation to the second
card unless a card was inserted between them. In any case the
correlation is too weak to compress the file by more than a few bits.

-- Matt Mahoney