review of RNGs

[phred53]

Hi Paul,

> I do have some issues with the use of ent for measuring entropy, I can't
>comment on its other numbers - I can easily modify ent's entropy output by
>very simple whitening (CRC16 in my case), operations that don't change the
>total entropy of the incoming bitstream - in short I think it's easily fooled
>and its output should be taken with an appropriate grain of salt

I built 15MB and 100MB files for cmd0-3/6-7 modes using a Windows based capture utility written for OneRNG. Ran ent and dieharder against files.

I don't pretend to really understand but I couldn't see how CRC16, or for that matter any deterministic algorithm, "post processing" of the random data could _increase_ entropy,, but ent said it did. In addition I can't help but be suspicious of any "improvements" in dieharder test results after whitening; didn't seem to be the case for OneRNG.

There is a Crypto StackExchange discussion about entropy here - 
http://crypto.stackexchange.com/questions/10404/estimating-bits-of-entropy 

which appears to indicate that entropy is really about the physics of the device; measuring a bit stream doesn't tell one the true entropy and I would add particularly if whitened.

My conclusion is that OneRNG's entropy is what you basically state and what Shanon Entropy (ent) claims for raw mode bit streams. Assuming my capture utility set the mode and captured data correctly these are the Shanon entropy (ent) for 15MB files from the three raw modes -:

cmd1 - 6.63
cmd3 - 7.49
cmd7 - 7.99

Cheers,
Phred

...ent output...

Raw avalanche noise
me at mpc7:~$ ent Downloads/cmd1.rnd
Entropy = 6.626787 bits per byte.

Optimum compression would reduce the size
of this 15728640 byte file by 17 percent.

Chi square distribution for 15728640 samples is 67313567.53, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 66.5722 (127.5 = random).
Monte Carlo value for Pi is 3.926576233 (error 24.99 percent).
Serial correlation coefficient is 0.008946 (totally uncorrelated = 0.0).

Raw avalanche noise and RF noise
me at mpc7:~$ ent Downloads/cmd3.rnd
Entropy = 7.492954 bits per byte.

Optimum compression would reduce the size
of this 15728640 byte file by 6 percent.

Chi square distribution for 15728640 samples is 24970229.93, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 93.4802 (127.5 = random).
Monte Carlo value for Pi is 3.728984070 (error 18.70 percent).
Serial correlation coefficient is -0.181730 (totally uncorrelated = 0.0).

Raw RF noise
me at mpc7:~$ ent Downloads/cmd7.rnd
Entropy = 7.999989 bits per byte.

Optimum compression would reduce the size
of this 15728640 byte file by 0 percent.

Chi square distribution for 15728640 samples is 249.38, and randomly
would exceed this value 50.00 percent of the times.

Arithmetic mean value of data bytes is 127.5147 (127.5 = random).
Monte Carlo value for Pi is 3.141052246 (error 0.02 percent).
Serial correlation coefficient is 0.000182 (totally uncorrelated = 0.0).