mirror/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2026-05-14 19:07:15 +02:00
Code Issues Projects Releases Packages Wiki Activity

Bug#22555: STDDEV yields positive result for groups with only one row

Browse source 
When only one row was present, the subtraction of nearly the same number 
resulted in catastropic cancellation, introducing an error in the 
VARIANCE calculation near 1e-15.  That was sqrt()ed to get STDDEV, the 
error was escallated to near 1e-8.  

The simple fix of testing for a row count of 1 and forcing that to yield 
0.0 is insufficient, as two rows of the same value should also have a
variance of 0.0, yet the error would be about the same.

So, this patch changes the formula that computes the VARIANCE to be one
that is not subject to catastrophic cancellation.

In addition, it now uses only (faster-than-decimal) floating point numbers
to calculate, and renders that to other types on demand.
This commit is contained in:
cmiller@zippy.cornsilk.net 2006-12-22 15:37:37 -05:00
commit 50726b2322
5 changed files with 495 additions and 244 deletions
Download patch file Download diff file Expand all files Collapse all files

1
BitKeeper/etc/collapsed
View file

@ -19,3 +19,4 @@
454f8960jsVT_kMKJtZ9OCgXoba0xQ
4554a95d7txO1DuO9G3nAizI3SkFAA
4554b3722d71SbPiI2Gx-RhbZjmuIQ
45771031yRCoM_ZfONdYchPvVEgLRg