statsutils
- Statistics fundamentals¶
statsutils
provides tools aimed primarily at descriptive
statistics for data analysis, such as mean()
(average),
median()
, variance()
, and many others,
The Stats
type provides all the main functionality of the
statsutils
module. A Stats
object wraps a given dataset,
providing all statistical measures as property attributes. These
attributes cache their results, which allows efficient computation of
multiple measures, as many measures rely on other measures. For
example, relative standard deviation (Stats.rel_std_dev
)
relies on both the mean and standard deviation. The Stats object
caches those results so no rework is done.
The Stats
type’s attributes have module-level counterparts for
convenience when the computation reuse advantages do not apply.
>>> stats = Stats(range(42))
>>> stats.mean
20.5
>>> mean(range(42))
20.5
Statistics is a large field, and statsutils
is focused on a few
basic techniques that are useful in software. The following is a brief
introduction to those techniques. For a more in-depth introduction,
Statistics for Software,
an article I wrote on the topic. It introduces key terminology vital
to effective usage of statistics.
Statistical moments¶
Python programmers are probably familiar with the concept of the mean or average, which gives a rough quantitiative middle value by which a sample can be can be generalized. However, the mean is just the first of four moment-based measures by which a sample or distribution can be measured.
The four Standardized moments are:
- Mean -
mean()
- theoretical middle value- Variance -
variance()
- width of value dispersion- Skewness -
skewness()
- symmetry of distribution- Kurtosis -
kurtosis()
- “peakiness” or “long-tailed”-ness
For more information check out the Moment article on Wikipedia.
Keep in mind that while these moments can give a bit more insight into the shape and distribution of data, they do not guarantee a complete picture. Wildly different datasets can have the same values for all four moments, so generalize wisely.
Robust statistics¶
Moment-based statistics are notorious for being easily skewed by
outliers. The whole field of robust statistics aims to mitigate this
dilemma. statsutils
also includes several robust statistical methods:
- Median - The middle value of a sorted dataset
- Trimean - Another robust measure of the data’s central tendency
- Median Absolute Deviation (MAD) - A robust measure of variability, a natural counterpart to
variance()
.- Trimming - Reducing a dataset to only the middle majority of data is a simple way of making other estimators more robust.
Online and Offline Statistics¶
Unrelated to computer networking, online statistics involve
calculating statistics in a streaming fashion, without all the data
being available. The Stats
type is meant for the more
traditional offline statistics when all the data is available. For
pure-Python online statistics accumulators, look at the Lithoxyl
system instrumentation package.
-
class
boltons.statsutils.
Stats
(data, default=0.0, use_copy=True, is_sorted=False)[source]¶ The
Stats
type is used to represent a group of unordered statistical datapoints for calculations such as mean, median, and variance.Parameters: - data (list) – List or other iterable containing numeric values.
- default (float) – A value to be returned when a given
statistical measure is not defined. 0.0 by default, but
float('nan')
is appropriate for stricter applications. - use_copy (bool) – By default Stats objects copy the initial
data into a new list to avoid issues with
modifications. Pass
False
to disable this behavior. - is_sorted (bool) – Presorted data can skip an extra sorting step for a little speed boost. Defaults to False.
-
clear_cache
()[source]¶ Stats
objects automatically cache intermediary calculations that can be reused. For instance, accessing thestd_dev
attribute after thevariance
attribute will be significantly faster for medium-to-large datasets.If you modify the object by adding additional data points, call this function to have the cached statistics recomputed.
-
count
¶ The number of items in this Stats object. Returns the same as
len()
on a Stats object, but provided for pandas terminology parallelism.
-
describe
(quantiles=None, format=None)[source]¶ Provides standard summary statistics for the data in the Stats object, in one of several convenient formats.
Parameters: - quantiles (list) – A list of numeric values to use as
quantiles in the resulting summary. All values must be
0.0-1.0, with 0.5 representing the median. Defaults to
[0.25, 0.5, 0.75]
, representing the standard quartiles. - format (str) – Controls the return type of the function,
with one of three valid values:
"dict"
gives back adict
with the appropriate keys and values."list"
is a list of key-value pairs in an order suitable to pass to an OrderedDict or HTML table."text"
converts the values to text suitable for printing, as seen below.
Here is the information returned by a default
describe
, as presented in the"text"
format:>>> stats = Stats(range(1, 8)) >>> print(stats.describe(format='text')) count: 7 mean: 4.0 std_dev: 2.0 mad: 2.0 min: 1 0.25: 2.5 0.5: 4 0.75: 5.5 max: 7
For more advanced descriptive statistics, check out my blog post on the topic Statistics for Software.
- quantiles (list) – A list of numeric values to use as
quantiles in the resulting summary. All values must be
0.0-1.0, with 0.5 representing the median. Defaults to
-
format_histogram
(bins=None, **kw)[source]¶ Produces a textual histogram of the data, using fixed-width bins, allowing for simple visualization, even in console environments.
>>> data = list(range(20)) + list(range(5, 15)) + [10] >>> print(Stats(data).format_histogram()) 0.0: 5 ################################ 4.4: 8 ################################################### 8.9: 11 ###################################################################### 13.3: 5 ################################ 17.8: 2 #############
In this histogram, five values are between 0.0 and 4.4, eight are between 4.4 and 8.9, and two values lie between 17.8 and the max.
You can specify the number of bins, or provide a list of bin boundaries themselves. If no bins are provided, as in the example above, Freedman’s algorithm for bin selection is used.
Parameters: - bins (int) – Maximum number of bins for the histogram. Also accepts a list of floating-point bin boundaries. If the minimum boundary is still greater than the minimum value in the data, that boundary will be implicitly added. Defaults to the bin boundaries returned by Freedman’s algorithm.
- bin_digits (int) – Number of digits to round each bin to. Note that bins are always rounded down to avoid clipping any data. Defaults to 1.
- width (int) – integer number of columns in the longest line in the histogram. Defaults to console width on Python 3.3+, or 80 if that is not available.
- format_bin (callable) – Called on each bin to create a label for the final output. Use this function to add units, such as “ms” for milliseconds.
Should you want something more programmatically reusable, see the
get_histogram_counts()
method, the output of is used by format_histogram. Thedescribe()
method is another useful summarization method, albeit less visual.
-
get_histogram_counts
(bins=None, **kw)[source]¶ Produces a list of
(bin, count)
pairs comprising a histogram of the Stats object’s data, using fixed-width bins. SeeStats.format_histogram()
for more details.Parameters: The output of this method can be stored and/or modified, and then passed to
statsutils.format_histogram_counts()
to achieve the same text formatting as theformat_histogram()
method. This can be useful for snapshotting over time.
-
get_quantile
(q)[source]¶ Get a quantile from the dataset. Quantiles are floating point values between
0.0
and1.0
, with0.0
representing the minimum value in the dataset and1.0
representing the maximum.0.5
represents the median:>>> Stats(range(100)).get_quantile(0.5) 49.5
-
get_zscore
(value)[source]¶ Get the z-score for value in the group. If the standard deviation is 0, 0 inf or -inf will be returned to indicate whether the value is equal to, greater than or below the group’s mean.
-
iqr
¶ Inter-quartile range (IQR) is the difference between the 75th percentile and 25th percentile. IQR is a robust measure of dispersion, like standard deviation, but safer to compare between datasets, as it is less influenced by outliers.
-
kurtosis
¶ Indicates how much data is in the tails of the distribution. The result is always positive, with the normal “bell-curve” distribution having a kurtosis of 3.
http://en.wikipedia.org/wiki/Kurtosis
See the module docstring for more about statistical moments.
-
mad
¶ Median Absolute Deviation is a robust measure of statistical dispersion: http://en.wikipedia.org/wiki/Median_absolute_deviation
-
max
¶ The maximum value present in the data.
-
mean
¶ The arithmetic mean, or “average”. Sum of the values divided by the number of values.
-
median
¶ The median is either the middle value or the average of the two middle values of a sample. Compared to the mean, it’s generally more resilient to the presence of outliers in the sample.
-
median_abs_dev
¶ Median Absolute Deviation is a robust measure of statistical dispersion: http://en.wikipedia.org/wiki/Median_absolute_deviation
-
min
¶ The minimum value present in the data.
-
pearson_type
¶
-
rel_std_dev
¶ Standard deviation divided by the absolute value of the average.
-
skewness
¶ Indicates the asymmetry of a curve. Positive values mean the bulk of the values are on the left side of the average and vice versa.
http://en.wikipedia.org/wiki/Skewness
See the module docstring for more about statistical moments.
-
std_dev
¶ Standard deviation. Square root of the variance.
-
trim_relative
(amount=0.15)[source]¶ A utility function used to cut a proportion of values off each end of a list of values. This has the effect of limiting the effect of outliers.
Parameters: amount (float) – A value between 0.0 and 0.5 to trim off of each side of the data.
-
trimean
¶ The trimean is a robust measure of central tendency, like the median, that takes the weighted average of the median and the upper and lower quartiles.
-
variance
¶ Variance is the average of the squares of the difference between each value and the mean.
-
boltons.statsutils.
describe
(data, quantiles=None, format=None)[source]¶ A convenience function to get standard summary statistics useful for describing most data. See
Stats.describe()
for more details.>>> print(describe(range(7), format='text')) count: 7 mean: 3.0 std_dev: 2.0 mad: 2.0 min: 0 0.25: 1.5 0.5: 3 0.75: 4.5 max: 6
See
Stats.format_histogram()
for another very useful summarization that uses textual visualization.
-
boltons.statsutils.
format_histogram_counts
(bin_counts, width=None, format_bin=None)[source]¶ The formatting logic behind
Stats.format_histogram()
, which takes the output ofStats.get_histogram_counts()
, and passes them to this function.Parameters: - bin_counts (list) – A list of bin values to counts.
- width (int) – Number of character columns in the text output, defaults to 80 or console width in Python 3.3+.
- format_bin (callable) – Used to convert bin values into string labels.
-
boltons.statsutils.
iqr
(data, default=0.0)¶ Inter-quartile range (IQR) is the difference between the 75th percentile and 25th percentile. IQR is a robust measure of dispersion, like standard deviation, but safer to compare between datasets, as it is less influenced by outliers.
>>> iqr([1, 2, 3, 4, 5]) 2 >>> iqr(range(1001)) 500
-
boltons.statsutils.
kurtosis
(data, default=0.0)¶ Indicates how much data is in the tails of the distribution. The result is always positive, with the normal “bell-curve” distribution having a kurtosis of 3.
http://en.wikipedia.org/wiki/Kurtosis
See the module docstring for more about statistical moments.
>>> kurtosis(range(9)) 1.99125
With a kurtosis of 1.99125, [0, 1, 2, 3, 4, 5, 6, 7, 8] is more centrally distributed than the normal curve.
-
boltons.statsutils.
mean
(data, default=0.0)¶ The arithmetic mean, or “average”. Sum of the values divided by the number of values.
>>> mean(range(20)) 9.5 >>> mean(list(range(19)) + [949]) # 949 is an arbitrary outlier 56.0
-
boltons.statsutils.
median
(data, default=0.0)¶ The median is either the middle value or the average of the two middle values of a sample. Compared to the mean, it’s generally more resilient to the presence of outliers in the sample.
>>> median([2, 1, 3]) 2 >>> median(range(97)) 48 >>> median(list(range(96)) + [1066]) # 1066 is an arbitrary outlier 48
-
boltons.statsutils.
median_abs_dev
(data, default=0.0)¶ Median Absolute Deviation is a robust measure of statistical dispersion: http://en.wikipedia.org/wiki/Median_absolute_deviation
>>> median_abs_dev(range(97)) 24.0
-
boltons.statsutils.
pearson_type
(data, default=0.0)¶
-
boltons.statsutils.
rel_std_dev
(data, default=0.0)¶ Standard deviation divided by the absolute value of the average.
http://en.wikipedia.org/wiki/Relative_standard_deviation
>>> print('%1.3f' % rel_std_dev(range(97))) 0.583
-
boltons.statsutils.
skewness
(data, default=0.0)¶ Indicates the asymmetry of a curve. Positive values mean the bulk of the values are on the left side of the average and vice versa.
http://en.wikipedia.org/wiki/Skewness
See the module docstring for more about statistical moments.
>>> skewness(range(97)) # symmetrical around 48.0 0.0 >>> left_skewed = skewness(list(range(97)) + list(range(10))) >>> right_skewed = skewness(list(range(97)) + list(range(87, 97))) >>> round(left_skewed, 3), round(right_skewed, 3) (0.114, -0.114)
-
boltons.statsutils.
std_dev
(data, default=0.0)¶ Standard deviation. Square root of the variance.
>>> std_dev(range(97)) 28.0
-
boltons.statsutils.
trimean
(data, default=0.0)¶ The trimean is a robust measure of central tendency, like the median, that takes the weighted average of the median and the upper and lower quartiles.
>>> trimean([2, 1, 3]) 2.0 >>> trimean(range(97)) 48.0 >>> trimean(list(range(96)) + [1066]) # 1066 is an arbitrary outlier 48.0
-
boltons.statsutils.
variance
(data, default=0.0)¶ Variance is the average of the squares of the difference between each value and the mean.
>>> variance(range(97)) 784.0