Monitor the stability of a pandas or spark dataframe in python
Population Shift Monitoring
popmon is a package that allows one to check the stability of a dataset. popmon works with both pandas and spark datasets.
popmon creates histograms of features binned in time-slices, and compares the stability of the profiles and distributions of those histograms using statistical tests, both over time and with respect to a reference. It works with numerical, ordinal, categorical features, and the histograms can be higher-dimensional, e.g. it can also track correlations between any two features. popmon can automatically flag and alert on changes observed over time, such as trends, shifts, peaks, outliers, anomalies, changing correlations, etc, using monitoring business rules.
Announcements
Spark 3.0
With Spark 3.0, based on Scala 2.12, make sure to pick up the