Values That Can't Be Right
Values That Can't Be Right: Values That Can't Be Right: Sentinels and Outliers, Checking Your Data with assert.
Updated
- Values That Can't Be Right: Sentinels and Outliers (Apprentice): A stand-in such as -999 for "no reading" and a sensor that reads 152 °F in the sun both throw off an average, so replace turns stand-ins into NaN, describe and between find the values that can't be true, and a suspect column flags them before they're blanked, while real but unusual readings stay.
- Checking Your Data with assert (Journeyman): assert stops the code with an AssertionError and your message whenever a condition is False, so checks on ranges with between and all, on repeated timestamps and on missing values, gathered in one check_readings function, catch a cleaning step that went wrong every time the cleaning runs.