Overview
The Anomaly Detection feature within DataTrust provides a robust set of tools to identify anomalies, which are also commonly referred to as outliers, within a given dataset. Anomalies hold significant relevance across diverse domains, including finance, healthcare, and security, as they serve as crucial indicators for detecting irregularities like fraudulent transactions or rare diseases.
These methods can be categorized into two primary types: Statistical Methods and Machine Learning Models
Statistical Methods
Definitions and Interpretations
Statistical methods offer a foundation for anomaly detection by leveraging mathematical principles and patterns inherent in the data. These methods excel in identifying anomalies based on statistical deviations from the norm.
1. IQR (Interquartile Range)
- Description: Measures the spread of a dataset between its upper and lower quartiles.
- Anomaly Criteria: A record is considered anomalous if it falls below Q1−threshold×IQR or above �3+threshold×���Q3+threshold×IQR.
Default Threshold: 1.5
2. MAD (Median Absolute Deviation)
- Description: Robust measure of dataset spread, calculated as the median of absolute deviations from the dataset's median.
- Anomaly Criteria: A record is considered anomalous if it falls below Median−threshold×MAD or above Median+threshold×���Median+threshold×MAD.
- Default Threshold: 3
3. Z-score
- Description: Indicates how many standard deviations a data point is from the mean.
- Anomaly Criteria: A record with a Z-score greater than the set threshold is considered anomalous.
- Recommended: Use for normally-distributed datasets; default threshold is 3.
Machine Learning Models
Explanation and Interpretations
Machine learning models harness the power of advanced algorithms to detect anomalies in datasets. They are particularly effective at recognizing complex patterns and outliers that may not be evident through traditional statistical approaches.
1. Isolation Forest (iforest)
- Description: Uses tree structures to isolate anomalies.
- Criteria: Records requiring fewer splits to isolate are anomalies.
2. Local Outlier Factor (lof)
- Description: Measures the density discrepancy of each data point compared to its neighbours.
- Criteria: Records with significantly lower density than neighbours are considered anomalies.
3. k-Nearest Neighbours (knn)
- Description: Calculates the distance between a record and its k nearest neighbours.
- Criteria: Records with notably higher distances to their neighbours are considered anomalies.
4. Support Vector Machine (svm)
- Description: Uses a hyperplane to classify data points.
- Criteria: Records falling on the incorrect side of the hyperplane or with a low margin are considered anomalies.
When to Use Which Model?
- Use IQR, MAD, and Z-score for univariate datasets; they are simple and straightforward.
- Use machine learning models for multivariate datasets; they can detect complex anomalies.
Anomaly Interpretations
Global Explanation
Ranks the overall importance of each feature for detecting anomalies using the trained machine learning model.
Local Explanation
Uses SHAP (SHapley Additive exPlanations) values to quantify each feature's contribution to a particular record being anomalous.
Anomaly Score
The anomaly score quantifies the degree of abnormality for each data point and varies depending on the algorithm used. A higher score typically indicates a higher likelihood of the data point being an anomaly.