README.md

Note: the current releases of this toolbox are a beta release, to test working with Haskell's, Python's, and R's code repositories.

Metrics provides implementations of various supervised machine learning evaluation metrics in the following languages:

• Python easy_install ml_metrics
• R install.packages("Metrics") from the R prompt
• Haskell cabal install Metrics
• MATLAB / Octave (clone the repo & run setup from the MATLAB command line)

For more detailed installation instructions, see the README for each implementation.

EVALUATION METRICS

Evaluation Metric	Python	R	Haskell	MATLAB / Octave
Absolute Error (AE)	✓	✓	✓	✓
Average Precision at K (APK, AP@K)	✓	✓	✓	✓
Area Under the ROC (AUC)	✓	✓	✓	✓
Classification Error (CE)	✓	✓	✓	✓
Gini				✓
Levenshtein	✓		✓	✓
Log Loss (LL)	✓	✓	✓	✓
Mean Log Loss (LogLoss)	✓	✓	✓	✓
Mean Absolute Error (MAE)	✓	✓	✓	✓
Mean Average Precision at K (MAPK, MAP@K)	✓	✓	✓	✓
Mean Quadratic Weighted Kappa	✓	✓		✓
Mean Squared Error (MSE)	✓	✓	✓	✓
Mean Squared Log Error (MSLE)	✓	✓	✓	✓
Normalized Gini				✓
Quadratic Weighted Kappa	✓	✓		✓
Relative Absolute Error (RAE)		✓
Root Mean Squared Error (RMSE)	✓	✓	✓	✓
Relative Squared Error (RSE)		✓
Root Relative Squared Error (RRSE)		✓
Root Mean Squared Log Error (RMSLE)	✓	✓	✓	✓
Squared Error (SE)	✓	✓	✓	✓
Squared Log Error (SLE)	✓	✓	✓	✓

TO IMPLEMENT

• F1 score
• Multiclass log loss
• Lift
• Average Precision for binary classification
• precision / recall break-even point
• cross-entropy
• True Pos / False Pos / True Neg / False Neg rates
• precision / recall / sensitivity / specificity
• mutual information

HIGHER LEVEL TRANSFORMATIONS TO HANDLE

• GroupBy / Reduce
• Weight individual samples or groups

PROPERTIES METRICS CAN HAVE

(Nonexhaustive and to be added in the future)

• Min or Max (optimize through minimization or maximization)
• Binary Classification
  • Scores predicted class labels
  • Scores predicted ranking (most likely to least likely for being in one class)
  • Scores predicted probabilities
• Multiclass Classification
  • Scores predicted class labels
  • Scores predicted probabilities
• Regression
• Discrete Rater Comparison (confusion matrix)