Evaluation procedures for forecasting with spatio-temporal data

This repository contains the research compendium of "Evaluation procedures for forecasting with spatio-temporal data", authored by Mariana Oliveira, Luis Torgo, and Vitor Santos Costa, and presented at ECML-PKDD 2018.

You are free to use and/or adapt the code we freely provide. However, we do require that if you do that you cite the paper where these results and code were published:

Oliveira, Mariana, Luís Torgo, and Vítor Santos Costa. "Evaluation Procedures for Forecasting with Spatio-Temporal Data." In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD (pp. 703–718). Springer, Cham, 2018. doi: 10.1007/978-3-030-10925-7_43

BibTeX citation:

@inproceedings{Oliveira2018, author = {Mariana Oliveira and Lu{'{\i}}s Torgo and V{'{\i}}tor Santos Costa}, editor = {Michele Berlingerio and Francesco Bonchi and Thomas G{"{a}}rtner and Neil Hurley and Georgiana Ifrim}, title = {Evaluation Procedures for Forecasting with Spatio-Temporal Data}, booktitle = {Machine Learning and Knowledge Discovery in Databases - European Conference, {ECML} {PKDD} 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part {I}}, series = {Lecture Notes in Computer Science}, volume = {11051}, pages = {703--718}, publisher = {Springer}, year = {2018}, url = {https://doi.org/10.1007/978-3-030-10925-7\_43}, doi = {10.1007/978-3-030-10925-7_43}}

If you adapt the code to your own needs, you are also required to maintain information on your code concerning the original source of the code (e.g. the URL of this page) and a reference to the original paper.

Other supplementary material (e.g., PDF of post-print, slides of conference presentation) available at http://www.dcc.fc.up.pt/~moliveira/publication/18-ecml-evaluation-spatiotemporal/.

Note: An extended version of this article has since been published at the open-access journal Mathematics. Code available here.

Prerequisites

To install this package, run:

library(devtools)  # You need to install this package!
install_github("mrfoliveira/Evaluation-procedures-for-forecasting-with-spatio-temporal-data", ref="master")

To replicate figures, installing the following package is also necessary:

if(!("ggplot2" in installed.packages())) install.packages("ggplot2")

Reproducing experiments

To run experiments, run the following lines from the main directory:

library(STEvaluationPaper)
PATH <- system.file("inst/", package="STEvaluationPaper")
source(paste0(PATH, "/analysis/step1_gen_data.R"))
source(paste0(PATH, "/analysis/step2_artificial_experiments.R"))
source(paste0(PATH, "/analysis/step3_real_experiments.R"))

To generate an HTML report containing all figures and tables in the article, run:

library(STEvaluationPaper)
knitr::knit(system.file("inst/results/report.Rmd", package="STEvaluationPaper"))

Contents

The repository is organized as follows:

• data - Contains the real-world data sets used in the experiments in an appropriate format
• inst/analysis - Contains scripts for generating data and launching experiments
• inst/results - Contains files of the results presented in the paper
• man - Contains function documentation
• R - Contains reusable functions

R

• analyse_utils.R - functions that can be used to summarize produced by experiments
• calc_metrics.R - functions to calculate error metrics
• experiment_utils.R - functions designed to launch experiments
• fold_alloc.R - functions for allocating observations to cross-validation folds
• gen_data_utils.R - functions needed to generate lists of STARMA-generated datasets, depends on starma_utils.R
• methods.R - functions implementing different error estimation methods, depends on fold_alloc.R, utils.R
• starma_utils.R - functions needed to generate STARMA artificial data and check coefficient stationarity
• st_indicators.R - functions designed to calculate spatio-temporal indicators
• utils.R - general utility functions used inside some evaluation methods
• workflow.R - functions needed for the multiple steps in experiments, including learning, generating and evaluating predictions

inst/analysis

• step1_gen_data.R - data generation script
• step2_artificial_experiments.R - running experiments on artificially generated datasets
• step3_real_experiments.R - calculating spatio-temporal indicators and running experiments on real data sets