ROC AUC for learners that don't predict lp scores

[bblodfon]

Hi!

Based on the code below, we see that the example ranger survival learner has no output prediction of type lp, while all ROC AUCs measures supported currently need that particular type to be calculated:

library(mlr3proba)
#> Loading required package: mlr3
library(mlr3extralearners)

task = tsk('rats')
task$select(cols = c('litter', 'rx'))

ranger_lrn = lrn('surv.ranger')
p = ranger_lrn$train(task)$predict(task)
# no `lp` prediction
p
#> <PredictionSurv> for 300 observations:
#>     row_ids time status     crank     distr
#>           1  101  FALSE  7.971548 <list[1]>
#>           2   49   TRUE 12.999444 <list[1]>
#>           3  104  FALSE 12.999444 <list[1]>
#> ---                                        
#>         298   92  FALSE  2.416444 <list[1]>
#>         299  104  FALSE  7.436310 <list[1]>
#>         300  102  FALSE  7.436310 <list[1]>

p$score(msr('surv.uno_auc'), task, train_set = 1:task$nrow)
#> surv.uno_auc 
#>          NaN

# all ROC AUCs measures require `lp` prediction type
for (measure in mlr_measures$keys(pattern = '^surv.*auc')) {
  print(mlr_measures$get(measure)$predict_type)
}
#> [1] "lp"
#> [1] "lp"
#> [1] "lp"
#> [1] "lp"

^{Created on 2022-06-07 by the reprex package (v2.0.1)}

Could crank be somehow used in this case instead of lp (setting lp == crank? - maybe doesn't make sense to do that in general though). Maybe consider supporting a more general ROC AUC score (if it exists, a research question :) that requires a relative risk or distribution prediction?

[RaphaelS1]

I love this question because it has an easy answer! They don't exist.

If you look at the documentation in the survAUC package (or any of the papers the code comes from) you'll see they either: 1) expect a linear predictor from a Cox model; or more generally 2) expect a one-to-one relationship between the predicted risk and the survival distribution (which basically translates to requiring a linear predictor).

However time-dependent C-indices exist, and we have an open issue for this here #267