This document introduces on how to use
mlr3fairness to create audit reports with different tasks throughout the fairness exploration process.
There are three main sections for this document. Which describe the details for the task, the model and the interpretability of the parameters.
Jump to section:
In this fairness report, we investigate the fairness of the following task:
#> <TaskClassif:compas> (6172 x 12) #> * Target: two_year_recid #> * Properties: twoclass #> * Features (11): #> - fct (6): age_cat, c_charge_degree, is_recid, race, score_text, sex #> - int (5): age, days_b_screening_arrest, decile_score, #> length_of_stay, priors_count
Here we could get the basic details for the task.
|Number of observations:||6172|
|Number of features:||12|
|Feature Names:||age, age_cat, c_charge_degree, days_b_screening_arrest, decile_score, is_recid, length_of_stay, priors_count, race, score_text, sex|
|The Protected Attribute:||sex|
We could also report the number of missing values, types and the levels for each feature:
|age||0||factor||25 - 45 , Greater than 45, Less than 25||FALSE|
|priors_count||0||factor||African-American, Asian , Caucasian , Hispanic , Native American , Other||FALSE|
|race||0||factor||High , Low , Medium||FALSE|
We first look at the label distribution:
We could see the model that has been used in
#> <LearnerClassifRpart:classif.rpart>: Classification Tree #> * Model: - #> * Parameters: xval=0 #> * Packages: mlr3, rpart #> * Predict Types: response, [prob] #> * Feature Types: logical, integer, numeric, factor, ordered #> * Properties: importance, missings, multiclass, selected_features, #> twoclass, weights
We could report more than one fairness metric, but keep in mind. Below the metrics are the mean of all the resample results.
We could also use visualization to report the fairness. For example, the fairness and accuracy trade off, compare metrics visualization and the fairness prediction density of the model
classif.rpart . For more detailed usage and examples, you may want to check the visualization vignette.
Finally, we could use the external package to analyze the interpretability of the parameters. For the following example we choose
iml as a demonstration. We need first extract the learner from
resample_result and retrain it.
You could generate the variable importance plot like this