This document introduces on how to use mlr3fairness
to
create audit reports with different tasks throughout the fairness
exploration process.
There are three main sections for this document. Which describe the details for the task, the model and the interpretability of the parameters.
Jump to section:
In this fairness report, we investigate the fairness of the following task:
#> <TaskClassif:adult_train> (700 x 6)
#> * Target: target
#> * Properties: twoclass
#> * Features (5):
#> - fct (4): education, marital_status, race, sex
#> - int (1): age
#> * Protected attribute: sex
Here we the basic details for the task.
Value | |
---|---|
Audit Date: | 2024-08-12 |
Task Name: | adult_train |
Number of observations: | 700 |
Number of features: | 6 |
Target Name: | target |
Feature Names: | age, education, marital_status, race, sex |
Protected Attribute(s): | sex |
We also report the number of missing values, types and the levels for each feature:
id | type | levels | label | fix_factor_levels | Missings…. | |
---|---|---|---|---|---|---|
2 | age | integer | NULL | NA | FALSE | 0 |
5 | education | factor | 10th , 11th , 12th , 1st-4th , 5th-6th , 7th-8th , 9th , Assoc-acdm , Assoc-voc , Bachelors , Doctorate , HS-grad , Masters , Preschool , Prof-school , Some-college | NA | FALSE | 0 |
8 | marital_status | factor | Divorced , Married-AF-spouse , Married-civ-spouse , Married-spouse-absent, Never-married , Separated , Widowed | NA | FALSE | 0 |
10 | race | factor | Amer-Indian-Eskimo, Asian-Pac-Islander, Black , Other , White | NA | FALSE | 0 |
12 | sex | factor | Female, Male | NA | FALSE | 0 |
13 | target | factor | <=50K, >50K | NA | FALSE | 0 |
We first look at the label distribution:
We could see the model that has been used in
resample_result
:
#> <LearnerClassifRpart:classif.rpart>: Classification Tree
#> * Model: -
#> * Parameters: xval=0
#> * Packages: mlr3, rpart
#> * Predict Types: response, [prob]
#> * Feature Types: logical, integer, numeric, factor, ordered
#> * Properties: importance, missings, multiclass, selected_features,
#> twoclass, weights
We furthermore report more than one fairness metric. Below metrics are the mean across all the resample results.
value | |
---|---|
fairness.acc | 0.1119356 |
fairness.equalized_odds | 0.1261559 |
fairness.fnr | 0.1051761 |
fairness.fpr | 0.1471356 |
fairness.ppv | 0.0931866 |
fairness.pp | 0.1493584 |
We can furthermore employ several visualizations to report the fairness. For example, the fairness and accuracy trade off, compare metrics visualization and the fairness prediction density of our model. For more detailed usage and examples, you may want to check the visualization vignette.
Finally, we use the external package to gain further insight into our
model. For the following example we use the iml
package as
a demonstration. We need first extract the learner from
resample_result
and wrap it in a Predictor
object.
You could generate the variable importance plot like this
Or generate the feature effects plot:
For more details on interpretability, check the documentation of the
iml
package.