Contents

Benchmarking

class evaluation.benchmark.Benchmark(mlmodel, recourse_method, factuals)

The benchmarking class contains all measurements. It is possible to run only individual evaluation metrics or all via one single call.

For every given factual, the benchmark object will generate one counterfactual example with the given recourse method.

Parameters
mlmodel: carla.models.MLModel

Black Box model we want to explain

recmodel: carla.recourse_methods.RecourseMethod

Recourse method we want to benchmark

factuals: pd.DataFrame

Instances we want to find counterfactuals

Methods

compute_ynn:

Computes y-Nearest-Neighbours for generated counterfactuals

compute_average_time:

Computes average time for generated counterfactual

compute_distances:

Calculates the distance measure and returns it as dataframe

compute_constraint_violation:

Computes the constraint violation per factual as dataframe

compute_redundancy:

Computes redundancy for each counterfactual

compute_success_rate:

Computes success rate for the whole recourse method.

run_benchmark:

Runs every measurement and returns every value as dict.

compute_average_time()

Computes average time for generated counterfactual

Returns
pd.DataFrame
Return type

DataFrame

compute_constraint_violation()

Computes the constraint violation per factual as dataframe

Returns
pd.Dataframe
Return type

DataFrame

compute_distances()

Calculates the distance measure and returns it as dataframe

Returns
pd.DataFrame
Return type

DataFrame

compute_redundancy()

Computes redundancy for each counterfactual

Returns
pd.Dataframe
Return type

DataFrame

compute_success_rate()

Computes success rate for the whole recourse method.

Returns
pd.Dataframe
Return type

DataFrame

compute_ynn()

Computes y-Nearest-Neighbours for generated counterfactuals

Returns
pd.DataFrame
Return type

DataFrame

run_benchmark()

Runs every measurement and returns every value as dict.

Returns
pd.DataFrame
Return type

DataFrame