Quickstart

Installation

To install quanda from a local clone of the repository, use the following command in your terminal:

(.venv) $ pip install -e .

quanda requires Python 3.10, 3.11 or 3.12. It is recommended to use a virtual environment to install the package.

Note

In the examples that follow, we will demonstrate the generation of explanations generated using SimilarityInfluence data attributor from Captum.

Using Metrics

To begin using quanda metrics, you need the following components:

  • Trained PyTorch Model ( model ): A PyTorch model that has already been trained on a relevant dataset. As a placeholder, we used the layer name “avgpool” below. Please replace it with the name of one of the layers in your model.

  • PyTorch Dataset ( train_set ): The dataset used during the training of the model.

  • Test Dataset ( eval_set ): The dataset to be used as test inputs for generating explanations. Explanations are generated with respect to an output neuron corresponding to a certain class. This class can be selected to be the ground truth label of the test points, or the classes predicted by the model. In the following we will use the predicted labels to generate explanations.

Next, we demonstrate how to evaluate explanations using the Model Randomization metric.

1. Import dependencies and library components

from torch.utils.data import DataLoader
from tqdm import tqdm

from quanda.explainers.wrappers import CaptumSimilarity
from quanda.metrics.heuristics import ModelRandomizationMetric

2. Create the explainer object

We now create our explainer. The device to be used by the explainer and metrics is inherited from the model, thus we set the model device explicitly.

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model.to(DEVICE)

explainer_kwargs = {
    "layers": "fc_2",
    "model_id": "default_model_id",
    "cache_dir": cache_dir,
}
explainer = CaptumSimilarity(
    model=model, train_dataset=dataset, **explainer_kwargs
)

3. Initialize the metric

The ModelRandomizationMetric needs to instantiate a new explainer to generate explanations for a randomized model. These will be compared with the explanations of the original model. Therefore, explainer_cls is passed directly to the metric along with initialization parameters of the explainer for the randomized model.

explainer_kwargs = {
    "layers": "fc_2",
    "model_id": "randomized_model_id",
    "cache_dir": cache_dir,
}
ckpt_path = os.path.join(cache_dir, "model_rand_ckpt.pth")
torch.save(model.state_dict(), ckpt_path)
model_rand = ModelRandomizationMetric(
    model=model,
    model_id="randomized_model_id",
    cache_dir=cache_dir,
    train_dataset=dataset,
    checkpoints=ckpt_path,
    explainer_cls=CaptumSimilarity,
    expl_kwargs=explainer_kwargs,
    correlation_fn="spearman",
    seed=42,
)

4. Iterate over test set to generate explanations and update the metric

We now start producing explanations with our TDA method. We go through the test set batch-by-batch. For each batch, we first generate the attributions using the predicted labels, and we then update the metric with the produced explanations to showcase how to concurrently handle the explanation and evaluation processes.

test_loader = DataLoader(eval_set, batch_size=batch_size, shuffle=False)
for test_data, _ in tqdm(test_loader):
    test_data = test_data.to(DEVICE)
    target = model(test_data).argmax(dim=-1)
    tda = explainer.explain(test_data=test_data, targets=target)
    model_rand.update(
        explanations=tda, test_data=test_data, test_targets=target
    )

print("Randomization metric output:", model_rand.compute())

Using Benchmarks

Using Pre-assembled Benchmarks

The pre-assembled benchmarks allow us to streamline the evaluation process by downloading the necessary data and models, and running the evaluation in a single command. The following code demonstrates how to use the mnist_subclass_detection benchmark:

1. Import dependencies and library components

from quanda.explainers.wrappers import CaptumSimilarity
from quanda.benchmarks.downstream_eval import SubclassDetection

2. Prepare arguments for the explainer object

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model.to(DEVICE)

explainer_kwargs = {
    "layers": "fc_2",
    "model_id": "default_model_id",
    "cache_dir": cache_dir,
}

3. Load a pre-assembled benchmark and score an explainer

subclass_detect = SubclassDetection.load_pretrained(
    bench_id="mnist_subclass_detection",
    cache_dir=cache_dir,
)

score = subclass_detect.evaluate(
    explainer_cls=CaptumSimilarity,
    expl_kwargs=explainer_kwargs,
    batch_size=batch_size,
    max_eval_n=16,
)["score"]
print(f"Subclass Detection Score: {score}")

Loading a benchmark from a configuration file

Next, we demonstrate loading a benchmark from a YAML configuration file. As in the Using Metrics section, we will assume that the user has already trained model on train_set, and a corresponding eval_set to be used for generating and evaluating explanations.

1. Import dependencies and library components

import yaml

from quanda.explainers.wrappers import CaptumSimilarity
from quanda.benchmarks.heuristics import TopKCardinality

2. Prepare arguments for the explainer object

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model.to(DEVICE)

explainer_kwargs = {
    "layers": "fc_2",
    "model_id": "default_model_id",
    "cache_dir": cache_dir,
}

3. Load the benchmark from config and run the evaluation

We now have everything we need: we can load the benchmark from a YAML configuration file and run the evaluation. This will encapsulate the process of instantiating the explainer, generating explanations and using the TopKCardinalityMetric to evaluate them.

with open(
    "tests/assets/mnist_local_bench/83edb41-default_ClassDetection.yaml",
    "r",
) as f:
    top_k_config = yaml.safe_load(f)

topk_cardinality = TopKCardinality.from_config(
    top_k_config,
)
score = topk_cardinality.evaluate(
    explainer_cls=CaptumSimilarity,
    expl_kwargs=explainer_kwargs,
    batch_size=batch_size,
)["score"]
print(f"Top K Cardinality Score: {score}")

Training the benchmark from scratch

While we provide a number of benchmarks with pre-computed assets, quanda Benchmark objects also expose a train interface for preparing benchmarks from scratch. To train a benchmark, specify its components in a single YAML file (see quanda/benchmarks/resources/configs).

For example, the MislabelingDetection benchmark requires a dataset with known mislabeled examples. The train method takes care of model training on the mislabeled dataset, and prepares the benchmark for evaluation.

1. Import dependencies and library components

import yaml

from quanda.explainers.wrappers import CaptumSimilarity
from quanda.benchmarks.downstream_eval import MislabelingDetection

2. Prepare arguments for the explainer object

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model.to(DEVICE)

explainer_kwargs = {
    "layers": "fc_2",
    "model_id": "top_k_model",
    "cache_dir": cache_dir,
}

3. Train the benchmark

For mislabeling detection, we will train a model from scratch using a dataset with a portion of labels flipped.

with open(
    "tests/assets/mnist_local_bench/83edb41-default_MislabelingDetection.yaml",
    "r",
) as f:
    mislabel_config = yaml.safe_load(f)

mislabel_config["bench_save_dir"] = os.path.join(
    cache_dir, "mislabeling_detection_bench"
)
mislabeling_detection = MislabelingDetection.train(
    mislabel_config,
    device=DEVICE,
)

4. Run the evaluation

We can now call the evaluate method to directly start the evaluation process on the benchmark.

score = mislabeling_detection.evaluate(
    explainer_cls=CaptumSimilarity,
    expl_kwargs=explainer_kwargs,
    batch_size=batch_size,
    max_eval_n=16,
)["score"]
print(f"Mislabeling Detection Score: {score}")

More detailed examples can be found in the tutorials page. You can also use Hydra for benchmark training configuration, as shown in scripts/train.py.

Custom Explainers

In addition to the built-in explainers, quanda supports the evaluation of custom explainer methods. This section provides a guide on how to create a wrapper for a custom explainer that matches our interface.

Step 1. Create an explainer class

Your custom explainer should inherit from the base Explainer class provided by quanda. The first step is to initialize your custom explainer within the __init__ method.

from quanda.explainers.base import Explainer

class CustomExplainer(Explainer):
    def __init__(self, model, train_dataset, **kwargs):
        super().__init__(model, train_dataset, **kwargs)
        # Initialize your explainer here

Step 2. Implement the explain method

The core of your wrapper is the explain method. This function should take test samples and their corresponding target values as input and return a 2D tensor containing the influence scores.

  • test: The test batch for which explanations are generated.

  • targets: The target values for the explanations.

You must ensure that the output tensor has the shape (test_samples, train_samples), where the entries in the train samples dimension are ordered in the same order as in the train_dataset that is being attributed.

def explain(
  self,
  test_data: torch.Tensor,
  targets: Union[List[int], torch.Tensor]
) -> torch.Tensor:
    # Compute your influence scores here
    return influence_scores

Step 3. Implement the self_influence method (Optional)

By default, quanda includes a built-in method for calculating self-influence scores. This base implementation computes all attributions over the training dataset, and collects the diagonal values in the attribution matrix. However, you can override this method to provide a more efficient implementation. This method should calculate how much each training sample influences itself and return a tensor of the computed self-influence scores.

def self_influence(self, batch_size: int = 1) -> torch.Tensor:
    # Compute your self-influence scores here
    return self_influence_scores

For detailed examples, we refer to the existing explainer wrappers in quanda.

Usage Tips and Caveats

  • Controlled Setting Evaluation: Many metrics require access to ground truth labels for datasets, such as the indices of the “shortcut samples” in the Shortcut Detection metric, or the mislabeling (noisy) label indices for the Mislabeling Detection Metric. However, users often may not have access to these labels. To address this, we recommend either using one of our pre-built benchmark suites or generating (train method) a custom benchmark for comparing explainers. Benchmarks provide a controlled environment for systematic evaluation.

  • Explainer Caching: Many explainers in our library generate re-usable cache. The cache_dir and model_id parameters passed to various class instances are used to store these intermediary results. Ensure each experiment is assigned a unique combination of these arguments. Failing to do so could lead to incorrect reuse of cached results. If you wish to avoid re-using cached results, you can set the load_from_disk parameter to False.

  • Benchmark Dataset Caching: Benchmark initialization methods involve caching a HuggingFace dataset locally to the HF_HOME cache path. We recommend ensuring that the environment variable is set as needed and caching the dataset into the directory in advance of loading the benchmark.

  • Explainers Are Expensive To Calculate: Certain explainers, such as CaptumTracInCPFastRandProj, may lead to OutOfMemory (OOM) issues when applied to large models or datasets. In such cases, we recommend adjusting memory usage by either reducing the dataset size or using smaller models to avoid these issues.