Towards Budget-Friendly Model-Agnostic Explanation Generation for Large Language Models

Authors: Junhao Liu, Haonan Yu, Xin Zhang
Affiliation: Peking University

This repository contains the code for our paper:
"Towards Budget-Friendly Model-Agnostic Explanation Generation for Large Language Models".

We also provide the dataset used in our experiments, which can be downloaded from Zenodo:
🔗 https://zenodo.org/records/15582507

The dataset includes perturbed samples for LIME and Kernel SHAP on three tasks:

• SST-2
• Five subjects from MMLU
• A subset of Natural Questions (NQ)

These perturbed samples are generated can be used to evaluate model explanations.

To install dependencies:

conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.4 huggingface_hub transformers ipykernel accelerate -c pytorch -c nvidia -c conda-forge
pip install modelscope spacy
pip install flash-attn --no-build-isolation

You can reproduce our experiments for the three tasks using the scripts in the following folders:

• sst_exp/
• mmlu_exp/
• nq_exp/

If you are using our pre-generated dataset, you can skip this step.

Otherwise, place the official SST dataset in the SST2/ directory (mmlu/ and nq/ for the other tasks), and run:

python generate_samples.py
python generate_samples_shap.py

Then obtain the model outputs for the perturbed data:

python generate_model_sample.py --model_name <model_name> --model_short_name <model_short_name> --sample_folder <lime_sample/shap_sample>
python generate_api_sample.py --model_name <model_name> --model_short_name <model_short_name> --sample_folder <lime_sample/shap_sample>

Model names and their short forms can be found in model_list.txt.

Run the following to generate explanations:

python generate_exp.py
python generate_exp_shap.py

If you're using our dataset, please place it in the lime_samples/ and shap_samples/ directories.

Use the following commands for evaluation:

# Fidelity
python calc_fidelity.py --exp_method <lime/shap>

# AOPC
python calc_aopc.py --exp_method <lime/shap>

Results will be saved in the heatmaps/ folder.

📥 Download: https://zenodo.org/records/15582507

/
├── sst_samples/
│   ├── lime_samples/
│   │   ├── sst_test_samples.csv
│   │   ├── sst_test_samples_deepseekv3.csv
│   │   ├── sst_test_samples_gpt-4o.csv
│   │   ├── ... (other model-specific CSVs)
│   └── shap_samples/
│       ├── sst_test_samples.csv
│       ├── ... (other model-specific CSVs)
├── mmlu_samples/
│   ├── lime_samples/
│   │   ├── high_school_chemistry/
│   │   │   ├── high_school_chemistry_perturb.csv
│   │   │   ├── high_school_chemistry_perturb_deepseekv3.csv
│   │   │   ├── high_school_chemistry_perturb_gpt-4o.csv
│   │   │   ├── ... (other model-specific CSVs)
│   │   ├── high_school_computer_science/
│   │   │   ├── high_school_computer_science_perturb.csv
│   │   │   ├── ... (other model-specific CSVs)
│   │   ├── high_school_microeconomics/
│   │   │   ├── high_school_microeconomics_perturb.csv
│   │   │   ├── ... (other model-specific CSVs)
│   │   ├── high_school_psychology/
│   │   │   ├── high_school_psychology_perturb.csv
│   │   │   ├── ... (other model-specific CSVs)
│   │   ├── high_school_world_history/
│   │   │   ├── high_school_world_history_perturb.csv
│   │   │   ├── ... (other model-specific CSVs)
│   └── shap_samples/
│       ├── high_school_chemistry/
│       │   ├── high_school_chemistry_perturb.csv
│       │   ├── ... (other model-specific CSVs)
│       ├── ... (other subjects with similar structure)
├── nq_samples/
│   ├── lime_samples/
│   │   ├── nq_perturb.csv
│   │   ├── nq_perturb_deepseekv3.csv
│   │   ├── nq_perturb_gpt-4o.csv
│   │   ├── ... (other model-specific CSVs)
│   └── shap_samples/
│       ├── nq_perturb.csv
│       ├── ... (other model-specific CSVs)

For those in SST-2, each CSV file contains perturbed samples for a specific model, with the following columns:

• sentence_index: The index of the original sentence in the dataset
• sentence: The original sentence text
• binary_representation: A binary vector indicating which words are perturbed (1 = present, 0 = masked)
• sample_sentence: The perturbed sentence generated according to the binary representation
• logits_positive: The model's output logit for the positive class
• logits_negative: The model's output logit for the negative class

For MMLU， each CSV file contains perturbed samples for a specific subject and model, with the following columns:

• question_index: The index of the original question in the dataset
• question: The original question text
• binary_representation: A binary vector indicating which words in the question are perturbed (1 = present, 0 = masked)
• sample_question: The perturbed question generated according to the binary representation
• logits_A, logits_B, logits_C, logits_D: The model's output logits for each answer option (A, B, C, D)
• A, B, C, D: The text of each answer option
• input: The full input text provided to the model for this sample (with few shot examples included)

The few shot examples are included in the dev folder of the raw MMLU dataset.

For Natural Questions, each CSV file contains perturbed samples for a specific model, with the following columns:

• question_index: The index of the original question in the dataset
• question: The original question text
• binary_representation: A binary vector indicating which words in the question are perturbed (1 = present, 0 = masked)
• sample_question: The perturbed question generated according to the binary representation
• Answer: The model's predicted answer for the perturbed question
• scores: The model's output scores or probabilities for each possible answer

This code is released under the CC BY 4.0 License.

If you use our code or dataset, please cite:

@misc{liu2025budgetfriendlymodelagnosticexplanationgeneration,
  title={Towards Budget-Friendly Model-Agnostic Explanation Generation for Large Language Models}, 
  author={Junhao Liu and Haonan Yu and Xin Zhang},
  year={2025},
  eprint={2505.12509},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2505.12509}
}