Candidate Set Query (CSQ) for Active Learning

This is the official implementation for "Enhancing Cost Efficiency in Active Learning with Candidate Set Query".

Enhancing Cost Efficiency in Active Learning with Candidate Set Query
Yeho Gwon^*, Sehyun Hwang^*, Hoyoung Kim, Jungseul Ok, and Suha Kwak (* indicates equal contribution)
POSTECH
Accepted to and published in TMLR
Arxiv
Project Page

This paper introduces a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query. Unlike traditional AL queries requiring the oracle to examine all possible classes, our method narrows down the set of candidate classes likely to include the ground-truth class, significantly reducing the search space and labeling cost. Moreover, we leverage conformal prediction to dynamically generate small yet reliable candidate sets, adapting to model enhancement over successive AL rounds. To this end, we introduce an acquisition function designed to prioritize data points that offer high information gain at lower cost. Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet64x64 demonstrate the effectiveness and scalability of our framework. Notably, it reduces labeling cost by 48% on ImageNet64x64.

The implementation has been validated using Python 3.10.18, PyTorch 2.7.1, and torchvision 0.22.1 with CUDA 12.9. Due to CUDA version dependencies, manual installation of PyTorch and torchvision may be required based on your system configuration and hardware compatibility.

To install requirements (may not work due to CUDA compatability):

$ pip install "torch>=2.4.1" torchvision
$ pip install -U sentence-transformers # only for NLP tasks
$ pip install "wandb==0.17.0" "numpy<2"
$ pip install pandas scipy scikit-learn pillow tqdm
$ pip install datasets # only for R52

Also, our implementation does support multi-GPU training with PyTorch DDP (Distributed Data Parallel). DDP will be automatically enabled if you have multiple GPUs.

Note You can specify which GPU to use with CUDA_VISIBLE_DEVICES environment variable to disable DDP. If you want to use a single GPU (i.e., disable DDP), you can set CUDA_VISIBLE_DEVICES to a single one.

We have evaluated our method on the following datasets, although the code supports more datasets.

• CIFAR-10
• CIFAR-100
• ImageNet64x64 (*Downsampled ImageNet-1k)
• CIFAR-100N (CIFAR-100 with noisy labels)
• CIFAR-100LT (CIFAR-100 with class imbalances)

The directory structure needs to be as follows:

.
├── dataset/
│   ├── CIFAR-10/cifar-10-batches-py/...
│   ├── CIFAR-100/cifar-100-python/...
│   ├── ImageNet64x64/
│   │   ├── train/
│   │   ├── val/
│   │   └── proc.py
├── ...

Note This implementation requires nothing specific for CIFAR-100N and CIFAR-100LT, with which you need to prepare your own datasets. You are required to pass the path to the datasets manually.

For CIFAR-10 and CIFAR-100, you can download them with --download flag.

Note ImageNet32x32 can also be constructed in the same way. The following instruction is adopted from https://patrykchrabaszcz.github.io/Imagenet32/.

Download the downsampled ImageNet pickled files from the official ImageNet website. It is highly recommended that you have your own account. You can download the files by clicking on the links with the following names:

• Train(64x64) part 1, npz format, 6 GB
• Train(64x64) part 2, npz format, 6 GB
• Val(64x64), npz format, 509 MB

❗ Important You shouldn't click on npz format links. They download the files in a different format.

Then you will have the following files:

• Imagenet64_train_part1.zip
• Imagenet64_train_part2.zip
• Imagenet64_val.zip

The next step is to preprocess the data. Under the root directory, run the following command:

$ cd dataset/ImageNet64x64
$ mkdir train val

After moving the files to approprirate directories (train files to train directory and validation file to val directory), run the following command to preprocess the data:

$ cd train
$ unzip Imagenet64_train_part1.zip
$ unzip Imagenet64_train_part2.zip
$ rm Imagenet64_train_part1.zip Imagenet64_train_part2.zip
$ cd ../val
$ unzip Imagenet64_val.zip
$ rm Imagenet64_val.zip
$ cd ..
$ python proc.py

Then you are all set. As a side note, running python proc.py will remove the extracted batch files.

You can download the Products-10K dataset from kaggle. Extract it and build the following directory structure:

.
├── dataset/
│   ├── Products10K/
│   │   ├── train/*.jpg
│   │   ├── test/*.jpg
│   │   ├── train.csv
│   │   └── test.csv
├── ...

This assumes that you have CIFAR-100 downloaded and a numpy array of noisy labels.

If you have not downloaded CIFAR-100, you can download it using --download flag. Also, you should prepare a numpy array of noisy labels with a length of 50000. It can be passed to --noisy_label_path argument.

Note This dataset just replaces the labels of CIFAR-100 with the provided noisy labels.

This assumes that you have CIFAR-100 downloaded and a numpy array of imbalanced indices.

If you have not downloaded CIFAR-100, you can download it using --download flag. Also, you should prepare a numpy array of imbalanced indices with a length of 50000, with which is equivalent to CIFAR-100[indices] (i.e., CIFAR-100 with only the indices). It can be passed to --indices_path argument.

Note This dataset works exactly the same way as torch.utils.data.Subset. So, the imbalance ratio is highly dependent on the provided indices.

We support a variety of AL acquisition functions. The following are available for you to use:

• Naive Sampling: CQ in the paper
  • Random Sampling
  • Entropy Sampling
  • BADGE Sampling (Ash et al. ICLR 2020)
  • Coreset Sampling (Sener & Savarese ICLR 2018)
  • ProbCover Sampling (Yehuda et al. NeurIPS 2022)
  • SAAL Sampling (Kim et al. ICML 2023)

Note These acquisition functions can be used either solely or in combination with CSQ.

ProbCover requires self-supervised features, and it is not scalable to large datasets with large budgets. Thus, we provide experiments with the features only for CIFAR-10 and CIFAR-100.

We adopt the features from self-supervised feature representation SimCLR for CIFAR-10 and CIFAR-100. You can download the features arrays from here: CIFAR-10 / CIFAR-100, which is provided in this GitHub repository.

✅ FYI You can find example commands for our experiments in script.sh, which are used to run experiments with different datasets, strategies, and hyperparameters.

The training and evaluation are done by running run.py. The following arguments are available:

• This code is built on top of Ash's BADGE implementation. Thanks for the great implementation!
• The implementation of ProbCover is based on, but not fully adopted from, Dekel's TypiClust implementation. We appreciate the great work.