Skip to content

PMLB

The Penn Machine Learning Benchmarks (PMLB) is a curated collection of benchmark datasets for evaluating machine learning methods. HGP is evaluated on the binary classification datasets in the suite.

Data preparation

The preprocessing script lives at scripts/preprocess/pmlb_preprocess.py. It fetches a named dataset with the pmlb package and writes it to the data folder as an HDF file.

python scripts/preprocess/pmlb_preprocess.py breast_cancer --data_path data

This writes data/breast_cancer.hdf.

The list of binary classification datasets comes from the PMLB summary table. The same query is used by the batch runner to decide which datasets to process.

from pmlb.dataset_lists import df_summary

binary_datasets = df_summary[df_summary["n_classes"] == 2.0]["dataset"].tolist()

Warning

Some datasets listed in older PMLB summaries are no longer available, because they were removed as duplicates. Fetching such a dataset raises an error. Skip the missing datasets and continue with the rest.

Running a benchmark

Benchmark the Boolean GP on a single preprocessed dataset with scripts/run_benchmark.py.

python scripts/run_benchmark.py --data_path data/breast_cancer.hdf

See python scripts/run_benchmark.py --help for the full list of options.

Hyperparameter tuning

Tune a single dataset with the Optuna script. The study name keeps trials grouped, and artifacts are written to the artifact directory.

python scripts/optuna_hypertuning.py \
    --data_path data/breast_cancer.hdf \
    --study_name pmlb_breast_cancer \
    --n_trials 25 \
    --max_n_trials 25 \
    --n_runs 5 \
    --n_folds 3 \
    --artifact_dir ./artifacts

Running on all datasets

The scripts/run_pmlb_comparison.py script compares HGP with interpretable baselines on every binary classification dataset.

python scripts/run_pmlb_comparison.py --n_jobs 5

The script runs each method through scripts/run_on_pmlb.py. Missing datasets are downloaded from PMLB first.

Each method writes one CSV to results/pmlb. For every dataset, it holds the mean and standard deviation of the test F1, the mean size and number of literals of the rule learned in each fold, and the runtime. When all methods are done, the script prints the comparison tables, saves them to results/pmlb/summary.md, and saves radar plots of the per-dataset scores in the same folder.

BoolXAI pins numpy 1.26.4 and scikit-learn 1.5.2, so it needs to run in its own Python 3.10 or 3.11 environment. The script looks for it in .venv/boolxai; pass another interpreter with --boolxai_python.

python3.11 -m venv .venv/boolxai
.venv/boolxai/bin/pip install "git+https://github.com/fidelity/boolxai.git" "pandas<3" tables pmlb tqdm
.venv/boolxai/bin/pip install --no-deps -e .

To tune HGP on every dataset with Optuna instead, run scripts/run_on_pmlb.py with its default model. It builds one optuna_hypertuning.py command per dataset and runs --n_jobs of them at a time.

python scripts/run_on_pmlb.py --n_jobs 5

Results on PMLB

Each method is scored with the F1 score on a held-out test set, averaged over 5 runs. Each run makes a stratified 80/20 train/test split and runs 3-fold cross-validation on the training part. The fold model with the best validation F1 is scored on the test set.

The HGP variants keep the library defaults, except for the number of epochs:

  • HGP (1000 epochs): the library defaults.
  • HGP (10000 epochs): the library defaults, trained ten times longer.

The baselines run with their default parameters:

  • DT: the scikit-learn decision tree.1
  • BoolXAI2: refines one expressive Boolean rule by simulated annealing.45
  • RIPPER: a greedy rule-set learner6, from the wittgenstein package.3

The tables report, for each method:

  • Mean rank: the rank on each dataset, averaged over datasets. Rank 1 is the best score, and tied methods share their ranks.
  • Best or tied: the number of datasets on which the method has the best score or ties for it.
  • Mean F1: the test F1, averaged over datasets.
  • Mean literals: the number of literals in the rule learned in each fold, averaged over folds, runs, and datasets.
  • Mean nodes: the number of nodes in the same rule, literals and operators together.
  • Mean runtime (s): the time to run all runs and folds on one dataset, averaged over datasets.
  • HGP wins / ties / losses: the number of datasets on which the HGP variant scores higher than the method, ties with it, or scores lower.

The baseline models are counted as hgp-lib rules. A decision tree becomes the disjunction of its paths to positive leaves, and a RIPPER rule set the disjunction of its rules.

All methods

Every HGP variant ranks ahead of the baselines. The decision tree grows without a depth limit, so its rules are by far the largest: 1173.7 nodes on average, and 92.8 on the median dataset.

Method Mean rank Best or tied Mean F1 Mean literals Mean nodes Mean runtime (s)
HGP (10000 epochs) 2.38 23 0.7356 12.0 16.5 1404.9
HGP (1000 epochs) 2.60 17 0.7369 10.7 14.8 106.2
DT 2.64 24 0.7337 1097.4 1173.7 0.2
BoolXAI 3.54 11 0.6847 3.1 4.3 252.1
RIPPER 3.84 10 0.6441 21.2 29.0 17.4

HGP against the baselines

Method Mean rank Best or tied Mean F1 Mean literals Mean nodes Mean runtime (s) HGP wins / ties / losses
HGP (1000 epochs) 1.99 30 0.7369 10.7 14.8 106.2 -
DT 2.11 27 0.7337 1097.4 1173.7 0.2 35 / 5 / 30
BoolXAI 2.80 12 0.6847 3.1 4.3 252.1 54 / 0 / 16
RIPPER 3.10 10 0.6441 21.2 29.0 17.4 48 / 2 / 20

HGP with the library defaults vs DT, BoolXAI, and RIPPER

The same comparison with the HGP variant that trains 10 times as long:

Method Mean rank Best or tied Mean F1 Mean literals Mean nodes Mean runtime (s) HGP wins / ties / losses
HGP (10000 epochs) 1.99 31 0.7356 12.0 16.5 1404.9 -
DT 2.11 27 0.7337 1097.4 1173.7 0.2 35 / 5 / 30
BoolXAI 2.77 12 0.6847 3.1 4.3 252.1 52 / 0 / 18
RIPPER 3.14 10 0.6441 21.2 29.0 17.4 50 / 3 / 17

HGP with 10000 epochs vs DT, BoolXAI, and RIPPER

HGP versions

Method Mean rank Best or tied Mean F1 Mean literals Mean nodes Mean runtime (s)
HGP (10000 epochs) 1.39 45 0.7356 12.0 16.5 1404.9
HGP (1000 epochs) 1.61 30 0.7369 10.7 14.8 106.2

HGP with 1000 and 10000 epochs

References

  1. scikit-learn DecisionTreeClassifier, https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html. ↩

  2. BoolXAI, https://github.com/fidelity/boolxai. ↩

  3. wittgenstein, https://github.com/imoscovitz/wittgenstein. ↩

  4. Gili Rosenberg, John Kyle Brubaker, Martin J. A. Schuetz, Grant Salton, Zhihuai Zhu, Elton Yechao Zhu, Serdar Kadıoğlu, Sima E. Borujeni, and Helmut G. Katzgraber. Explainable artificial intelligence using expressive boolean formulas. Machine Learning and Knowledge Extraction, 5(4):1760–1795, 2023. doi:10.3390/make5040086. ↩

  5. Serdar Kadioglu, Elton Yechao Zhu, Gili Rosenberg, John Kyle Brubaker, Martin J. A. Schuetz, Grant Salton, Zhihuai Zhu, and Helmut G. Katzgraber. Boolxai: explainable ai using expressive boolean formulas. Proceedings of the AAAI Conference on Artificial Intelligence, 39(28):28900–28906, Apr. 2025. doi:10.1609/aaai.v39i28.35157. ↩

  6. William W. Cohen. Fast Effective Rule Induction. In Machine Learning Proceedings 1995, 115–123. Morgan Kaufmann, 1995. doi:10.1016/B978-1-55860-377-6.50023-2. ↩