Benchmark
The results database for benchmark experiments: SQLite by default (via the SQLiteDb base), holding the tables described by the table-data classes of this package.
Append semantics: by default the database is NOT deleted if it exists, and only missing tables are created — so successive experiments (including a later trace-enabled rerun of a problem) accumulate in one file under fresh experiment and run ids. Set deleteIfExists true to start clean.
Capture is post-run and bulk: run the ksl.simopt.benchmark.BenchmarkExperiment, then hand its summary to saveSummary. The returned experiment id keys every table.
Post-processing helpers pull typed rows back out (one function per table), feed a MultipleComparisonAnalyzer with per-problem final objectives via mcbDataMap, and compute performance-profile data from captured traces via performanceProfile.
Parameters
the database file name
the directory holding the database file
true deletes an existing file first; the default (false) appends to it
Constructors
Functions
Confirmation rows, optionally restricted to one experiment.
All experiment rows.
A MultipleComparisonAnalyzer over one problem's final objectives (see mcbDataMap); null when the data cannot support a multiple comparison — fewer than two solver cases with complete data, unequal numbers of observations across the cases, or fewer than two observations per case.
A MultipleComparisonAnalyzer over one problem's final objectives pooled across the blocks of one study (see the pooled mcbDataMap); null when the data cannot support a multiple comparison — fewer than two solver cases with complete data, unequal numbers of observations across the cases, or fewer than two observations per case.
The multiple-comparison feed for one problem of one experiment: solver case label mapped to the final objective values (or gaps) across the completed, valid macro-replications, ordered by replication number.
The multiple-comparison feed for one problem pooled across several experiments — the blocks of one study, run as separate experiments so that a long run can be checkpointed and resumed. Solver case label mapped to the final objective values (or gaps) across the completed, valid macro-replications of every supplied experiment, ordered by replication number.
Performance-profile data across an experiment's traced runs: for each solver case and each budget fraction on a grid, the fraction of (problem, macro-rep) cells whose best penalized objective reached the problem's gap basis plus tau within that fraction of the replication budget.
Performance-profile data pooled across the blocks of one study (see the pooled mcbDataMap). The blocks must share a replication budget, since the profile's x-axis is the fraction of that budget; pooling experiments with different budgets would put incomparable cells on one axis.
Problem rows, optionally restricted to one experiment.
Run rows, optionally restricted to one experiment.
Persists a benchmark summary: the experiment row, its problems, solver cases and their captured parameters, every cell run, confirmation finalists, verification estimates, and (when captured) iteration traces keyed by run id.
Solver case parameter rows, optionally restricted to one experiment.
Solver case rows, optionally restricted to one experiment.
Iteration-trace rows, optionally restricted to one experiment's runs.
Verification rows, optionally restricted to one experiment.