Benchmark
The benchmark grid: problems × solver configurations × macro-replications, run under the experiment-level fairness policies and concurrent execution, producing a BenchmarkSummary of cell-level run records, per-problem confirmation outcomes, and optimality gaps.
Fairness and reproducibility policies:
Equal replication budgets. Every cell's solver is wrapped to receive a replication-budget stopping criterion for replicationBudgetPerRun and an iteration ceiling equal to the budget (every iteration requests at least one replication, so the budget itself is a generous ceiling). The criterion is checked between iterations, so batch solvers can overshoot by up to one generation; the run record captures the actual consumption for normalization. If a case's factory installed its own solution-quality criterion, the wrap composes them (either stops the solver); the solver-internal convergence fallbacks that live behind the criterion slot are superseded, which is the intended equal-effort semantics.
Common starting points. For each (problem, macro-replication), one starting point is pre-drawn on the launching thread from the experiment's stream and shared by every solver case — solvers race from common starts and macro-reps vary the start. Each cell receives its own copy. Population-based solvers that ignore starting points simply ignore them.
Confirmation (optional). After all of a problem's cells complete, the best solutions are re-evaluated under common random numbers via a dedicated evaluator (provisioned as an extra member, so it has its own stream block) and the winner is picked from the confirmed estimates.
Determinism. Cell identity, starting points, member stream blocks, and solver-side streams are all fixed at launch time, and results are consumed in cell order — for a fixed experiment configuration the summary does not depend on the worker count or thread timing (the concurrent substrate's guarantee, inherited).
Gap recording: when a problem has a ReferenceSolution, every run's gap is computed against it (gap type from the reference); otherwise runs are gapped against the best objective found across the problem's valid runs (gap type BEST_FOUND). Gaps are oriented so that larger is worse regardless of the problem's optimization type.
Problems run sequentially; the cells of each problem run concurrently on at most numWorkers workers.
Parameters
the experiment's name; flows into every run record
the problem cases; names must be unique
the solver configurations; labels must be unique
the number of macro-replications per (problem, solver) pair
the per-cell replication budget
confirmation-stage options; null disables confirmation
when true, every cell solver's per-iteration progress (iteration, cumulative replications, best penalized objective) is captured into the summary's traces, keyed by cell label — opt-in because traces grow with the budget
when non-null, each problem's winning point is re-simulated at this replication count on a dedicated evaluator and recorded — the classic verify-at-elevated-replications step
the maximum number of cells running at the same time; null uses the smaller of the cell count and the available processors
the stream provider for experiment-level draws (currently: the common starting points); defaults to a fresh provider so identically configured experiments reproduce each other exactly
invoked with each freshly created cell solver before it runs, on the cell's worker thread — the attachment hook for per-cell trackers and instrumentation; anything it touches must be safe to use from worker threads