KSLCore/ksl.simopt.benchmark.io

Package-level declarations

Types

class BenchmarkResultsDb @JvmOverloads constructor(dbName: String, dbDirectory: Path = KSL.dbDir, deleteIfExists: Boolean = false) : SQLiteDb, BenchmarkResultSink

The results database for benchmark experiments: SQLite by default (via the SQLiteDb base), holding the tables described by the table-data classes of this package.

ConfirmationReplay

object ConfirmationReplay

Re-runs the confirmation stage of a finished benchmark against stored results, under a different selection rule, without re-running the search.

ConfirmationSummaryTableData

data class ConfirmationSummaryTableData(var expId: Int = -1, var problemName: String = "", var numCandidates: Int = 0, var numConfidentlyFeasible: Int = 0, var selectionDegenerate: Boolean = false, var numConfidentlyFeasibleAfterScreening: Int? = null, var numOracleCalls: Int = 0, var numReplicationsRequested: Int = 0) : DbTableData

One row per (experiment, problem) confirmation stage: how many candidates were ranked, how many of them were confidently response-feasible at the confirmation's CI level, and whether the selection was therefore degenerate.

ConfirmationTableData

data class ConfirmationTableData(var expId: Int = -1, var problemName: String = "", var candidateNum: Int = 0, var inputsJson: String = "", var objective: Double = 0.0, var penalizedObjective: Double = 0.0, var numReplications: Double = 0.0, var isWinner: Boolean = false) : DbTableData

One row per confirmed finalist of a problem's confirmation stage: the finalist's point, its CRN-confirmed estimates, and whether it won.

ExperimentTableData

data class ExperimentTableData(var expId: Int = -1, var expName: String = "", var startTime: String = "", var endTime: String = "", var replicationBudgetPerRun: Int = 0, var macroReplications: Int = 0, var numProblems: Int = 0, var numSolverCases: Int = 0, var confirmationTopK: Int? = null, var confirmationReplications: Int? = null, var verificationReplications: Int? = null, var tracesCaptured: Boolean = false, var solverStateCaptured: Boolean = false, var kslVersion: String? = null) : DbTableData

One row per benchmark experiment. Experiment ids are assigned by the database (max + 1), so re-running an experiment — for example a trace-enabled rerun of a problem — appends into the same database under a new id.

IterationTraceStateTableData

data class IterationTraceStateTableData(var runId: Int = -1, var iteration: Int = 0, var stateName: String = "", var stateValue: Double = 0.0) : DbTableData

One row per (run, iteration, state name) of a captured trace (opt-in): the cell solver's algorithm-specific state for that iteration.

IterationTraceTableData

data class IterationTraceTableData(var runId: Int = -1, var iteration: Int = 0, var cumulativeReplications: Int = 0, var bestPenalizedObjective: Double = 0.0) : DbTableData

One row per captured iteration of a run's trace (opt-in): the cumulative requested replications and the best penalized objective so far. Iteration 0 is the initialized state. Keyed by run id, so a trace-enabled rerun lands in the same database next to earlier trace-free experiments.

PerformanceProfilePoint

data class PerformanceProfilePoint(val solverLabel: String, val budgetFraction: Double, val fractionSolved: Double)

One point of performance-profile data: the fraction of a solver case's (problem, macro-replication) cells that reached the solve threshold within the given fraction of the replication budget.

ProblemConstraintTableData

data class ProblemConstraintTableData(var expId: Int = -1, var problemName: String = "", var responseName: String = "", var rhsValue: Double = 0.0, var inequalityType: String = "", var target: Double = 0.0, var tolerance: Double = 0.0) : DbTableData

One row per (experiment, problem, response constraint): the constraint as the problem defines it. Without these rows the database records what a run achieved but not what it had to meet, so answering "was every verified winner feasible?" meant hand-coding the constraint table from the specification and joining it externally. With them the question is a single join and the archive is self-describing.

ProblemReplayResult

data class ProblemReplayResult(val problemName: String, val candidates: List<Solution>, val outcome: ConfirmationOutcome, val originalWinnerInputs: Map<String, Double>?, val winnerChanged: Boolean)

One problem's replayed selection.

ProblemTableData

data class ProblemTableData(var expId: Int = -1, var problemName: String = "", var dimension: Int = 0, var optimizationType: String = "", var numResponseConstraints: Int = 0, var tagsJson: String? = null, var referenceType: String? = null, var referenceObjective: Double? = null, var referenceInputsJson: String? = null, var gapType: String? = null, var gapBasisObjective: Double? = null, var winnerInputsJson: String? = null, var winnerObjective: Double? = null) : DbTableData

One row per problem per experiment: identity, shape (dimension, orientation, constraints), the reference solution when one exists, the gap basis the runs were measured against, and the problem's winning point.

RunConstraintTableData

data class RunConstraintTableData(var runId: Int = -1, var responseName: String = "", var estimate: Double = 0.0, var violation: Double = 0.0, var ciUpperLimit: Double? = null, var feasibleAtCI: Boolean = false) : DbTableData

One row per (run, response constraint): the cell best's estimate for the constrained response, by how much the constraint is violated, the one-sided upper confidence limit, and whether that constraint on its own can be declared feasible.

RunResponseTableData

data class RunResponseTableData(var runId: Int = -1, var responseName: String = "", var average: Double = 0.0, var variance: Double = 0.0, var count: Double = 0.0) : DbTableData

One row per (run, response): the cell best's estimate for every response of the problem, the objective included.

RunTableData

data class RunTableData(var runId: Int = -1, var expId: Int = -1, var problemName: String = "", var solverLabel: String = "", var repNum: Int = 0, var cellLabel: String = "", var status: String = "", var startingPointJson: String = "", var bestInputsJson: String = "", var bestObjective: Double = 0.0, var bestPenalizedObjective: Double = 0.0, var bestValid: Boolean = false, var inputFeasible: Boolean = false, var responseConstraintViolation: Double = 0.0, var numOracleCalls: Int = 0, var numReplicationsRequested: Int = 0, var totalIterations: Int? = null, var wallClockMillis: Long? = null, var cpuTimeMillis: Long? = null, var gap: Double? = null, var gapType: String? = null, var errorMessage: String? = null) : DbTableData

One row per benchmark cell: one solver case run once (one macro-replication) on one problem under the experiment's budget. numReplicationsRequested is the actual budget consumption used to normalize cross-algorithm comparisons.

SolverCaseParameterTableData

data class SolverCaseParameterTableData(var expId: Int = -1, var solverLabel: String = "", var paramName: String = "", var paramValue: String = "") : DbTableData

The flattened configuration properties of a solver case, read from the first solver instance the case actually created during the experiment — the configuration that ran, not what was assumed.

SolverCaseTableData

data class SolverCaseTableData(var expId: Int = -1, var solverLabel: String = "", var description: String = "") : DbTableData

One row per solver case per experiment; the case's parameters live in tblSolverCaseParameter.

VerificationTableData

data class VerificationTableData(var expId: Int = -1, var problemName: String = "", var responseName: String = "", var inputsJson: String = "", var average: Double = 0.0, var variance: Double = 0.0, var count: Double = 0.0) : DbTableData

One row per response of a problem's verification stage (opt-in): the winning point re-simulated at the experiment's verification replication count.