Experimental Design Methods

Learning Objectives

• Part 1 exposed model inputs as controls and ran a handful of named scenarios (Scenario, ScenarioRunner, ConcurrentScenarioRunner) compared with MCB. Now we turn to formal experimental design.

• To be able to use the terminology of experimental design and explain why designed experiments beat one-factor-at-a-time (OFAT) experimentation

• To be able to define factors, factorial and fractional designs, and run them with the KSL DesignedExperiment pipeline

• To be able to fit regression meta-models and use central composite designs to estimate a response surface

What Is a Designed Experiment?

• Running a simulation model is experimentation on a computer. An experimental design is the plan for which inputs to set and what outputs to observe, in the most efficient and effective way.

• Core terminology:

  • Factor — a model input/parameter you deliberately vary (e.g. reorder quantity).
  • Levels — the settings a factor takes (e.g. 5, 20, 50 units).
  • Design point — one specific combination of factor settings.
  • Replication — one execution (run) of a design point; one observation of the response.
  • Design — the set of design points plus how many replications each requires.

• Experimental design is iterative — expect some preliminary ad-hoc experimentation before the plan settles.

The Cost of Many Factors

• With \(k\) factors, factor \(i\) having \(q_i\) levels, and \(r\) replications, total runs are:

\[ r \times \prod_{i=1}^{k} q_i \]

• If every factor has just two levels, this is \(r \times 2^{k}\).
  • Example: \(r = 20\), \(k = 10 \Rightarrow 20 \times 2^{10} = 20{,}480\) runs.
• The design is the lever: we want the most information from the fewest runs. This motivates two-level designs, fractional designs, and screening.

Defining Factors and Coding

• The KSL Factor class holds a name and its levels, supplied in uncoded (natural) units, and translates to/from coded space.
• Coded value \(x = (w - m)/h\), with half-range \(h = (u-l)/2\) and midpoint \(m = (u+l)/2\). A two-level factor always codes to low \(=-1\), high \(=+1\).

// levels are supplied in natural (uncoded) units
val f1 = Factor("A", doubleArrayOf(1.0, 2.0, 3.0, 4.0))
val f2 = Factor("B", doubleArrayOf(5.0, 9.0))
val factors = setOf(f1, f2)

Factor: A
Levels = 1.0,2.0,3.0,4.0
halfRange = 1.5
midPoint = 2.5
Coded Levels = -1.0,-0.333,0.333,1.0

The Factor Class

Defining Designs: FactorialDesign

• A factorial design is every combination of the factors’ levels. KSL designs implement ExperimentalDesignIfc, which lets the design points be listed and iterated.

val fd = FactorialDesign(factors)            // A: 4 levels, B: 2 levels -> 4 x 2 = 8 points
println(fd.designPointsAsDataframe())         // uncoded design points
println(fd.designPointsAsDataframe(true))     // coded design points (-1..+1)

     A   B
 0 1.0 5.0
 1 1.0 9.0
 2 2.0 5.0
 ...
 6 4.0 5.0
 7 4.0 9.0

The ExperimentalDesignIfc Interface

Two-Level and Fractional Factorial Designs

• Use TwoLevelFactor (only a low and high) with TwoLevelFactorialDesign for a full \(2^{k}\) design. A fractional design runs a \(2^{k-p}\) subset via an iterator.

val design = TwoLevelFactorialDesign(
    setOf(
        TwoLevelFactor("A", 5.0, 15.0),
        TwoLevelFactor("B", 2.0, 11.0),
        TwoLevelFactor("C", 6.0, 10.0),
        TwoLevelFactor("D", 3.0, 9.0),
        TwoLevelFactor("E", 4.0, 16.0)
    )
)
// positive half-fraction: 16 of the 32 design points
val hitr = design.halfFractionIterator()
hitr.asSequence().toList().toDataFrame(coded = true).print(rowsLimit = 36)

• For smaller fractions, design.fractionalIterator(relation) takes a defining relation, e.g. setOf(setOf(1,2,4), setOf(1,3,5), setOf(2,3,4,5)) gives a resolution III \(2^{5-2}\) design (8 points).
• Fewer runs means confounding: effects become aliased. Higher resolution means less confounding among low-order effects.

Executing a Designed Experiment

• DesignedExperiment simulates each design point for its replications, reusing one Model. The factorSettings map ties each Factor to a control key or RV-parameter key (from Part 1).

val fA = Factor("Server", doubleArrayOf(1.0, 2.0, 3.0))
val fB = Factor("MeanST", doubleArrayOf(0.6, 0.7))
val fC = Factor("MeanTBA", doubleArrayOf(1.0, 5.0))
val factors = mapOf(                              // factor -> control / RV-parameter key
    fA to "MM1Q.numServers",
    fB to "MM1_Test:ServiceTime.mean",
    fC to "MM1_Test:TBA.mean"
)
val fd = FactorialDesign(factors.keys)
val de = DesignedExperiment("FactorDesignTest", m, factors, fd)
de.simulateAll(numRepsPerDesignPoint = 3)         // simulate every design point
de.resultsToCSV()                                 // also captured to a KSLDatabase

• Results are available as data frames (replicatedDesignPointsWithResponses(), coded or uncoded) and export to CSV.

The DesignedExperiment Class

Running Design Points in Parallel

• ParallelDesignedExperiment is a drop-in counterpart that runs every design point concurrently. The only change: it takes a ModelBuilderIfc instead of a built Model, so each point gets its own private Model.

object BuildMM1Q : ModelBuilderIfc {
    override fun build(/* config, runParams */): Model {
        val m = Model("DesignedExperimentDemo")
        GIGcQueue(m, 1, name = "MM1Q")
        return m
    }
}

val de = ParallelDesignedExperiment("FactorDesignTest", BuildMM1Q, factors, fd)
de.simulateAll(numRepsPerDesignPoint = 3)          // design points run in parallel

• Default is independent random streams per point (the opposite of ConcurrentScenarioRunner’s CRN default) — the right choice for OLS regression, which assumes independent observations. Call useCommonRandomNumbers() to override.
• Everything after construction (simulateAll(), resultsToCSV(), data-frame and regression methods) is identical to DesignedExperiment.

Analyzing a Designed Experiment

• LinearModel is a string specification of the regression model in factor names (e.g. "A B C A*B A*C B*C A*B*C"; build with specifyAllTerms() or design.linearModel(type = ...)). regressionResults() fits it via OLS.

val design = TwoLevelFactorialDesign(setOf(r, q))  // r, q are TwoLevelFactors
val settings = mapOf(
    r to "RQInventory:Item.initialReorderPoint",
    q to "RQInventory:Item.initialReorderQty"
)
val de = DesignedExperiment("R-Q Inventory Experiment", m, settings, design)
de.simulateAll(numRepsPerDesignPoint = 20)

val lm = design.linearModel(type = LinearModel.Type.AllTerms)   // main effects + interaction
val regressionResults = de.regressionResults("RQInventory:Item:TotalCost", lm)
regressionResults.showResultsInBrowser()

R-Squared = 0.9213
                Predictor  parameter  t0-ratio  2*P(T>|t0|)
                Intercept   1.489621  152.7924     0.000000
             ReorderLevel   0.235961   24.2028     0.000000
               ReorderQty  -0.032746   -3.3588     0.001226
  ReorderLevel*ReorderQty   0.166625   17.0909     0.000000

The LinearModel Class

CCD Geometry: Corners + Axial + Center

Fitting a Response Surface

• Simulate the CCD with DesignedExperiment, then fit two models and compare. .quadratic(name) must be added explicitly for each factor — nothing adds squared terms automatically.

val de = DesignedExperiment("Queue Response Surface", m, settings, ccd)
de.simulateAll()                 // NO argument: keep each point's own rep count

val firstOrderModel  = ccd.linearModel(type = LinearModel.Type.FirstAndSecond)
val firstOrderResults = de.regressionResults("System Time", firstOrderModel)

val secondOrderModel = ccd.linearModel(type = LinearModel.Type.FirstAndSecond)
    .quadratic("MeanST")         // add curvature terms explicitly
    .quadratic("MeanTBA")
val secondOrderResults = de.regressionResults("System Time", secondOrderModel)

println("First-order  R-squared: ${firstOrderResults.rSquared}")
println("Second-order R-squared: ${secondOrderResults.rSquared}")

• A meaningfully larger second-order \(R^2\) (with significant quadratic terms) means the response really curves. simulateAll() with no argument preserves the extra center-point replications. Never trust predictions outside the studied region.

Scenarios vs. Designs, and Key Classes

• Use Scenario + ScenarioRunner / ConcurrentScenarioRunner when comparing a few specific, named configurations — vendor staffing plans, “1 vs 2 vs 3 workers” — especially if the alternatives are structurally different models. Compare with MCB.
• Use a design + DesignedExperiment / ParallelDesignedExperiment when studying how many factors jointly affect a response, including interactions, screening, or curvature.
  • Interactions / main effects, many factors: FactorialDesign, TwoLevelFactorialDesign (and fractions).
  • Optimizing with suspected curvature near the optimum: CentralCompositeDesign + response surface.
• Key classes this deck introduced:
  • Factor, ExperimentalDesignIfc, FactorialDesign, TwoLevelFactorialDesign, CentralCompositeDesign
  • DesignedExperiment / ParallelDesignedExperiment
  • LinearModel (+ regressionResults, showResultsInBrowser)