Finite Horizon Simulation Experiments

Learning Objectives

• To be able to recognize the different types of statistical quantities used within and produced by simulation models

• To be able to analyze finite horizon simulations via the method of replications

Types of Statistical Variables

• A replication is the generation of one sample path which represents the evolution of the system from its initial conditions to its ending conditions.

• The statistical quantities collected during a replication are called within replication statistics.

• The statistical quantities collected across replications are called across replication statistics. Across replication statistics are collected based on the observation of the final values of within replication statistics.

The Concept of Replicated Sample Paths

If you have multiple replications, each replication represents a different sample path, starting from the same initial conditions and being driven by the same input parameter settings.

Types of Observations Within a Replication

• Tally observations represent a sequence of equally weighted data values that do not persist over time. This type of data is associated with the duration or interval of time that an object is in a particular state or how often the object is in a particular state. As such it is observed by marking (tallying) the time that the object enters the state and the time that the object exits the state. Once the state change takes place, the observation is over (it is gone, it does not persist, etc.).

• Time-persistent observations represent a sequence of values that persist over some specified amount of time with that value being weighted by the amount of time over which the value persists.

• Tally-based statistics and time-persistent statistics both are collected during a replication and form within replication statistical quantities. When we compute statistics across the replications, we call these statistics across replication statistics.

Time-Persistent Sample Path

The following figure illustrates a single sample path for the number of customers in a queue over a period of time. From this sample path, events and subsequent statistical quantities can be observed.

Types of Simulation Horizons

• Finite horizon: In a finite-horizon simulation, a well defined ending time or ending condition can be specified which clearly defines the end of the simulation.
  • Finite horizon simulations are often called terminating simulations, since there are clear terminating conditions.
• Infinite horizon: In an infinite horizon simulation, there is no well defined ending time or condition. Also, there are no clearly defined initial conditions.
  • The planning period is over the life of the system, which from a conceptual standpoint lasts forever.
  • Infinite horizon simulations are often called steady state simulations because in an infinite horizon simulation you are often interested in the long-term behavior of the system.

Within Replication Averages

• Suppose that \(n\) replications of a simulation are available where each replication is terminated by some event \(E\) and begun with the same initial conditions.

• Let \(Y_{rj}\) be the \(j^{th}\) observation on replication \(r\) for \(j = 1,2,\cdots,m_r\) where \(m_r\) is the number of observations in the \(r^{th}\) replication, and \(r = 1,2,\cdots,n\), and define the sample average for each replication to be:

\[\bar{Y}_r = \frac{1}{m_r} \sum_{j=1}^{m_r} Y_{rj}\]

If the data are time-based then,

\[\bar{Y}_r = \frac{1}{T_E} \int_0^{T_E} Y_r(t) \mathrm{d}t\]

Across Replication Statistics

• \(\bar{Y}_r\) is the sample average based on the observations within the \(r^{th}\) replication. Therefore, \(\bar{Y}_r\) for \(r = 1,2,\ldots,n\) forms a random sample. This leads to the across replication statistics for the average and variance:

\[ \bar{\bar{Y}} = \frac{1}{n}\sum_{r=1}^{n}\bar{Y}_r \]

\[ S^{2}(n) = \frac{1}{n-1}\sum_{r=1}^{n}(\bar{Y}_r-\bar{\bar{Y}})^2 \]

• And, a \((1-\alpha)100\%\) two-sided confidence interval estimate:

\[ \bar{\bar{y}} \pm t_{1-(\alpha/2), n-1} \dfrac{s}{\sqrt{n}} \]

Analyzing Finite Horizon Simulations

• The method of independent replications is used to analyze finite horizon simulations.

• All the statistical concepts that you have learned that can be applied to a random sample of observations are applicable to the analysis of across replication data from a finite horizon simulation.

• Use the already discussed methods to determine the sample size to determine the number of replications.

• Collect and analyze the data across the replications. The KSL facilitates this process.

KSL Functionality for Capturing Statistical Results

• Automatically capture within and across replication results to comma separated value files

• Capture replication data via the ReplicationDataCollector class

• Trace a response variable via the ResponseTrace class

• Use the SimulationReporter class to output results in a variety of formats

• Capture all simulation results to a database using the KSLDatabaseObserver class

KSL Output Directory

• When a model is run, an output directory is created to contain results from the model’s execution.

• The directory is given the same name as the model. For example, if the model was called “Pallet Processing”, then the directory will be called “Pallet_Processing_OutputDir”.

• The following sub-folders will be created:

  • csvDir - to hold CSV files
  • dbDir - to hold database related files
  • excelDir - to hold files related to Excel
  • plotDir - to hold files generated by plotting functions

• The model’s output directory will be placed in the kslOutput directory, which will be found in the directory where the model was executed.

  • In an IntelliJ project, this is in the project’s main directory.

Organization of KSL Output Directory

Automatically capture within and across replication results to CSV files

• Set the autoCSVReports option to true to generate two CSV files that contain the within replication statistics and across replication statistics for every Response, TWResponse, and Counter. If the model is named “Pallet Processing”, then the CSV files can be found in the csvDir directory as:
  • Pallet_Processing_CSVExperimentReport.csv
  • Pallet_Processing_CSVReplicationReport.csv

fun main() {
    // notice setting the autoCSVReports option to true
    val model = Model("Pallet Processing", autoCSVReports = true)
    model.numberOfReplications = 10
    model.experimentName = "Two Workers"
    // add the model element to the main model
    val palletWorkCenter = PalletWorkCenter(model)
    // simulate the model
    model.simulate()
}

Collect Replication Data into Arrays for Post Processing

• Attach a ReplicationDataCollector before running the simulation
• Add specific responses or all responses to the collector

fun main() {
    // notice setting the autoCSVReports option to true
    val model = Model("Pallet Processing", autoCSVReports = true)
    model.numberOfReplications = 10
    model.experimentName = "Two Workers"
    // add the model element to the main model
    val palletWorkCenter = PalletWorkCenter(model)
    // demonstrate capture of replication data for specific response variables
    val repData = ReplicationDataCollector(model)
    repData.addResponse(palletWorkCenter.totalProcessingTime)
    repData.addResponse(palletWorkCenter.probOfOverTime)
    // simulate the model
    model.simulate()
    //output the collected replication data to prove it was captured
    println(repData)
}

Results of ReplicationDataCollector

• Each added response variable captures the response from the replication.
  • In the case of totalProcessingTime, this is the ending within replication average.
  • In the case of probOfOverTime, this is the indicator of whether there was overtime.

   Total Processing Time P{total time > 480 minutes}
 0            482.417201                         1.0
 1            461.544205                         0.0
 2            521.417293                         1.0
 3            476.025297                         0.0
 4            534.281150                         1.0
 5            485.735690                         1.0
 6            477.468018                         0.0
 7            482.557886                         1.0
 8            499.628817                         1.0
 9            471.007443                         0.0

Write Report as a MarkDown Table

    // the reports can have specified confidence level
    val sr = model.simulationReporter
    sr.printHalfWidthSummaryReport(confLevel = .99)
    // the report can be written to MarkDown as a table in the output directory
    var out = model.outputDirectory.createPrintWriter("hwSummary.md")
    sr.writeHalfWidthSummaryReportAsMarkDown(out)
    println()

Capture the Results into a Database

• Attach a KSLDatabaseObserver prior to running the simulation.
  • The database will be stored in the dbDir and labeled with the name of the simulation model. In this case: Pallet_Processing.db

    // demonstrate capturing data to database with an observer
    val kslDatabaseObserver = KSLDatabaseObserver(model)
    // simulate the model
    model.simulate()
    // use the database to create a Kotlin DataFrame of results
    val dataFrame = kslDatabaseObserver.db.acrossReplicationStatistics
    println(dataFrame)

Many Results can be Extracted from the Database

      exp_name                   stat_name stat_count    average   std_dev
 0 Two Workers           Num Pallets at WC       10.0   9.314533  3.088238
 1 Two Workers               Num Processed       10.0  80.400000  3.502380
 2 Two Workers              NumBusyWorkers       10.0   1.917471  0.064525
 3 Two Workers              PalletQ:NumInQ       10.0   7.397062  3.050233
 4 Two Workers             PalletQ:TimeInQ       10.0  44.659671 17.832513
 5 Two Workers P{total time > 480 minutes}       10.0   0.600000  0.516398
 6 Two Workers                 System Time       10.0  56.320448 17.813306
 7 Two Workers       Total Processing Time       10.0 489.208300 22.823412

Sequential Sampling Across Replications

• While previous discussion has indicated how to determine the sample size, it is possible to sequentially sample until a half-width criterion is met: \[h(n) = t_{\alpha/2, n - 1} \dfrac{s(n)}{\sqrt{n}} \leq E\]

    override fun afterReplication(modelElement: ModelElement) {
        if (modelElement.model.currentReplicationNumber <= 2){
            return
        }
        val statistic = myResponse.myAcrossReplicationStatistic
        val hw = statistic.halfWidth(confidenceLevel)
        if (hw <= desiredHalfWidth){
            modelElement.model.endSimulation("Half-width = ($desiredHalfWidth) 
              condition met for response ${myResponse.name}")
        }
    }

Using the Half-Width Checker

• Attach an instance of AcrossReplicationHalfWidthChecker to a response.
• Set the number of replications very high. The checker will stop the replications when the criterion is met.

fun main() {
    val model = Model("Pallet Processing Ex 2")
    model.numberOfReplications = 10000
    model.experimentName = "Two Workers"
    // add the model element to the main model
    val palletWorkCenter = PalletWorkCenter(model)
    val hwc = AcrossReplicationHalfWidthChecker(palletWorkCenter.totalProcessingTime)
    hwc.desiredHalfWidth = 5.0
    // simulate the model
    model.simulate()
    // demonstrate that reports can have specified confidence level
    val sr = model.simulationReporter
    sr.printHalfWidthSummaryReport()
}

Summary of New Concepts

• Finite horizon simulations have clear initial and ending conditions.
• Within replication statistics are collected from the sample path.
• Across replication are collected across the sample paths.
• The statistical analysis of finite horizon simulation involves the analysis of a random sample formed from the replications.
• The KSL provides many options for collecting and analyzing simulation data.
  • Output is stored in a directory formed from the name of the simulation.
  • CSV within and across replication data can be automatically collected.
  • In memory arrays of the responses can be collected using the ReplicationDataCollector class.
  • Output can be written in many forms (e.g. MarkDown, LaTeX).
  • All statistical results can be captured by using an instance of KSLDatabaseObserver.
  • The AcrossReplicationHalfWidthChecker class facilitates sequential sampling.