Comparing System Configurations

Learning Objectives

• To be able to compare simulation alternatives and make valid decisions based on the statistical output of a simulation

Difference Between Systems

• Assume that samples from two different populations (system configurations) are available:

\[X_{11}, X_{12},\ldots, X_{1 n_1} \ \ \text{a sample of size $n_1$ from system configuration 1}\] \[X_{21}, X_{22},\ldots, X_{2 n_2} \ \ \text{a sample of size $n_2$ from system configuration 2}\]

• Assume that each system configuration has an unknown population mean for the performance measure of interest, \(E[X_1] = \theta_1\) and \(E[X_2] = \theta_2\).
• The problem is to determine, with some statistical confidence, whether \(\theta_1 < \theta_2\) or alternatively \(\theta_1 > \theta_2\).
  • Since the system configurations are different, an analysis of the situation of whether \(\theta_1 = \theta_2\) is of less relevance in this context.
• Define \(\theta = \theta_1 - \theta_2\) as the mean difference in performance between the two systems.
  • Clearly, if you can determine whether \(\theta >\) 0 or \(\theta <\) 0 you can determine whether \(\theta_1 < \theta_2\) or \(\theta_1 > \theta_2\).

Indifference Zone Concept

• A confidence interval on the difference can assist in making decisions regarding relative performance of the systems from a statistically significant standpoint.
• However, difference in the system performance may be statistically significant but the actual difference is of no practical use.
• One way to characterize the notion of practical significance is to conceptualize a zone of performance for which you are indifferent between the two systems.
  • If the difference between the two systems falls in this zone, you are indifferent between the two systems (i.e. there is no practical difference).

Types of Statistical Comparisons

• Independent samples from two populations
  • The sample sizes, \(n_1\) and \(n_2\) do not have to be equal.
  • Unknown population variances are either equal or not equal.
  • Tests for equality of variances can be performed using an F-test.
  • If equal variances then estimates for variance can be pooled.
  • If variances are not equal, then an approximate confidence interval is available.
• Dependent samples from two populations
  • While the sample sizes, \(n_1\) and \(n_2\) do not have to be equal, the most common practice is to have a common sample size \(n_1=n_2=n\).
• These cases are covered in detail within introductory statistics courses.

Two Dependent Samples

• Assume that each system is run for the same number of replications, i.e. \(n_1\) = \(n_2\) = \(n\).
• Assume that we run the model to produce \((X_{11}, X_{12}, \ldots, X_{1n_1})\) for configuration 1. Then, change the model (or use a different model) and re-executed the model to produce \((X_{21}, X_{22}, \ldots, X_{2n_2})\) for configuration 2.
• Assume that no changes to the random number streams were made such that both simulation runs utilized the same underlying random numbers.
• This causes the output, \((X_{11}, X_{12},\ldots, X_{1n_1})\) and \((X_{21}, X_{22}, \ldots, X_{2n_2})\) from the two systems to be dependent. That is, within a pair the random variables are dependent, but across the pairs, they will be independent.
• This approach is called common random numbers (CRN)
• The basic approach to analyzing this situation is to compute the difference for each pair: \[D_j = X_{1j} - X_{2j} \ \ \text{for} \; j = 1,2,\ldots,n\]

Analyzing Two Dependent Samples

• The \((D_1, D_2, \ldots, D_n)\) will form a random sample, which can be analyzed via traditional methods.
• A (\(1 - \alpha\))% confidence interval on \(\theta = \theta_1 - \theta_2\) is:

\[\bar{D} = \dfrac{1}{n} \sum_{j=1}^n D_j\]

\[S_D^2 = \dfrac{1}{n-1} \sum_{j=1}^n (D_j - \bar{D})^2\]

\[\bar{D} \pm t_{1-\alpha/2, n-1} \dfrac{S_D}{\sqrt{n}}\]

• The interpretation of the resulting confidence interval \([l, u]\) is the same as any confidence interval.
• This is the paired-t confidence interval presented in statistics textbooks.

Paired Difference Example

• Two configurations are under consideration for the design of an airport security checkpoint. The replication values of the throughput per minute for the security station for each design are provided in the following table.

  	Design 1	Design 2
  1	10.98	8.93
  2	8.87	9.82
  3	10.53	9.27
  4	9.40	8.50
  5	10.10	9.63
  6	10.28	9.55
  7	8.86	9.30
  8	9.00	9.31
  9	9.99	9.23
  10	9.57	8.88

Common Random Numbers and Variance Reduction

• The paired difference method is the preferred method for comparing two systems.
• In the case of independent samples, the estimator of performance is \(\hat{D} = \bar{X}_1 - \bar{X}_2\). Since

\[\begin{aligned} \bar{D} & = \dfrac{1}{n} \sum_{j=1}^n D_j = \dfrac{1}{n} \sum_{j=1}^n (X_{1j} - X_{2j}) \\ & = \dfrac{1}{n} \sum_{j=1}^n X_{1j} - \dfrac{1}{n} \sum_{j=1}^n X_{2j} \\ & = \bar{X}_1 - \bar{X}_2 \\ & = \hat{D}\end{aligned}\]

• The estimators have the same form, when \(n_1 = n_2 = n\); however, the variance of the estimators are not the same.
  • The estimated values will be different because the sampling is performed differently.
  • The method of common random numbers is called a variance reduction technique.

Common Random Numbers and Variance Reduction

• Under the assumption of independence, computing the variance of the estimator yields: \[V_{\text{IND}} = var(\bar{X}_1 - \bar{X}_2) = \dfrac{\sigma_1^2}{n} + \dfrac{\sigma_2^2}{n}\]
• Under the assumption that the samples are not independent, the variance of the estimator is: \[V_{\text{CRN}} = var(\bar{X}_1 - \bar{X}_2) = \dfrac{\sigma_1^2}{n} + \dfrac{\sigma_2^2}{n} - 2\text{cov}(\bar{X}_1, \bar{X}_2)\]
• If you define \(\rho_{12} = corr(\bar{X}_1, \bar{X}_2)\), the variance for the common random number situation is: \[V_{\text{CRN}} = V_{\text{IND}} - 2\sigma_1 \sigma_2 \rho_{12}\]
• Therefore, whenever there is positive correlation \(\rho_{12} > 0\) within the pairs we have that, \(V_{\text{CRN}} < V_{\text{IND}}\).

Executing Independent Experiments using the KSL

fun withoutCommonRandomNumbers(){
    val model = Model("Pallet Model Experiments")
    model.numberOfReplications = 30
    val palletWorkCenter = PalletWorkCenter(model)
    val kslDatabaseObserver = KSLDatabaseObserver(model)
    model.experimentName = "Two Workers"
    palletWorkCenter.numWorkers = 2
    model.simulate()
    model.experimentName = "Three Workers"
    palletWorkCenter.numWorkers = 3
    model.simulate()
    val responseName = palletWorkCenter.totalProcessingTime.name
    val db = kslDatabaseObserver.db
    val expNames = listOf("Two Workers", "Three Workers")
    val comparisonAnalyzer = db.multipleComparisonAnalyzerFor(expNames, 
      responseName)
    println(comparisonAnalyzer)
}

Comments on Executing Independent Experiments using the KSL

• The two invocations of simulate() are sequential with no manipulation of the streams.
• Notice that the name of the experiment was changed between invocations of simulate().
• The random numbers for the second invocation continue within the streams causing the simulation runs to be independent.
• The MultipleComparisonAnalyzer class is used to compute the differences.
• If we had run the two worker model today and then changed the number of workers to 3 for a run tomorrow, the simulations would have used the same random number streams (common random numbers by default).
• If you want results tomorrow to be different from results from yesterday, then you can manipulate the streams before you simulate. A simply way is to advance the substreams:

model.advanceToNextSubStream()
model.advanceSubStreams(n: Int)

Results for Independent Experiments

• Note the sample standard deviation on the difference, \(s_d = 57.8792\).
• The confidence interval does not contain zero, we can conclude with 95% confidence that the system with two workers has a longer time to complete the processing of the pallets.

Statistical Summary Report

Difference Data 
Name                              Count       Average       Std. Dev. 
---------------------------------------------------------------------- 
Two Workers - Three Workers       30          64.7723         57.8792 
----------------------------------------------------------------------- 
95.0% Confidence Intervals on Difference Data
Two Workers - Three Workers [43.15986335728935, 86.38476883829914]

Executing CRN Experiments using the KSL

• Note the line model.resetStartStreamOption = true

fun withCommonRandomNumbers(){
    val model = Model("Pallet Model Experiments")
    model.numberOfReplications = 30
    val palletWorkCenter = PalletWorkCenter(model)
    val kslDatabaseObserver = KSLDatabaseObserver(model)
    model.resetStartStreamOption = true
    model.experimentName = "Two Workers"
    palletWorkCenter.numWorkers = 2
    model.simulate()
    model.experimentName = "Three Workers"
    palletWorkCenter.numWorkers = 3
    model.simulate()
    val responseName = palletWorkCenter.totalProcessingTime.name
    val db = kslDatabaseObserver.db
    val expNames = listOf("Two Workers", "Three Workers")
    val comparisonAnalyzer = db.multipleComparisonAnalyzerFor(expNames, responseName)
    println(comparisonAnalyzer)
}

Comments on Executing CRN Experiments using the KSL

• Note the line model.resetStartStreamOption = true
• This causes the random number streams to be reset before each invocation of simulate()
• Notice that the name of the experiment was changed between invocations of simulate().
• If we had run the two worker model today and then changed the number of workers to 3 for a run tomorrow, the simulations would have used the same random number streams (common random numbers by default).
  • To do this and capture the results in the database takes special care.
  • You need to reattach the previously created database so that the second run is also written to the database. You must make sure to change the experiment name so that the first run’s results are not over written.
• If you do not want to use the database, then similar functionality is available via the ExperimentDataCollector class.
  • It uses instances of the ReplicationDataCollector class to capture response data across replications and can produce an instance of the MultipleComparisonAnalyzer class.

Results for CRN Experiments

• We see the that the sample standard deviation on the difference, \(s_d = 33.0561\) for the case of common random numbers has been significantly reduced from the independent case (\(s_d = 57.8792\), about a 43% reduction in this case).
• Notice that the estimates are also different \(\bar{x}_{IND} = 64.7723\) versus \(\bar{x}_{CRN} = 64.9215\). This is because the two cases used different random numbers. That is, they were sampled differently.
• We get more precision on the estimate of the difference for no extra computing.

Statistical Summary Report

Difference Data 
Name                              Count       Average       Std. Dev. 
---------------------------------------------------------------------- 
Two Workers - Three Workers       30          64.9215         33.0561 
---------------------------------------------------------------------- 
95.0% Confidence Intervals on Difference Data
Two Workers - Three Workers [52.57814571744155, 77.26483056807177]

Individual Decision Risk versus Joint Decision Risk

• Suppose that you are interested in analyzing \(k\) systems based on performance measures \(\theta_i, \, i = 1,2, \ldots, k\)
  • For example, \(\theta_i\) might be the inventory cost for the \(i^{th}\) policy setting.
• In order to perform an analysis, each \(\theta_i\) must be estimated, which results in sampling error for each individual estimate of \(\theta_i\).
• The overall decision on which configuration is best will be based upon all the estimates of \(\theta_i\) (e.g. for every policy setting).
• The sampling error includes the sampling error associated with each configuration.
• This compounds the risk associated with an overall decision.

Bonferroni Concepts

• The Bonferroni inequality states that given a set of (not necessarily independent) events, \(E_i\), which occur with probability, \(1-\alpha_i\), for \(i = 1,2,\ldots, k\) then a lower bound on the probability of the intersection of the events is given by:

\[P \lbrace \cap_{i=1}^k E_i \rbrace \geq 1 - \sum_{i=1}^k \alpha_i\]

• In words, the Bonferroni inequality states that the probability of all the events occurring is at least one minus the sum of the probability of the individual events occurring.

Bonferroni Effects on Confidence Intervals

• Suppose that you have \(c\) confidence intervals each with confidence \(1-\alpha_i\).
• The \(i^{th}\) confidence interval is a statement \(S_i\) that the confidence interval procedure will result in an interval that contains the parameter being estimated.
• If you define events, \(E_i = \lbrace S_i \text{is true}\rbrace\), then the intersection of the events can be interpreted as the event representing all the statements being true. \[P\lbrace \text{all} \, S_i \, \text{true}\rbrace = P \lbrace \cap_{i=1}^k E_i \rbrace \geq 1 - \sum_{i=1}^k \alpha_i = 1 - \alpha_E\] where \(\alpha_E = \sum_{i=1}^k \alpha_i\). The value \(\alpha_E\) is called the overall error probability.
• This statement can be restated in terms of its complement event as: \[P\lbrace \text{one or more} \, S_i \, \text{are false} \rbrace \leq \alpha_E\]
• This gives an upper bound on the probability of a false conclusion based on the procedures that generated the confidence intervals.

A Depressing Example

• Suppose that you have, \(c = 10\) confidence intervals, each at the 90% confidence level to interpret. Thus, \(\alpha_i = 0.10\), so that \[\alpha_E = \sum_{i=1}^{10} \alpha_i = \sum_{i=1}^{10} (0.1) = 1.0\]
• Thus, \(P\lbrace \text{all} \, S_i \, \text{true}\rbrace \geq 0\) or \(P\lbrace\text{one or more} \, S_i \, \text{are false} \rbrace \leq 1\).
• In words, this is implying that the chance that all the confidence intervals procedures result in confidence intervals that cover the true parameter is greater than zero and less than 1.
• Think of it this way: If your boss asked you how confident you were in your decision, you would have to say that your confidence is somewhere between zero and one.
  • This would not be very reassuring to your boss (or for your job!).

Comparing Multiple Systems Against a Base Case

• If you are comparing \(k\) systems where one of the systems is the standard (e.g. base case, existing system, etc.), you can reduce the number of confidence intervals by analyzing the difference between the other systems and the standard.

• That is, suppose system one is the base case, then you can form confidence intervals on \(\theta_1 - \theta_i\) for \(i = 2,3,\ldots, k\). Since there are \(k - 1\) differences, there are \(c = k - 1\) confidence intervals to compare.

Comparing Multiple Systems Against Each Other

• If you are interested in developing an ordering between all the systems, then one approach is to make all the pair wise comparisons between the systems. That is, construct confidence intervals on \(\theta_j - \theta_i\) for \(i \neq j\).

• The number of confidence intervals in this situation is

\[c = \binom{k}{2} = \frac{k(k - 1)}{2}\]

• The trade-off between overall error probability and the width of the individual confidence intervals will become severe for most practical situations.
  • For example, if \(k = 8\), then there will be \(c = (8\times 7)/2 = 28\) confidence intervals.
  • If all intervals have the same \(\alpha\) and we want an overall \(\alpha_E\) to be 0.05, then \(\alpha = 0.001786\), which will result in much wider confidence intervals.
  • This will increase the likelihood that the confidence intervals overlap and make it difficult to provide an ordering.

Multiple Comparison with the Best Concepts

• Multiple comparison with the best (MCB) procedures are statistical procedures designed to select the best alternative out of a set of alternatives under conditions of sampling error.
• The MCB confidence interval is based on the difference between the best and the best of the rest.
• Suppose we have \(k\) system configurations to compare and suppose that somehow you knew that the \(i^{th}\) system is the best. Now, consider a confidence interval for system’s \(i\) performance metric, \(\theta_i\), of the following form: \[ \theta_i - \max_{j \neq i} \theta_j \]
• This difference is the difference between the best (\(\theta_i\)) and the second best. This is because if \(\theta_i\) is the best, when we find the maximum of those remaining, it will be next best (i.e. second best)
• Now, let us suppose that system \(i\) is not the best. Reconsider, \(\theta_i - \max_{j \neq i} \theta_j\). Then, this difference will represent the difference between system \(i\), which is not the best, and the best of the remaining systems.
• Thus, this difference is sufficient to indicate which is the best.

MCB Confidence Intervals

• MCB procedures build \(k\) confidence intervals of the form:

\[ \theta_i - \max_{j \neq i} \theta_j \ \text{for} \ i=1,2, \dots,k \]

• Therefore, only \(k\) confidence intervals need to be considered to determine the best rather than, \(\binom{k}{2}\).

• This form of confidence interval has also been combined with the concept of an indifference zone.

  • Indifference zone procedures use a parameter \(\delta\) that indicates that the decision maker is indifferent between the performance of two systems, if the difference is less than \(\delta\).
  • These procedures attempt to ensure that the overall probability of correct selection of the best is \(1-\alpha\), whenever, \(\theta_{i^{*}} - \max_{j \neq i^{*}} \theta_j > \delta\), where \(i^{*}\) is the best system and \(\delta\) is the indifference parameter.

MCB Confidence Intervals

• The results of a MCB analysis may not be able to distinguish between the best and others.

• For example, assuming that we desire the maximum (bigger is better), then let \(\hat{i}\) be the index of the system found to be the largest and let \(\hat{\theta}_{i}\) be the estimated performance for system \(i\), then, these procedures allow the following:

• If \(\hat{\theta}_{i} - \hat{\theta}_{\hat{i}} + \delta \leq 0\), then we declare that system \(i\) not the best.

• If \(\hat{\theta}_{i} - \hat{\theta}_{\hat{i}} + \delta > 0\), then we can declare that system \(i\) is not statistically different from the best. In this case, system \(i\), may in fact be the best.

Using the MultipleComparisonAnalyzer Class

• The MultipleComparisonAnalyzer class depends on having the data to be compared in the form of named arrays within a map dataMap: Map<String, DoubleArray>.

• Each array is associated with the name of the array via the map’s key parameter. There needs to be at least 2 data arrays, and the length of each data array must be the same.

• The KSL facilitates extracting and organizing the simulation data in this form from the KSL database.

  • Such data can also be readily extracted from instances of a ReplicationDataObserver or an ExperimentDataObserver.

MultipleComparisonAnalyzer Class Computes the Following

• statistical summaries and confidence intervals on the individual data arrays
• statistical summaries and confidence intervals on all pair-wise differences
• the variance of the difference that is the largest
• the data set that has the smallest average and its average
• the data set that has the largest average and its average
• the pair-wise difference that is the smallest and its value
• the pair-wise difference that is the largest and its value
• multiple comparison confidence intervals based on finding the maximum value
• multiple comparison confidence intervals based on finding the minimum value
• screening intervals for best designs by constructing subsets containing the best configuration

Two-Stage Bonferroni Procedure

• Specify the indifference parameter, \(\delta\), PCS as \((1-\alpha)\), the initial sample size, \(n_0\), and compute \(t= t_{1-\alpha/(k-1),n_0 - 1}\)

• Execute the \(n_0\) replications for each system and capture \(Y_{ij}\) as the \(i^{th}\) observation of system \(j\)

• Compute \(\bar{Y}_{\cdot j}\) as the mean performance of system \(j\), for \(j=1,2,\cdots,k\). Then for all \(i \neq j\), compute the sample variance of the differences and the largest sample variance \(\hat{S}^2 = max_{i \neq j}S^{2}_{ij}\)

\[ S^{2}_{ij} = \frac{1}{n_0 - 1} \sum_{r=1}^{n_0} ((Y_{ri} - Y_{rj}) - (\bar{Y}_{\cdot i} - \bar{Y}_{\cdot j}))^2 \]

• Calculate the second stage sample size:

\[ n = max \Bigl\{ n_0, \Bigl \lceil \frac{t^2\hat{S}^2}{\delta^2} \Bigr \rceil \Bigr\} \]

Two-Stage Bonferroni Procedure

• Make \(n - n_0\) additional replications (or just make \(n\) by re-running). Capture \(Y_{ij}\) as the \(i^{th}\) observation of system \(j\)

• Calculate the overall sample means for \(j=1, 2,\cdots, k\) for each design.

\[ \bar{Y}_{\cdot j} = \frac{1}{n}\sum_{r=1}^{n}Y_{rj} \]

• If maximizing, select the system with the largest \(\bar{Y}_{\cdot j}\) as the best and form the confidence intervals:

\(\min\{0, \bar{Y}_{\cdot j} - \max_{j \neq i}\bar{Y}_{\cdot j}- \delta \} \leq \theta_i - \max_{j \neq i} \theta_j \leq \max\{0, \bar{Y}_{\cdot j} - \max_{j \neq i}\bar{Y}_{\cdot j}- \delta \}\)

If minimizing, select the system with the smallest \(\bar{Y}_{\cdot j}\) as the best and form the confidence intervals:

\(\min\{0, \bar{Y}_{\cdot j} - \min_{j \neq i}\bar{Y}_{\cdot j}- \delta \} \leq \theta_i - \max_{j \neq i} \theta_j \leq \max\{0, \bar{Y}_{\cdot j} - \min_{j \neq i}\bar{Y}_{\cdot j}- \delta \}\)

Using the MultipleComparisonAnalyzer Class for Two-Stage Bonferroni

• The MultipleComparisonAnalyzer class automatically computes everything that is needed to apply the Two-Stage Bonferroni procedure:

  • use the toString() method to see all the results

• The second stage sample size can be computed via the function secondStageSampleSizeNM()

• The maximum variance is found from property, maxVarianceOfDifferences

• The maximum or minimum performer is available from nameOfMaximumAverageOfData and nameOfMinimumAverageOfData as well has the respective performance via maximumAverageOfData and minimumAverageOfData

• The intervals for each case are found from mcbMaxIntervals(delta) and mcbMinIntervals(delta) respectively, where delta is the indifference zone parameter.

MCB Example Results

MCB Maximum Intervals
Indifference delta: 0.0
Name        Interval
One     [0.0, 0.7999999999999972]
Two     [-0.7999999999999972, 0.0]
Three       [-5.686000000000007, 0.0]
Four        [-1.2579999999999956, 0.0]

MCB Minimum Intervals
Indifference delta: 0.0
Name        Interval
One     [0.0, 5.686000000000007]
Two     [0.0, 4.88600000000001]
Three       [-4.4280000000000115, 0.0]
Four        [0.0, 4.4280000000000115]

Comments on Interpretting MCB Confidence Intervals

• Notice how the form of these confidence interval equations require that for the case of finding the maximum, systems that are candidates for the maximum have a lower interval limit of 0.0, while candidate systems that should be ruled out have an upper limit of 0.0.

• Note that the confidence intervals (as defined here) do not have a distribution requirement for computing a test statistic value. Thus, their forms do not depend on the distributional assumptions.

• However, the probability of correctly selecting the best will depend upon some distributional assumptions.

• If the confidence interval contains zero, then it cannot be ruled out as the best. When this occurs, it is advisable to increase the sample size to see if a clear winner emerges.

Two-Stage Bonferroni Example Data

Two-Stage Bonferroni Application

• Row 4 contains the sample variances. Clearly, the maximum variance is \(\hat{S}^2 = 28.498\)
• Suppose that we want to be 95% confident of selecting the best (smallest) performance when the best differs from the second best by at least 5 minutes.
• Note that \(\delta = 5\), \(\alpha = 0.05\), \(n_0 = 10\), and \(t= t_{1-0.0167,9}=2.508\)
• Applying step 4 of the Two-Stage Bonferroni Procedure indicates:

\[ n = \max \Bigl\{ n_0, \Bigl \lceil \frac{t^2\hat{S}^2}{\delta^2} \Bigr \rceil \Bigr\}=\max(10, \frac{2.508^2 \, 28.498}{5^2})=\max(10,7.17) = 10 \]

• Thus, the \(n_0 = 10\) of the first stage are sufficient.
  • If more samples were needed, the second stage sampling would proceed.
• The MCB confidence intervals can be computed based on the first stage sampling in this case.

Two-Stage Bonferroni Example Confidence Interval Results

Screening Procedures

• Assuming that the data are normally distributed (with or without CRN), screening procedures can be designed to select a subset of the system designs such that the retained subset will contain the true best system with probability greater than \(1-\alpha\).

• Such procedures are useful when there are a large number of designs or during automated search procedures.

• Once a reliable subset has been found then MCB procedures can be used to select the best from the designated subset.

• The following procedure is based on: Nelson et al. (2001) “Simple Procedures for Selecting the Best System when the Number of Alternatives is Large”, Operations Research, vol. 49, pp.950-963.

Screening Example Data

Screening Example Calculations

• Note that \(\hat{se}_{ij} = S_{ij}/\sqrt{n}\).

• If smaller is better, then place system \(i\) in the selection subset if for all \(j \ne i\), \(\bar{Y}_{\cdot i} \le \bar{Y}_{\cdot j} + t * \hat{se}_{ij}\)

• Example: \(k=4\), \(\alpha = 0.05\), \(n=10\), \(t= t_{1-\alpha/(k-1),n - 1} = 2.5096\) \[ \begin{aligned} \bar{Y}_{\cdot 1} = 50.036 & \overset{?}{\le} \bar{Y}_{\cdot 2} + t * \hat{se}_{12} = 49.236 + 2.5096 \times 0.6707=50.9191\\ & \overset{?}{\le} \bar{Y}_{\cdot 3} + t * \hat{se}_{13} = 44.35 + 2.5096 \times 1.6881 = 48.5865\\ & \overset{?}{\le} \bar{Y}_{\cdot 4} + t * \hat{se}_{14} = 48.778 + 2.5096 \times 0.7409 = 50.6374\\ \end{aligned} \]

• Note that system 1 fails to be less than all the limits. Thus, system 1 is eliminated.

• This calculation is continued for all the systems.

Screening Example Results

KSL MCB Example

In this KSL code example, three configurations are compared using CRN and MCB.

fun main() {
    val model = Model("Pallet Model MCB")
    // add the model element to the main model
    val palletWorkCenter = PalletWorkCenter(model)
    // use a database to capture the response data
    val kslDatabaseObserver = KSLDatabaseObserver(model)
    // simulate the model
    model.experimentName = "One Worker"
    palletWorkCenter.numWorkers = 1
    model.resetStartStreamOption = true
    model.numberOfReplications = 30
    model.simulate()
    model.experimentName = "Two Workers"
    palletWorkCenter.numWorkers = 2
    model.simulate()
    model.experimentName = "Three Workers"
    palletWorkCenter.numWorkers = 3
    model.simulate()
    val responseName = palletWorkCenter.totalProcessingTime.name
    val db = kslDatabaseObserver.db
    val expNames = listOf("One Worker","Two Workers", "Three Workers")
    val comparisonAnalyzer = db.multipleComparisonAnalyzerFor(expNames, responseName)
    println(comparisonAnalyzer)
}

KSL MCB Example Abbrevated Results

It should be obvious that 3 workers results in the smallest total processing time. Notice in the screening output that the 3 worker case is the only system that passes all the screening tests.

MCB Minimum Intervals
Indifference delta: 0.0
Name        Interval
One Worker      [0.0, 517.7287463179446]
Two Workers     [0.0, 64.92148814275674]
Three Workers       [-64.92148814275674, 0.0]

Min Screening Intervals
One Worker  average = 943.8223691355911     Two Workers [-Infinity, 499.44125093353716]
One Worker  average = 943.8223691355911     Three Workers   [-Infinity, 444.40016711694113]
Two Workers average = 491.0151109604032     One Worker  [-Infinity, 952.2485091087251]
Two Workers average = 491.0151109604032     Three Workers   [-Infinity, 438.43696524296155]
Three Workers   average = 426.09362281764646        One Worker  [-Infinity, 962.1289134348857]
Three Workers   average = 426.09362281764646        Two Workers [-Infinity, 503.3584533857183]

Alternatives remaining after screening for minimum:
[Three Workers]