Chapter 10 Experimental Design and Simulation Optimization Methods

Learning Objectives

To be able to explain and discuss the application of experimental design methods within a simulation modeling context
To be able to recognize how simulation experiments are different from standard application of experimental methods
To be able to apply experimental design methods within the KSL for simulation analysis
To be understand how optimization methods can be applied within a simulation context
To be able to discuss the challenges associated with applying optimization methods to simulation models
To recognize and explain some common simulation optimization algorithms
To be able to apply KSL constructs for simulation optimization

This chapter builds on the methods presented in Chapter 5. Section 5.7 introduced how simulation models can be used to compare different system configurations. An important aspect of the comparison process was to select the best simulation configuration (or to screen out inferior configurations). In essence, this is optimization. Section 5.8 illustrates how to simulate many configurations. Both of these situations beg one particular question. How to determine the configurations to be analyzed?

A simulation model has many parameters that can be varied. Each parameter setting represents a different system configuration. All possible parameter settings represent the design space of the problem. This chapter presents concepts for how to explore the design space of a simulation model. We do this to make inferences about how the real system will behave under different parameter settings (i.e. for different design configurations).

Section 10.1 of this chapter presents methods that allow the analyst to systematically explore the design space using experimental design methods. Using experimental design methods facilitates efficient allocation of simulation runs for the purpose of understanding the relationships between the inputs and outputs of the model. Within this context, the analyst must explicitly specify the framework for setting the parameters to obtain the desired analysis. That is, the analyst must specify an experimental design.

Section 10.2 of the chapter presents an approach to exploring the design space that is driven by optimal search methods. Specifically, the section illustrates standard search methods such as stochastic hill climbing, simulated annealing, and the cross-entropy method. An important aspect of applying these approaches to “optimize” a simulation model is the fact that the output from the simulation model has uncertainty. The uncertainty of the objective response or in constraints will require additional understanding of the effect of uncertainty on the search process and on the interpretation of the results from the search. Unlike the approach based on experimental designs, a simulation optimization approach is driven via an algorithm that may or may not have guarantees on the quality of the solution produced.

NOTE! This chapter provides example code of using the KSL to implement experimental design and optimization techniques. The full source code of the examples can be found in the accompanying KSLExamples project associated with the KSL repository. The files for each example of this chapter can be found here.