Jorge Quitério Search
Mathematics

Input distributions in simulation models

Simulation models depend on their input distributions. Three small experiments, on distribution fitting, a newsvendor model and two queues, show how strongly the shape of the inputs affects the results and how easily it is misjudged.

Simulation is often presented as an alternative to analytical models: instead of assuming a convenient formula, the randomness is modelled explicitly and the computer generates its consequences. The results, however, are only as reliable as the inputs.

In the simulation part of my optimisation course, three small experiments showed how strongly the results depend on the inputs, and specifically on the parts of the input distributions that an average does not describe. This article describes the three experiments and reproduces two of them: a goodness-of-fit test that cannot tell two families apart, and two queues with the same average service time and very different lengths.

The relevant literature is as follows.

Law (2011) and Biller and Nelson (2002) summarise the practice of input modelling for simulation: choose candidate families, fit them, test the fit, and above all use knowledge of the physical process.

Kolmogorov (1933) and Smirnov (1948) gave the test based on the largest gap between the empirical and the fitted distribution functions. Lilliefors (1967) showed that when the parameters are estimated from the same data, the standard Kolmogorov-Smirnov p-values are too high, and tabulated corrected values for the normal case.

Pollaczek (1930) and Khinchine (1932) derived the mean queue length of a single server with random arrivals and general service times, the formula used below.

1. Goodness-of-fit tests and similar families

I generated 50 observations each from known distributions and asked a standard Kolmogorov-Smirnov procedure to identify them. For two of three samples it accepted the wrong family with a high p-value: a gamma for the normal sample (p=0.975p = 0.975) and a Weibull for the lognormal one (p=0.988p = 0.988).

The test was functioning correctly. A high p-value says only that the data are consistent with a family, and with 50 points several families are consistent at once. The practical recommendation is therefore to choose families for physical reasons (a duration cannot be negative, a count is discrete), check the tails, and treat the fit as a modelling decision rather than a statistical verdict.

Fifty is a realistic number in practice. An analyst who times fifty transactions at a bank counter, or collects fifty ambulance call-out durations, before simulating a new layout or rota, will often find that a normal and a lognormal model both pass the test. The two models give similar averages but very different numbers of very long transactions, and section 3 shows that the long transactions are the ones that build queues.

2. The newsvendor model

A shop orders QQ copies of a book before demand DD is known. Each sells for 24 €, costs 12 € and can be returned for 6 €:

π(Q,D)=24 min⁡(D,Q)+6 max⁡(Q−D,0)−12 Q. \pi(Q, D) = 24\,\min(D, Q) + 6\,\max(Q - D, 0) - 12\,Q .

With 10,000 simulated demands per order size, the best order was Q=80Q = 80, with an expected profit of 742.02 €. The profit function is asymmetric around the optimum: ordering 70 cost only about 3 € in expectation, ordering 90 about 31 €. The prices fix where the optimum lies: a missed sale costs 12 € of margin and an unsold copy 6 €, so the best order is the demand quantile F(Q)=12/(12+6)=2/3F(Q) = 12/(12+6) = 2/3. A florist ordering roses before Valentine's Day, or a café deciding how many sandwiches to prepare, solves the same problem with its own prices. How steeply profit falls on either side depends on the shape of demand near that quantile. The same logic, with uncertainty made explicit, drives the value of the stochastic solution.

3. Service-time variability and queue length

A single queue, like a cash machine, with random arrivals at 48 per hour and a mean service time of one minute: the server is busy 80% of the time. Two versions differed only in the shape of the service time:

Service time Mean Average number in system
Exponential (M/M/1) 1 min 3.68 (simulated)
Triangular (0.25, 1, 1.75) min 1 min 2.43 (simulated)

With the same arrivals and the same mean service time, the triangular case had about a third fewer customers in the system. The effect is familiar from a post-office counter: the queue grows long less because of the average customer than because of the occasional customer whose parcel needs several forms. Supermarket express lanes for small baskets work partly on this principle, since separating short transactions from long ones reduces the variability each queue faces. The Pollaczek-Khinchine formula makes the reason explicit:

L=ρ+ρ2 (1+cs2)2 (1−ρ), L = \rho + \frac{\rho^2\,(1 + c_s^2)}{2\,(1-\rho)},

with ρ\rho the utilisation and csc_s the coefficient of variation of the service time. For exponential service cs2=1c_s^2 = 1; for this triangular distribution cs2≈0.094c_s^2 \approx 0.094. At ρ=0.8\rho = 0.8 the formula gives 4.0 and about 2.55. The course simulation came out a little lower than both, consistent with a slightly lower observed utilisation in its finite run.

The figures below illustrate two results: with fifty observations a goodness-of-fit test usually cannot reject a wrong but plausible family, and at the same utilisation and mean service time, the queue length depends on the variance of the service time exactly as Pollaczek-Khinchine predicts.

The data come from a lognormal distribution. The test assesses whether a normal distribution is also consistent with the data.

  1. Fifty points. With σ = 0.5 and 50 observations, press "Repeat 1,000 times". The normal is accepted in nearly 90% of samples. The test is not in error; it lacks the power to detect the skewness with so few observations.
  2. More data. Raise the sample to 200: acceptance falls to about 10%. At 1,000, the normal is essentially never accepted.
  3. Lower skewness. Set σ = 0.25. Even with 200 observations the normal is accepted almost 90% of the time. A mildly skewed distribution and a symmetric model are indistinguishable until hundreds of observations are available, and the difference lies in the tails, which generate a simulation's rare events.
  1. Exponential versus triangular. At 80% utilisation, the exponential service gives about 4.0 in the system and the triangular about 2.55, each simulation landing on its formula value.
  2. Constant and highly variable service. Constant service gives about 2.4 and the highly variable mix with c2=4c^2 = 4 (about 8.8). With the same mean and utilisation, the two differ by a factor of almost four.
  3. High utilisation. Set ρ to 95%. All the curves rise steeply, and the differences between them grow faster. Variability has the largest effect when the system is heavily loaded.

Derivation of the queue formula. In a single-server queue with Poisson arrivals at rate λ\lambda and service time SS, an arriving customer finds on average LqL_q customers waiting and, with probability ρ=λE[S]\rho = \lambda E[S], one in service whose remaining service time has mean E[S2]/(2E[S])E[S^2]/(2E[S]) (a long service is more likely to be in progress at an arrival). So the mean wait is Wq=LqE[S]+ρ E[S2]/(2E[S])W_q = L_q E[S] + \rho\, E[S^2]/(2E[S]). Little's law gives Lq=λWqL_q = \lambda W_q, and solving,

Wq=λE[S2]2(1−ρ)=ρ E[S](1+cs2)2(1−ρ), W_q = \frac{\lambda E[S^2]}{2(1-\rho)} = \frac{\rho\, E[S](1 + c_s^2)}{2(1-\rho)},

using E[S2]=E[S]2(1+cs2)E[S^2] = E[S]^2(1 + c_s^2). Then L=ρ+λWqL = \rho + \lambda W_q gives the formula. The variance enters only through cs2c_s^2, which is why two service distributions with the same mean behave so differently. For the triangular (0.25,1,1.75)(0.25, 1, 1.75), the variance is (a2+b2+c2−ab−ac−bc)/18=0.09375(a^2+b^2+c^2-ab-ac-bc)/18 = 0.09375, so cs2=0.094c_s^2 = 0.094.

In each experiment the average was not enough: the tail of a distribution, the asymmetry of costs and the variance of service times changed the answer. Recommendations for simulation studies:

  • Choose input families for physical reasons, then check their tails as well as the p-value.
  • Fifty observations constrain the middle of a distribution and say little about its tails.
  • Report results for more than one plausible input family when the data cannot choose between them.

Decisions often depend on parts of a distribution that the mean does not describe, as discussed in the γ-order generalized normal distribution.

References

  1. F. Pollaczek, "Über eine Aufgabe der Wahrscheinlichkeitstheorie", Mathematische Zeitschrift, 32, 64-100, 1930.
  2. A. Y. Khinchine, "Mathematical theory of a stationary queue", Matematicheskii Sbornik, 39(4), 73-84, 1932.
  3. A. N. Kolmogorov, "Sulla determinazione empirica di una legge di distribuzione", Giornale dell'Istituto Italiano degli Attuari, 4, 83-91, 1933.
  4. N. Smirnov, "Table for estimating the goodness of fit of empirical distributions", Annals of Mathematical Statistics, 19(2), 279-281, 1948.
  5. H. W. Lilliefors, "On the Kolmogorov-Smirnov test for normality with mean and variance unknown", Journal of the American Statistical Association, 62(318), 399-402, 1967.
  6. B. Biller and B. L. Nelson, "Answers to the top ten input modeling questions", Proceedings of the 2002 Winter Simulation Conference, 35-40, 2002.
  7. A. M. Law, "How to select simulation input probability distributions", Proceedings of the 2011 Winter Simulation Conference, 1389-1402, 2011.