Overview of the Tolerance Limit Calculations with Application to TSURFER

Abstract

To establish confidence in the results of computerized physics models, a key regulatory requirement is to develop a scientifically defendable process. The methods employed for confidence, characterization, and consolidation, or C³, are statistically involved and are often accessible only to avid statisticians. This manuscript serves as a pedagogical presentation of the C³ process to all stakeholders—including researchers, industrial practitioners, and regulators—to impart an intuitive understanding of the key concepts and mathematical methods entailed by C³. The primary focus is on calculation of tolerance limits, which is the overall goal of the C³ process. Tolerance limits encode the confidence in the calculation results as communicated to the regulator. Understanding the C³ process is especially critical today, as the nuclear industry is considering more innovative ways to assess new technologies, including new reactor and fuel concepts, via an integrated approach that optimally combines modeling and simulation and minimal targeted validation experiments. This manuscript employs intuitive, analytical, numerical, and visual representations to explain how tolerance limits may be calculated for a wide range of configurations, and it also describes how their values may be interpreted. Various verification tests have been developed to test the calculated tolerance limits and to help delineate their values. The manuscript demonstrates the calculation of tolerance limits for TSURFER, a computer code developed by the Oak Ridge National Laboratory for criticality safety applications. The goal is to evaluate the tolerance limit for TSURFER-determined criticality biases to support the determination of upper, subcritical limits for regulatory purposes.

1. Introduction and Mathematical Background

To establish confidence in the results of computerized physics models, a key regulatory requirement is to develop a scientifically defendable process, known as model validation. To ensure that code predictions can be trusted for a given application, such as the domain envisaged for code use, the regulatory process requires the consolidation of two independent sources of knowledge: (1) measurements collected from experimental conditions similar to those of the application, and (2) code predictions that model the same experimental conditions. This is equivalent to the idea of cross-referencing knowledge obtained from independent sources. The idea is that the consolidated knowledge will provide a higher level of confidence than the knowledge obtained from a single source. Drawing upon an example from criticality safety, which is the focus of this work, the analyst is interested in predicting a response, such as k_eff, for an arrangement of fuel assemblies in a shipping cask, serving as a representative application. The two sources of knowledge are (1) prior code predictions and (2) measurements for a number of critical experiments containing fuel similar in composition, geometry, and arrangement to that of the application. The premise is that, while both the code-calculated and experimentally measured responses have unavoidable uncertainties, the consolidation process could produce a better estimate of k_eff with a higher level of confidence than an estimate obtained with the experiments or code predictions alone. This process also demonstrates the applicability of the computational method to real-world measurements. Validation to real-world measurements is required by consensus standards and regulation promulgated by the US Nuclear Regulatory Commission (NRC).

This approach represents the core principle behind model validation, and it is designed to establish confidence in code predictions. Details on how the confidence in each knowledge source is characterized and how the knowledge consolidation process works are not prescribed by the regulators. Instead, the regulation process mandates a minimum level of confidence that the licensee must demonstrate in their code predictions, such as the 95%/95% criterion often employed by nuclear practitioners. This leaves enough freedom to the licensees to design their own technical basis for confidence, characterization, and consolidation, or C³.

Many statistical methods have been developed over the years to describe the C³ process: see for example Fisher 1950 [1], Hotel 1971 [2], Jaynes 1976 [3], Samaniego 2010 [4], Stanton 2017 [5], Bolstad 2017 [6], Bishop 2006 [7], Wakefield 2013 [8], and Thornber 1965 [9]. The theory has appeared in many textbooks and is presented from two distinct, heavily debated viewpoints—the frequentist and Bayesian statistics—with various renditions, such as parametric, nonparametric, and order-statistics. Despite the striking differences between the two viewpoints and their associated analysis methods, the final results could look very similar, making it very difficult for practitioners to see the value of all the theoretical subtleties. This work maintains that this classical presentation diminishes the value to engineering practitioners, who need an intuitive understanding of the how the C³ process works, thus allowing for a transparent approach through which the values of different methods may be assessed. This manuscript presents an intuitive exposition of the C³ process and its relevance to criticality safety problems. The presentation is supported by basic mathematical/statistical principles which are accessible to most engineering practitioners, as well as graphical aids delineating the process by which confidence can be established.

This presentation supports the development of a tolerance limit for bias calculations performed by ORNL’s TSURFER code, which is used for criticality safety [10]. Currently, TSURFER only calculates bias and bias uncertainty. The objective of this work is to evaluate the tolerance limit for the calculated bias to support determination of upper subcritical limits for regulatory purposes. In support of criticality safety analysis licensing, the regulator requires development of a confidence characterization and consolidation process (C³) when calculating quantities of interest, such as k_eff, for a given application condition. The consolidation process is mandated to develop confidence collected from two independent sources: physics-based calculations and experimental measurements. In practice, it is not feasible to experiment with every application condition, so a method is needed that can leverage similar experiments to complete the C³ process. The TSURFER methodology attempts to achieve this goal by consolidating experimental measurements and calculational results from a number of experiments that are judged to be sufficiently similar to the application; this is accomplished by using a Bayesian-based generalized least-squares methodology [10]. The measurements and calculations for the experimental conditions are consolidated by identifying a common source of uncertainty across all of the experiments and the target application. In the TSURFER methodology, this common source is the nuclear cross sections, which are adjusted by TSURFER to improve the agreement between calculations and measurements. The adjusted cross sections serve as a basis for updating the value of k_eff for the application (while the generalized least-squares methodology is based on the concept of cross section adjustments, TSURFER is able to directly calculate the impact on the response of interest in the form of calculational biases without having to store the adjusted cross sections.) The difference between the prior and adjusted k_eff values is referred to as a bias.

To establish confidence in the calculated bias, TSURFER also calculates the standard deviation of the bias assuming the bias uncertainty is described by a normal distribution. In this effort, TSURFER was augmented with another measure of confidence, denoted by the tolerance limit [11,12], an upper threshold that is a function of the mean value of the bias and its standard deviation in the form μ_t + K𝜎_t. The K is a coverage parameter calculated to ensure that there is a high probability that the true bias remains below the upper threshold value. This supports the calculation of the upper subcritical limit (USL) [13]. The regulator enforces the use of USL to ensure, with high confidence, that the true value of k_eff for the given application remains upper-bounded. To support this goal, TSURFER must be able to calculate an upper threshold limit for the bias, also with high confidence. Typically, US NRC requires that there is 95% confidence that at least 95% of the population of bias values are covered.

Demonstration of this specific requirement has been an obstacle to implementation of the TSURFER methodology in the United States since its initial release. Since TSURFER represents a specific rendition of the C³ process, it is important to first develop a general understanding of the value, interpretation, and calculational methods for tolerance intervals. This is important to establish before navigating through the various methods appearing in the literature, so that tolerance calculations from both the frequentist and Bayesian viewpoints can be managed. This manuscript is not intended to support or criticize the TSURFER methodology; instead, it provides clear guidance to practitioners on how to select a specific tolerance calculational method, as demonstrated herein, for the TSURFER methodology.

The C³ process does not provide any details on the technical method used to mathematically characterize confidence nor does it provide instructions on how confidence can be algorithmically consolidated from multiple sources. To make such decisions, one must first determine the type of uncertainties present in the problem. The presence of uncertainty implies that one cannot make a deterministic statement about the outcome of an experiment: that is, one cannot say with 100% certainty that k_eff is equal to 1.0. This results from two distinct types of uncertainty. The first type of uncertainty results from the inability to control every aspect of an experiment. For example, one cannot exactly know, with 100% confidence, the geometry, the composition, and other experimental conditions. If employing a code, the same situation occurs wherein the lack of knowledge—regarding the physics model itself and/or its associated input parameters—contributes to the overall uncertainty. The second type of uncertainty results from the physical nature of the quantity being measured, which may not assume a single value due to its inherent randomness. The first type of uncertainty is referred to as epistemic or Bayesian uncertainty, and the second type is associated with more conventional frequency-based interpretation of uncertainty, referred to as aleatory uncertainty. Both these types of uncertainties are mathematically described using probability density functions (PDFs), albeit with different interpretations and different consolidation approaches. For example, as explained below, aleatory uncertainties are consolidated via averaging of their PDFs, whereas epistemic uncertainties, interpreted as degree-of-belief, are consolidated via PDFs multiplication.

Despite the striking differences in the interpretations of these two types of uncertainties and their markedly different consolidation approaches, their final results, including confidence intervals, credible intervals, or tolerance intervals, often look very similar, with only subtle variations that are heavily debated by statisticians, while seeming mostly of unclear value to practitioners [3]. Statements like the following are typically made about a response such as k_eff for a given application, for example, a shipping cask containing nuclear fuel: With 95% confidence, the probability that k_eff will not exceed the stated threshold is 95%. Ideally, a straightforward interpretation of this statement would read as follows: there is 95% chance that the true value of k_eff will not exceed the stated threshold. However, this simple interpretation is not 100% accurate; it is only 95% accurate. In layman terms, this is equivalent to stating: You certainly need to get a college education to succeed in life, but not necessarily, or with no surprises, the surgery will be 99% safe. Intuitive interpretation of these statements is troubling because they provide conflicting, ambiguous, probabilistic information. On the one hand, they make a positive probabilistic assertion for the outcome, but on the other hand, they hedge, thus reducing the confidence in that assurance. In other words, they hedge twice, once by employing a probabilistic rather than deterministic assertion to describe the outcome, and then they hedge a second time against that very assertion. It is natural to ask, why cannot one hedge only once, resulting in a much simpler/intuitive interpretation? The goal of this work is to explain that this double-hedging strategy is a result of the theoretical methods employed to describe confidence. While this strategy affords mathematical convenience, it entangles the meanings of aleatory and epistemic uncertainty. However, simple interpretation can still be achieved using basic arguments and graphical aids, which is one of the key contributions of this work. Another goal is to explain how both epistemic and aleatory uncertainties are essential in establishing confidence. In many textbooks, the two viewpoints are discussed separately, as independent tracks in statistics [6]. However, all practical problems contain a mix of epistemic and aleatory uncertainties, necessitating a flexible approach that can transition seamlessly between the two viewpoints.

Methods used to determine the tolerance limits (tolerance intervals) are based on order statistics as defined by Wilks [14,15], in which the distribution of the proportion of the population between two-order statistics is independent of the sample population. Wald [16] extends Wilks’ method and constructs the tolerance limits for normally distributed data. Somerville tabularized the tolerance limits for nonparametric tolerance limits to minimize the interpolation, and Patel [17] discusses the tolerance limits for various distributions in both parametric and nonparametric methods. In nuclear engineering, tolerance limits attracted attention because the US NRC’s licensing process switched to best-estimate methods plus uncertainty (BEPU) analyses in 1988 [18]. Tolerance limits were introduced to nuclear systems by Pál [19] and Guba [20] for the application of safety analyses. They have been incorporated within the framework of standard uncertainty analysis methods like the GRS method [21], thus serving as measures of confidence. Many papers in nuclear literature apply tolerance limits and propose suitable data treatment in their specific studies. For example, nonparametric tolerance limits are applied on quantities like fission gas release for spent fuel assembly [22]. One- and two-sided tolerance limits are used to improve the precision of measurement uncertainty for engineering systems and are applied on a contaminant transport model [23]. The distribution-free Wilks tolerance limit is used to evaluate the nuclear data uncertainty impact on power and reactivity in rod-withdrawal transients for pressurized water reactor (PWR) mini-cores when the output is time dependent and its distribution varies across time [24]. Porter [25] provides a comprehensive discussion on the nonparametric tolerance limits, including derivations, statistical properties, application to computational tools, requirements imposed on the applied code, verification process, and higher-rank Wilks formula. The tolerance limits are tested with various orders to obtain recommendations on safety audit analysis for energy systems [26]. The Wilks formula is used to evaluate uncertainty quantification (UQ) results on a TRACE boiling water reactor (BWR) model to simulate a spray cooling experiment [27]. The upper tolerance limits of parametric and nonparametric methods under mixture of normal distributions for nuclear power plant safety (e.g., peak cladding temperature in accident scenario) are derived and calculated [28]. Wilks’ method was compared to the Monte Carlo method for performing UQ on loss-of-coolant-accident (LOCA) application [29] and was found to be conservative and flexible at the 95/95 tolerance. Shockling [30] evaluated the number of thermal-hydraulics calculations required to meet the safety limits using nonparametric order statistics on peak cladding temperature. Suggestions on the order and number of random samples needed in the use of Wilks’ formula for more appropriate nuclear safety applications are provided in the Nuclear Energy Agency Committee on the Safety of Nuclear Installations workshop proceedings from 2013 [31] and based on the authors’ in-depth understanding. The appropriate quantitative “high level of probability”, from a regulatory point of view, are explained in Martin and Nutt’s article from 2011 [32], in which different assumptions and limitations of acceptance criteria are reviewed, and recommendations for LOCA applications regarding univariate and multivariate situations for UQ with order statistics are provided.

To set the stage for the C³ process, the reader is strongly recommended to visit Appendix A before proceeding to the next section. Appendix A starts with a few preliminary definitions on two key concepts in probability theory—the frequency of occurrence and the degree of belief—both of which are dubbed as probability and described by PDFs. The goal is to show that despite the differences, which arguably appear to be purely philosophical to practitioners, the combined use of Bayesian and frequentist statistics is essential to establish/consolidate confidence in most practical problems. The presentation in Appendix A is meant to be pedagogical, to allow for a transparent dialogue about the effectiveness and value of various C³ methods. For experienced statisticians, Appendix A may be skipped. For other readers, it is strongly recommended as it describes in detail the basic principles of the C³ process which is essential before proceeding to the TSURFER application in the next section.

2. TSURFER Tolerance Limit Development

This section demonstrates how the upper threshold for a 95%/95% tolerance interval may be calculated for the TSURFER methodology. The goal is to ensure that the TSURFER bias will not result in a k_eff value that exceeds the upper subcritical limit, as illustrated in Figure 1.

The reference value for k_eff prior to the application of TSURFER is marked by the blue horizontal line $k_{\mathrm{app}}^{(\mathrm{cal})}$. The bias $k_{}^{(\mathrm{app})}$ calculated by TSURFER is marked by the double-pointing arrow, bringing the best-estimate of k_eff to the green line. Since the TSURFER methodology is statistical, the confidence in the bias estimate is described by an epistemic PDF, as marked by the blue curve. This PDF is assumed to be normal in the TSURFER methodology. The standard deviation ${\sigma }_{{}_{\Delta k^{(\mathrm{app})}}}$ of this PDF is calculated by TSURFER, as marked by the two double-pointing orange arrows around the best-estimate adjusted value. The upper threshold of the tolerance interval is marked by the red line, so the area underneath the PDF is at least 0.95, implying with 95% confidence that the true value for the bias will ensure a k_eff value below the upper threshold limit. The interval upper-limited by the red line (marked by a solid red bar) is the tolerance interval for TSURFER’s bias. Finally, an additional administrative margin is added to reach the USL [13].

2.1. TSURFER’s Problem Definition

With the TSURFER theory well-documented [10], a simplified rendition is presented below, consistent with the mathematical treatment provided in Appendix A. Effectively, TSURFER applies a customized rendition of the C³ process to update knowledge about the target application for which no experimental data exist using two sources of knowledge—the application’s model-predicted k_eff, given by a prior PDF of the form:

$$p_{\mathrm{app}}^{(\mathrm{cal})}\left(k_t^{(\mathrm{app})}|k_{\mathrm{app}}^{(\mathrm{cal})}\right)\propto e^{-\frac{{\left(k_{\mathrm{app}}^{(\mathrm{cal})}-k_t^{(\mathrm{app})}\right)}^2}{2{\sigma }_t^{(\mathrm{app},\mathrm{cal})2}}},$$

(1)

and the measurements from N experiments,

$$p_i^{(\exp )}\left(k_t^{(i)}|k_i^{(\exp )}\right)\propto e^{-\frac{{\left(k_i^{(\exp )}-k_t^{(i)}\right)}^2}{2{\sigma }_t^{({\exp }_i)2}}},i=1,2,\dots ,N,$$

(2)

consolidated with their associated model-predicted values,

$$p_i^{(\mathrm{cal})}\left(k_t^{(i)}|k_i^{(\mathrm{cal})}\right)\propto e^{-\frac{{\left(k_i^{(\mathrm{cal})}-k_t^{(i)}\right)}^2}{2{\sigma }_t^{({\exp }_i,\mathrm{cal})2}}},i=1,2,\dots ,N.$$

(3)

The distributions in Equations (1) and (3) are centered around the reference prior k_eff values of $k_i^{(\mathrm{cal})}{|}_{i=1}^N$ and $k_{\mathrm{app}}^{(\mathrm{cal})}$, which were calculated using the reference values for the cross sections, leaving the true values, $k_t^{(i)}{|}_{i=1}^N$ and $k_t^{(\mathrm{app})}$, as unknown in the epistemic sense. Notwithstanding the aleatory nature of the interaction between a neutron and a target nucleus, the epistemic treatment is more adequate because the range of cross section variations that result from the random nature of the neutron-nucleus interaction is much smaller in magnitude than the uncertainties associated with the cross-section measurement and subsequent evaluation procedures. This situation is depicted in Figure A1 in the Appendix A, where the narrow PDF represents the true aleatory spread of the cross-section values, whereas the wider PDF represents the degree-of-belief about the mean values of the aleatory PDF.

All three sets of distributions in Equations (1)–(3) assume the normal shape because only the mean and standard deviations are employed to construct the PDFs. This approach was used based on a famous statistical result [33] stating that the least-informative PDF about a parameter is the normal distribution when the mean value and standard deviations are the only known features about the distribution. The true standard deviation of the calculated k_eff value for the N experiments—${\sigma }_t^{({\exp }_i,\mathrm{cal})}{|}_{i=1}^N$ and the application, ${\sigma }_t^{(\mathrm{app},\mathrm{cal})}$—are obtained by propagating the cross-sectional uncertainties from Evaluated Nuclear Data File (ENDF) libraries, which are available in the form of covariance information [34]. Arguably, the propagation process may result in non-normal distributions for the calculated k_eff values. Assessing the adequacy of the normality assumption is outside the scope of this work. If indeed the prior distributions given in Equations (1) and (3) considerably deviate from the normal shape, then nonparametric methods are needed to determine the tolerance limits. Nonparametric methods do not make assumptions about the shape of the true PDF, so the shapes of the sampling distributions (e.g., normal distribution, χ²-distribution, t-distribution) cannot be evaluated analytically. Instead, sampling techniques that emulate the numerical results described in Appendix A.7 are performed to numerically integrate the resulting joint sampling distributions of the mean and standard deviations. However, this discussion is outside the scope of this study, which is limited to the assumptions made by TSURFER, whose justification may be found in earlier publications by TSURFER developers [10].

The measurement PDFs in Equation (2) are centered around the measured values, which are assumed to be a single measurement per experiment—$k_i^{(\exp )}{|}_{i=1}^N$ leaving the true k_eff value as the unknown. The true standard deviations ${\sigma }_t^{({\exp }_i)}{|}_{i=1}^N$ are assumed to be known which greatly simplifies the calculation of the tolerance limit as will be shown later. The standard deviations are often based on conservative estimates of the various sources of uncertainties that are believed to contaminate the experimental model, including composition, geometry, and experimental conditions, often referred collectively to as the benchmark uncertainties, where a benchmark denotes an experiment. This uncertainty can also be inflated to account for statistical uncertainties resulting from Monte Carlo simulation.

The mathematical procedure used to combine these uncertainties is selected by the experimentalist (the quadrature approach is often considered adequate to combine these uncertainties since they tend to be independent of each other; this discussion is, however, considered outside the scope of this work) which is typically unchallenged and may not be well documented. Essentially, it is left to the experimentalist to decide on the best approach to capture and combine the different types of uncertainties, often employing conservativism. When performing downstream analysis (e.g., TSURFER), the analyst employing these uncertainties must decide whether these uncertainties are aleatory or epistemic. TSURFER assumes that the reported experimental standard deviations are the true values and that the associated PDF is normal per Equation (2). Furthermore, it assumes that the PDF is epistemic and thus employs the concept of PDF multiplication for consolidation with the code-calculated PDFs shown in Equation (3). As shown below, the choice of epistemic treatment is straightforward for TSURFER because the true standard deviations for the consolidated PDFs are assumed to be known. With more explicit treatment of the uncertainties from the experimental procedure, the choice between aleatory and epistemic treatment would become more relevant.

2.2. TSURFER’s C³ Process

In the above formulation, the number of parameters for which knowledge is to be updated is equal to N + 1, representing the k_eff values for the N experiments $k_t^{(i)}{|}_{i=1}^N$ and the one application $k_t^{(\mathrm{app})}$. Each experiment has two sources of knowledge: one from the experimental measurements $k_i^{(\exp )}{|}_{i=1}^N$, and one from prior code calculations $k_i^{(\mathrm{cal})}{|}_{i=1}^N$, and, finally, the target application has a single knowledge source from code calculations $k_{\mathrm{app}}^{(\mathrm{cal})}{|}_{i=1}^N$. A straightforward application of the C³ process, as outlined in Appendix A.4, only allows for updating knowledge about the true value for each experiment $k_t^{(i)}{|}_{i=1}^N$ independently of all other experiments, as obtained by consolidating the two respective PDFs for the ith experiment in Equations (2) and (3), as follows:

$$p_i^{(\mathrm{con})}\left(k_t^{(i)}\right)\propto e^{-\frac{{\left(k_i^{(\exp )}-k_t^{(i)}\right)}^2}{2{\sigma }_t^{({\exp }_i)2}}-\frac{{\left(k_i^{(\mathrm{cal})}-k_t^{(i)}\right)}^2}{2{\sigma }_t^{({\exp }_i,\mathrm{cal})2}}}\propto e^{-\frac{{\left(k_i^{(\mathrm{con})}-k_t^{(i)}\right)}^2}{2{\sigma }_t^{({\mathrm{con}}_i)2}}},$$

where the respective mean values and the standard deviations of the consolidated PDFs are given by

$$k_i^{(\mathrm{con})}=\frac{\frac{k_i^{(\mathrm{cal})}}{{\sigma }_t^{2({\exp }_i,\mathrm{cal})}}+\frac{k_i^{(\exp )}}{{\sigma }_t^{2({\exp }_i)}}}{\frac{1}{{\sigma }_t^{2({\exp }_i,\mathrm{cal})}}+\frac{1}{{\sigma }_t^{2({\exp }_i)}}},$$

$$\frac{1}{{\sigma }_t^{({\mathrm{con}}_i)}}=\frac{1}{{\sigma }_t^{2({\exp }_i,\mathrm{cal})}}+\frac{1}{{\sigma }_t^{2({\exp }_i)}},$$

where the superscript con refers to the consolidated values. This brute-force application of the C³ process is not interesting, because it does not exploit the similarity between the experiments, and more importantly, it does not provide a way to update the application’s PDF in Equation (1). If the PDFs in Equations (1)–(3) are multiplied directly, then 2N + 1 would be unknown, and all PDFs would be treated as independent of each other, hence, providing no real value from consolidation. To overcome this, the unknown epistemic parameters must be common to all PDFs to realize the value of PDF multiplication. TSURFER achieves that by assuming the following:

$$k_t^{(\mathrm{app})}=k_{\mathrm{app}}^{(\mathrm{cal})}+{\stackrel{\rightharpoonup }{\theta }}_{\mathrm{app}}^T{\stackrel{\rightharpoonup }{\delta }}_t+\epsilon ,$$

(4)

$$k_t^{(i)}=k_i^{(\mathrm{cal})}+{\stackrel{\rightharpoonup }{\theta }}_i^T{\stackrel{\rightharpoonup }{\delta }}_t+\epsilon ,i=1,2,\dots ,N.$$

(5)

These N + 1 equations imply that the true k_eff values for the N experiments and the one application all depend linearly on a common set of n epistemic parameters represented by a vector ${\overrightarrow{\delta }}_t^T\in R^n=[\begin{array}{cccc}{\delta }_{t1}& {\delta }_{t2}& \cdots & {\delta }_{tn}\end{array}]$, where t denotes the unknown true values. These epistemic parameters are selected in TSURFER to be the multigroup nuclear cross sections, and the coefficients of linearity, represented by the components of the vectors ${\overrightarrow{\theta }}_i$ and ${\overrightarrow{\theta }}_{\mathrm{app}}$, are precalculated using a sensitivity (parametric) analysis, which estimates the first-order derivative of the calculated k_eff with respect to each cross section.

It is important to note here that while these equations allow for application of the C³ process, they are based on two key assumptions that require proper validation: (a) cross sections represent the key sources of the observed discrepancies between calculated and measured k_eff values, and (b) the unexplained errors ε have no structure and are purely aleatory, so they can be reduced with repeated measurements or many experiments. In reality, these unknown errors are likely a combination of epistemic and aleatory uncertainties, with the epistemic uncertainties resulting from other parameters not captured by the physics model, and with the aleatory uncertainties resulting from uncontrolled experimental conditions. With epistemic uncertainties present but unaccounted for, the error compensation phenomenon is likely to happen. Thus, analysis is required to ensure that these assumptions are adequate. However, this is outside the scope of the current work.

With a source of epistemic uncertainties that is common to all experiments, the C³ process can consolidate all the PDFs in Equations (2) and (3) into a single updated PDF for the cross-sectional adjustments. Since the cross sections are described by a vector, the resulting consolidated PDF—$p\left({\stackrel{\rightharpoonup }{\delta }}_t|k_i^{(\exp )}{|}_{i=1}^N,k_i^{(\mathrm{cal})}{|}_{i=1}^N\right)$—is in the form of a multi-dimensional normal distribution,

$$p\left({\stackrel{\rightharpoonup }{\delta }}_t|k_i^{(\exp )}{|}_{i=1}^N,k_i^{(\mathrm{cal})}{|}_{i=1}^N\right)\propto e^{-\frac{1}{2}{\left({\stackrel{\rightharpoonup }{\delta }}_t-{\stackrel{\rightharpoonup }{\delta }}_m\right)}^T{C}_m^{(\delta )-1}\left({\stackrel{\rightharpoonup }{\delta }}_t-{\stackrel{\rightharpoonup }{\delta }}_m\right)}.$$

(6)

Although it is more complicated in form than the simpler single-variable PDFs such as Equation (A15), it can be shown that this PDF is centered around a vector of mean values ${\overrightarrow{\delta }}_m$, and the spread (i.e., as measured by the variance or standard deviation) of the PDF along the n dimensions is described by a squared n × n covariance matrix $C_m^{(\delta )}$, both of which can be analytically determined. Expressions for ${\overrightarrow{\delta }}_m$ and $C_m^{(\delta )}$ may be found in the TSURFER literature [10]. To support the pursuit for the tolerance interval, simpler expressions are presented here for ${\overrightarrow{\delta }}_m$ which may be written as:

$${\stackrel{\rightharpoonup }{\delta }}_m={\rm B}\left[\begin{array}{c}{k}_1^{(\exp )}-k_1^{(\mathrm{cal})}\\ k_2^{(\exp )}-k_2^{(\mathrm{cal})}\\ \dots \\ k_N^{(\exp )}-k_N^{(\mathrm{cal})}\end{array}\right]={\rm B}\left({\stackrel{\rightharpoonup }{k}}^{(\exp )}-{\stackrel{\rightharpoonup }{k}}^{(\mathrm{cal})}\right).$$

This equation emulates the TSURFER final expression for ${\overrightarrow{\delta }}_m$ by lumping all the mathematical operators into one matrix operator B, operating on a vector describing the discrepancies between the measured and reference calculated k_eff values for the N experiments. Combing this equation with Equation (4), one can update the epistemic PDF for the application k_eff value $k_t^{(\mathrm{app})}$, which is also expected to be a normal distribution, with mean value given by

$$k_m^{(\mathrm{app})}=k_{\mathrm{app}}^{(\mathrm{cal})}+{\stackrel{\rightharpoonup }{\theta }}_{\mathrm{app}}^T{\stackrel{\rightharpoonup }{\delta }}_m+\epsilon =k_{\mathrm{app}}^{(\mathrm{cal})}+{\stackrel{\rightharpoonup }{\theta }}_{\mathrm{app}}^T{\rm B}\left({\stackrel{\rightharpoonup }{k}}^{(\exp )}-{\stackrel{\rightharpoonup }{k}}^{(\mathrm{cal})}\right).$$

Rewriting this equation with the substitution $\overrightarrow{\lambda }={\stackrel{\rightharpoonup }{\theta }}_{\mathrm{app}}^T{\rm B}$ results in

$$k_m^{(\mathrm{app})}-k_{\mathrm{app}}^{(\mathrm{cal})}={\displaystyle \sum _{i=1}^N{\lambda }_i\left(k_i^{(\exp )}-k_i^{(\mathrm{cal})}\right)}.$$

Since the cross-sections PDF in Equation (6) is normal, and $k_t^{(\mathrm{app})}$ is assumed to depend linearly on ${\overrightarrow{\delta }}_t$ per Equation (4), the resulting PDF for $k_t^{(\mathrm{app})}$ will also be normal, with mean value and variance that are analytically determined. In doing so, the standard deviation of the consolidated PDF is an analytic function of the true experimental standard deviation and the prior standard deviation, implying that it represents the true (not an estimate) standard deviation of the consolidated PDF for $k_t^{(\mathrm{app})}$. This greatly simplifies the calculation of the tolerance limit as compared to the general case when both the mean and standard deviations are not known and have to be estimated from the consolidation analysis (see Appendix A.7). Rewriting the equation above in the form of deviations (i.e., biases) from the prior calculated values results in

$$\Delta k^{(\mathrm{app})}={\displaystyle \sum _{i=1}^N{\lambda }_i\Delta k_i^{(\exp )}}={\stackrel{\rightharpoonup }{\lambda }}^T\Delta {\stackrel{\rightharpoonup }{k}}_{}^{(\exp )}.$$

(7)

This equation describes a linear relationship between the initial experimental biases for the N experiments and the TSURFER calculated bias for the application, representing the change in the location of the mean k_eff value for the application. The goal is to establish a tolerance interval for the application bias. Since the standard deviations for both the experimental measurements and reference calculated values are known, the true standard deviation for the application bias can be calculated as

$${\sigma }_{\Delta k^{(\mathrm{app})}}=\sqrt{{\displaystyle \sum _{i=1}^N{\lambda }_i^2\left({\sigma }_t^{({\exp }_i)2}+{\sigma }_t^{({\exp }_i,\mathrm{cal})2}\right)}}.$$

This equation strongly emulates the derivation of the sample standard deviation using the sample mean, as discussed at the beginning of Appendix A.7, specifically in Equation (A23). Unlike Equation (A23), the weights are not unity. By analogy, the ∆$k_t^{(\mathrm{app})}$ may be thought of as a feature derived from the N PDFs representing the N experiments, each contributing a single sample. The standard deviation of the estimated quantity ∆$k_t^{(\mathrm{app})}$ may be calculated by first projecting the samples along the unit vector:

$${\stackrel{\rightharpoonup }{e}}_N={\left[\begin{array}{cccc}{\lambda }_1& {\lambda }_2& \dots & {\lambda }_N\end{array}\right]}^T/\sqrt{{\displaystyle \sum _{i=1}^N{\lambda }_i^2}},$$

and calculating the residual ${\mathrm{P}}_e\Delta {\stackrel{\rightharpoonup }{k}}_{}^{(\exp )}$ and the estimated standard deviation as

$$s_{\Delta k^{(\mathrm{app})}}=\sqrt{\frac{{‖{\mathrm{P}}_e\Delta {\stackrel{\rightharpoonup }{k}}_{}^{(\exp )}‖}_2^2}{N-1}}.$$

(8)

Since the true standard deviation of the bias can be calculated analytically, the above quantity may simply be used for verification purposes by checking its likelihood from the χ²-distribution. Specifically, per the discussion in the Appendix A.7, the following quantity $(N-1)s_{\Delta k^{(\mathrm{app})}}^2/{\sigma }_{\Delta k^{(\mathrm{app})}}^2$ should have a χ²-distribution with N − 1 degrees of freedom. If the calculated value is significantly different from the analytically calculated value, then it serves as a measure of the adequacy of the assumptions given in Equation (5). If the standard deviation of the experiment is assumed to be unknown, then the above quantity provides an approach for estimating a realistic estimate of the experimental procedure standard deviation. However, this is outside the scope of the current work.

As noted in Appendix A, the determination of the tolerance interval for normally distributed quantities is largely dependent on what the unknowns are, i.e., the mean and/or the standard deviation. When both the mean and standard deviation are unknown, then the treatment in Appendix A.7 must be used, which requires the use of the noncentral t-distribution. When only the mean is unknown, then the tolerance interval can be evaluated directly using the normal distribution (see Appendix A.5). Finally, when only the standard deviation is unknown, the χ²-distribution must be used (see Appendix A.6). Recall that TSURFER analysis assumes that the standard deviations of the experimental measurement and the calculated k_eff value are both known. These assumptions, coupled with the linearity assumption in Equations (4) and (5), imply that the standard deviation of the bias is also known: that is, it can be analytically calculated. Furthermore, since only a single measurement is available from each experiment, the implication is that the mean value is unknown, so the analysis in Appendix A.5 is the most suitable for estimating the tolerance interval for the application bias.

As shown in that discussion, equivalent results may be obtained with either the epistemic or aleatory treatment, simply because the true standard deviation of the parameter is assumed to be known. However, if the measurement procedure results in non-normally distributed uncertainties, or if it lacks a credible estimate of the standard deviation, then the assumption of a known standard deviation is no longer applicable, so the methods in Appendix A.7 must be used to determine the tolerance limits. Finally, when the reported standard deviation for the measurement is based on a conservative analysis, then the resulting tolerance intervals will be conservative, as well. Therefore, the explicit treatment of measurement uncertainties would be warranted only when the tolerance intervals for the associated applications biases are deemed unacceptably high.

Based on the TSURFER assumptions, a tolerance limit is given by (see Appendix A.5):

$$\Delta k_{\mathrm{K}-\mathrm{Tolerance}}^{(\mathrm{app})}=({\displaystyle \sum _{i=1}^N{\lambda }_i\Delta k_i^{(\exp )}})+K_{95}(1+\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\sqrt{N}$}\right.)\sqrt{{\displaystyle \sum _{i=1}^N{\lambda }_i}({\sigma }_t^{({\exp }_i)2}+{\sigma }_t^{({\exp }_i,\mathrm{cal})2})},$$

(9)

where K₉₅ is the coverage parameter calculated from a normal distribution, and $K_{95/95}=K_{95}(1+\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\sqrt{N}$}\right.)$ is the corrected value to compensate for the uncertainty in the estimate mean bias. If a 95%/95% tolerance limit is required, then a value of K₉₅ = 1.65 should be used.

3. Numerical Verification Results

This section introduces a number of numerical tests to validate the calculated tolerance interval. The objective is to employ a random sampling approach to calculate the distribution of biases, considering the uncertainties in the experimental measurements and the code’s prior calculations. The resulting distributions are numerically evaluated to calculate the upper threshold limit for the tolerance interval per the discussion in Appendix A. The distributions are compared to the values analytically calculated by Equation (9). These results are generated using a TSURFER model containing 17 critical experiments and 1 application. The experiments are all from the LEU-COMP-THERM-008 benchmark experiments set, denoted hereafter, LCT-008 [35]. The LCT-008 experiments are critical lattices with low-enriched (2~4 wt.%) UO₂ fuel rods and perturbing rods in a cylindrical, borated water tank. Examples of the experiments are depicted in a cross-sectional view in Figure 2, in which the fuel rods with aluminum alloy are presented in blue dots and the perturbing rods are shown in yellow surrounded by borated water, which is shown in green. Different critical experiments in the LCT-008 set include various perturbing rods materials and arrangements, or the boron concentrations are varied. For instance, Figure 2a,d presents two LCT-008 experiments without perturbing rods, whereas experiment LCT-008-016 contains a borated water slab in the center, of 1158-ppm boron, whereas the boron concentration in experiment LCT-008-001 was 1511 ppm.

Figure 2b,c show the layout of two experiments with different perturbing rod materials, in which both experiments have 3 × 3 central arrays of water holes and perturbing rods using Pyrex in LCT-008-005, whereas alumina is used in LCT-008-012. The application is an accident-tolerant fuel (ATF) assembly which employs a 10 × 10 BWR-dominant lattice of GE14 design, using UO₂ fuel pins with FeCrAl cladding. The layout of the application model is shown in Figure 3, with 92 fuel pins and two water rods in the center, surrounded by coolant water in a channel box. There should be 78 UO₂ fuel pins with enrichments ranging from 2~4%, and 14 rods with gadolinium.

Table 1. TSURFER k_eff Values.

Benchmark Experiment	Code-Prior Value	Measured Value	Adjusted by TSURFER
LEU-COMP-THERM-008-001	0.999830	1.00000	0.999694
LEU-COMP-THERM-008-002	1.000356	1.00000	1.000293
LEU-COMP-THERM-008-003	1.001015	1.00000	1.000947
LEU-COMP-THERM-008-004	1.000146	1.00000	1.000116
LEU-COMP-THERM-008-005	0.999817	1.00000	0.999773
LEU-COMP-THERM-008-006	1.000374	1.00000	1.000380
LEU-COMP-THERM-008-007	0.999652	1.00000	0.999657
LEU-COMP-THERM-008-008	0.998910	1.00000	0.999157
LEU-COMP-THERM-008-009	0.999103	1.00000	0.999334
LEU-COMP-THERM-008-010	0.999998	1.00000	0.999937
LEU-COMP-THERM-008-011	1.000675	1.00000	1.000614
LEU-COMP-THERM-008-012	1.000216	1.00000	1.000148
LEU-COMP-THERM-008-013	1.000363	1.00000	1.000292
LEU-COMP-THERM-008-014	1.000220	1.00000	1.000115
LEU-COMP-THERM-008-015	1.000051	1.00000	0.999962
LEU-COMP-THERM-008-016	1.000088	1.00000	1.000087
LEU-COMP-THERM-008-017	0.999352	1.00000	0.999487
B-GE14DOM-hf.v40-FeCrAl-1	0.933281	-	0.932013

lists the prior values for k_eff and the measured values, as well as the adjusted values resulting from the TSURFER application for all experiments and the application. Note that the application does not have a measured value. The adjusted values post-TSURFER represent the mean values of the PDFs generated by TSURFER. Sample results of the prior and post-TSURFER PDFs for a number of experiments and the application are shown in Figure 2.

In the discussion below, prior PDFs refer to the PDFs consolidated by TSURFER, including the code-based PDF, centered around the calculated value, and the measured PDF, centered around the measured value. For example, Figure 4 shows the code-based prior PDF (in red), the measured PDF in black, and the consolidated PDF in blue for LEU-COMP-THERM-008-008, as denoted by “Fused-Post”.

The x-axis is given in units of pcm, subtracting all k_eff values from the reference code-calculated prior values for the respective experiments. Notice the measured-prior PDF (in black) has a mean value slightly above zero, implying that the code underestimates the measured value, which is assumed to be 1.0 for all experiments. The spread of the measured-prior PDF is consistent with the measurement uncertainty, which is assumed to be 150 pcm for all experiments. The TSURFER-consolidated PDF (also referred to as post-PDF) has a narrower width than the code-prior PDF because the measurement uncertainty is much smaller than the code-calculated prior uncertainty (on the order of 500 pcm).

Figure 5 shows a similar plot but for the application, which does not have a measured-prior PDF. Although the mean value of the consolidated PDF did not change by much, the standard deviation has experienced a drop from 670 pcm as calculated from code-prior PDF (in red) down to 388 pcm (in blue). The drop is not as significant as in the case with the experiments. This is because there is a low similarity between the application and the experiments. The similarity index is a scalar quantity used by TSURFER to determine the relevance of experiments to the given application; its use is not required by the Bayesian methodology.

As mentioned in the discussion above, TSURFER can calculate all its quantities of interest, including the experimental biases and the application bias and their standard deviations, via an analytical procedure, based on two reasons: (a) the model relating the variations in k_eff to the cross sections is assumed to be linear, and (b) the standard deviations of the code-calculated k_eff and measurements are assumed to be known. The calculated standard deviations represent the basis for tolerance-limit evaluation, so the next series of numerical experiments is designed to estimate the mean value of the bias and the standard deviation using a sampling-based (i.e., aleatory) approach to test the adequacy of the estimated quantities for tolerance limits estimation.

Equation (7) represents the key equation employed by TSURFER to calculate the application bias as a function of the discrepancy between measured and code-calculated k_eff values for all available experiments. This equation calculates an estimator for the mean value of the TSURFER bias, so a sampling-based approach is employed, emulating the discussion in the Appendix A.5. To achieve this, N random samples are generated for the discrepancy term $\Delta {\overrightarrow{k}}_j^{(\exp )}$ by generating a random sample from the measured-prior PDF—Equation (2)—and a random sample from the code-prior PDF—Equation (3)—and averaging the N biases calculated from Equation (7), as follows:

$$\Delta k_{}^{(\mathrm{app})}=\frac{1}{N}{\displaystyle \sum _{i=1}^N{\stackrel{\rightharpoonup }{\lambda }}^T\Delta {\stackrel{\rightharpoonup }{k}}_j^{(\exp )}}.$$

This process is repeated 10,000 times, producing 10,000 estimates of the mean, each based on N samples of the discrepancy vector. Figure 6 shows the sampling distribution for the mean using different values of N. The blue PDF corresponds to N = 1, which is equivalent to the PDF generated by TSURFER, shown in red in Figure 5. The standard deviations for the various values of N are shown in the legend, thus confirming the $1/\sqrt{N}$ behavior for mean value estimation. Figure 7 shows the plots of the quantity $(N-1)s_{\Delta k^{(\mathrm{app})}}^2/{\sigma }_{\Delta k^{(\mathrm{app})}}^2$, which are found to follow a χ² distribution, with 16 degrees of freedom, as expected from Equation (8).

To calculate the K-coverage parameter, the process described in the Appendix A.7 is used, (1) assuming that neither the mean value nor the standard deviation is known, (2) using a fixed number of samples, and (3) estimating the sample mean and standard deviation. This produces Figure 8, which is similar to Figure A7. The red line passes approximately through the center of the cloud corresponding to the K₉₅ = 1.64, which is calculated from the normal distribution. A higher value K_95/95 is selected to ensure that 95% of the values are above the line, which ensures that the upper threshold of the tolerance limit is covering 95% of the bias values. Similar plots can be generated for different values of N, recording the corresponding K_95/95 value. Figure 9 shows the result of this experiment, with a sharp initial decline in the ratio K_95/95/K₉₅ for a smaller value of N. In reality, the K-coverage should approach 1.0 for an infinite number of samples, which is taken to be 10,000 in this numerical experiment. Assuming the mean value is based on 17 samples, one per experiment, K_95/95/K₉₅ =1.54.

Finally, given that TSURFER bias PDF is analytically available as shown in Figure 10, the true mean and standard deviation are known, which is expected to give a lower upper threshold value than the one calculated above. The upper thresholds per Equation (9), with and without the correction factor $1+1/\sqrt{N}$, are shown in Figure 10. The area below the non-corrected value K₉₅ is found to be 94.4%, and the corrected value K_95/95 covers 98% of the area. It is not surprising that the two values are close, differing only by a factor of $1.24=1+1/\sqrt{17}$.

4. Conclusions

In support of using TSURFER for criticality safety applications, this work has focused on the development of a defendable methodology to calculate tolerance limit for TSURFER-calculated bias. TSURFER currently calculates both the bias and its uncertainty, but it lacks the ability to determine the probability that the bias may exceed a certain upper threshold value, which is needed to support the calculations of upper subcritical limits. In response to this need, the mathematical basis for tolerance limit calculation has been developed in a manner consistent with TSURFER Bayesian-based methodology. Deviating markedly from other mathematically heavy presentations, the current work adopts a pedagogical presentation style intended to target practitioners to support the end-users of TSURFER. In this spirit, the work focuses on providing an intuitive understanding of the meaning of tolerance limits and their calculational methods. Basic mathematical equations supported by graphical aids are employed to illustrate the meanings of the terms and methods developed. Finally, numerical experiments using realistic critical benchmark experiments are used to exemplify application of the TSURFER code. Beyond TSURFER, this paper is expected to benefit a wide range of model-validation and uncertainty-quantification activities, for which the goal is to characterize and consolidate confidence from multiple knowledge sources. The work presents general description for the C³ process, which may be adapted to various end-user applications, such as the assimilation of results from experiments and calculations, from codes with different levels of fidelity, and so on.