Skip to navigation – Site map

Quantifying uncertainty in an industrial approach : an emerging consensus in an old epistemological debate

E. de Rocquigny

Abstract

Uncertainty is ubiquitous in modern decision-making supported by quantitative modeling. While uncertainty treatment has been initially largely developed in risk or environmental assessment, it is gaining large-spread interest in many industrial fields generating knowledge and practices going beyond the classical risk versus uncertainty or epistemic versus aleatory debates. On the basis of years of applied research in different sectors at the European scale, this paper discusses the emergence of a methodological consensus throughout a number of fields of engineering and applied science such as metrology, safety and reliability, protection against natural risk, manufacturing statistics, numerical design and scientific computing etc. In relation with the applicable regula-tion and standards and a relevant quantity of interest for decision-making, this approach involves in particular the proper identification of key steps such as : the quantification (or modeling) of the sources of uncertainty, possibly involving an inverse approach ; their propagation through a pre-existing physical-industrial model; the ranking of importance or sensitivity analysis and sometimes a subsequent optimisation step. It aims at giving a consistent and industrially-realistic framework for practical mathematical modeling, assumingly restricted to quantitative and quantifiable uncertainty, and illustrated on three typical examples. Axes of further research proving critical for the environmental or industrial issues are outlined: the information challenges posed by uncertainty modeling in the context of data scarcity, and the corresponding calibration and inverse probabilistic techniques, bound to be developed to best value industrial or environmental monitoring and data acquisition systems under uncertainty; the numerical challenges entailing considerable development of high-performance computing in the field; the acceptability challenges in the context of the precautionary principle.

Top of page

Editor's notes

The article has been reviewed by Enrico Zio, Roger Flague and a third anonymous referee.

Publication history

Received: 27 February 2009 — Revised: 16 April 2009 — Accepted: 17 April 2009— Published: 25 May 2009.

Full text

Introduction

1Uncertainty and associated risk assessment is rapidly developing on large-scale industrial or environmental systems along with the dissemination of advanced quantitative modeling in support of decision-making, the increased public awareness or appetite for questioning expertise and the enforcement of tighter safety or environmental control standards. In fact, uncertainty may be viewed as consubstantial to any human activity well beyond the issues of applied science and decision-making. Yet, this paper concentrates on the narrower field of quantitative and quantifiable uncertainty, with a modeling view, as the subject of considerably growing interest in industrial or environmental fields. This may sound somehow restrictive to those having in mind the real-world risk situations where on the one side human and organisational factors play an essential role (as evidenced by classical examples of the Three-Mile-Island accident in the nuclear field or of the Challenger space shuttle in the aerospace industry) and on the other side uncertainty is to a large extent poorly or not at all quantifiable (think about quantifying lack of knowledge about the plausibility of September 11th or another similar event). Those aspects evidence obvious limitations to the applicability of uncertainty quantification, although some limited modeling contributions may still apply here and there, such as the human reliability probabilistic models. Quantitative risk and uncertainty remains however essential existing or future tools for the regulation of industrial activity or environmental control, as well as the decision-making in the corporate investment or public infrastructure or health.

2Indeed, a growing number of industrial studies involving quantitative models include treatments of the numerous sources of uncertainties affecting their results. Uncertainty treatment in physical, environmental or risk modeling is the subject of a long-standing theoretical literature rooted in fundamental statistical and economical thinking (Knight 1921; Savage 1954). It then developed in the risk and environmental assessment fields (Beck 1987; Granger Morgan & Henrion 1990; Helton 1993; Hamby 1994; Paté-Cornell 1996) as well as in metrology (ISO GUM, 1995) and signal-processing (Shannon, 1948; Cover & Thomas, 1990). Such developments included a central debate on the relevance of classifying the large variety of uncertainties encountered in two categories : namely the epistemic (or reducible, lack of knowledge, …) type referring to uncertainty that decreases with the injection of more data, physical knowledge or model runs and the aleatory (or irreducible, intrinsic, variability, …) type for which there is a variation of the true characteristics of the systems that may not be reduced by the increase of data or knowledge. More recently, uncertainty treatment gained large-scale industrial importance as a number of major applications gradually included some uncertainty treatment, including in particular : (i) nuclear safety studies involving large scientific computing (thermo-hydraulics, mechanics, neutronics etc.) or Probabilistic Safety Analyses (PSA); (ii) advanced design in the aerospace, automotive or more generally mechanical industries; (iii) oil exploration and underground waste disposal control; (iv) environmental impact studies.

3On the one hand, appropriate uncertainty management allows for improved system designs, more accountable forecasts, better natural and industrial risk control, and more robust performance, thereby generating an “internal/business” driver. On the other hand, many institutional bodies or regulators now demand technical justifications, including an understanding of the influence of uncertainties in industrial safety or environmental control, thereby generating an “external/regulatory” motivation. Both may take place at many different steps of the industrial cycle from upstream research to in-service operation and control (see Figure 1).

Figure 1: A European panel of recent industrial uncertainty studies (de Rocquigny et al 2008).

Figure 1: A European panel of recent industrial uncertainty studies (de Rocquigny et al 2008).
  • 1  European Safety, Reliability and Data Association, www.esreda.org

4Most recently, the ESReDA1 European network of industries and academics undertook a large review and methodological research resulting in a consistent generic framework applicable to most industrial studies (de Rocquigny et al 2008), in which the author of this paper took a leading role. For most of the uncertainty treatment undertaken in the industrial world, at least partial probabilistic modeling of the uncertainties is considered. Yet, deterministic uncertainty treatment is generally still involved and more isolated non-probabilistic ventures (such as fuzzy sets, evidence theory or possibilistic approaches) may be encountered, resulting in a rather heterogeneous set of decision criteria and differing interpretations of the probabilistic or non-probabilistic figures. One of the challenges that motivated such research was to properly formulate in consistent mathematical terms the variety of decision criteria encountered in practice, as a prior to direct meaningful uncertainty modeling.

5A number of generic challenges may be encountered in developing industrial uncertainty treatment:

  • cultural and organizational ones: the consideration of uncertainty disrupts to some extent the traditional engineering habits and regulatory settings and often faces claims of costly, confusing or insufficiently guided sophistication, as risk analysis proves generally difficult to communicate to clients or furthermore to the general public;

  • policy ones: decision-making under uncertainty requires the tricky choice of quantities of interest or risk measures (e.g. expected utilities) that should properly represent risk aversion or behavior in front of uncertainty;

  • scientific ones: the coupling of probabilistic and phenomenological modeling generate new concepts and open mathematical and epistemological issues;

    • 2  Open Treatment of Uncertainty Risk aNd Statistics : www.openturns.org (Andrianov et al, 2007).

    technical and computational ones: regarding for instance the need to tackle large-scale physical-numerical models as most design or operation control studies rely on sophisticated engineering processes involving complex numerical codes, all the more so since high performance computing unlashes ever-increasing meshing sizes or micro-macro equation formulations. They require large CPU budget to run and are fed by quite heterogeneous sources of information (noised data on various physical variables, incomplete expertise), far away from the closed-form examples originally studied in the literature. They lead to an essential scientific computing challenge, as any uncertainty treatment increases by orders of magnitude the number of runs required by best-estimate studies. Dedicated tools are required in support : the present research was closely related to the development of the Open Source software platform called Open TURNS2.

6This paper provides a theoretical review of the subject inspired by a large panel of cases in various fields of engineering and environmental studies, evidencing the emergence of a practical consensus upon such an old epistemological debate, and setting forward a number of top-priority open challenges. The remaining is structured as follows. Section §2 introduces the versatility of the concepts, recalling the main threads of scientific debates on the subject, to which the generic uncertainty treatment approach explained in §3 contributes a practitioner answer. In §4, three examples illustrate the versatility of the framework in environmental monitoring, industrial safety and natural risk assessment. §5 open up the discussion on a number of key practical challenges mentioned earlier regarding respectively the definition of risk criteria, and the data modeling, calibration and inverse methods. Conclusion and references end up the paper.

An old epistemological issue

Defining uncertainty and risk

7As mentioned already in the beginning, uncertainty is the subject of long-standing epistemological interest as it stands in fundamental connection both with any type of modeling activity or the scientific consideration of risk. The former relates to the fact that any model endeavor brings along more or less explicit consideration of the uncertain deviation of the model to empiric evidence. Regarding the latter, risk and uncertainty analysis prove so much connected in applications that the precise delimitation between the practical meaning of the two terms does not appear to be central modeling-wise, as it essentially depends on the system definition and the terminological habits of the field considered: to put it as simple as possible, some would interpret risk analysis as the consideration of the possible consequences and associated uncertainties about what will be the consequences, while uncertainty analysis could be limited to describe the initiating events.

8Indeed, in the perspective of the present paper, the practicality of undertaking an uncertainty (or risk) study of a given system is generated by the following common features : (i) the fact that the state of the system considered, conditional to taking some given actions, is imperfectly known at a given time; (ii) the fact that some of characteristics of the state of the system, incorporated in a given type of “performance” or “consequence”, are at stake for the decision-maker. Because of (i), the best that may be looked after are possible or likely ranges for the variables of interest quantifying those characteristics ; more specifically, inference will be made under a probabilistically-defined quantity of interest or risk measure, such as an event probability, coefficient of variation of the best-estimate, confidence intervals around the prediction, value-at-risk etc. The rationale of risk or uncertainty modeling is to estimate those quantities, aggregating any information available on any type of variable linked to the system, plus the information brought by statements defining the system phenomenology and structure (physical laws, accident sequence analysis etc.). Combining both statistical and phenomenological models aims at producing the highest-informed inference, hopefully less uncertain than a straightforward prediction obtained through pure empiric data (frequency records of observations) or expert opinion.

9A large variety of causes or considerations give rise to practical uncertainty in the state of a system as defined in (i) here above, such as: uncertain inputs and operating conditions in the industrial processes; model inaccuracy (simplified equations, numerical errors, etc.); metrological errors; operational unknowns; unpredictable (or random) meteorological influence; natural variability of the environment; conflicting expert views etc. A rich literature has described the variety of natures of uncertainty and discussed the key issue of whether they should or could receive the same type of quantification efforts, particularly a probabilistic representation. Such a debate may be traced back to the early theory of probability rooted in the 17th century while modern thinking may be originally inherited from economics and decision theory, in close link with the renewed interpretation of statistics. (Knight 1921) has introduced the famous distinctions between “risk” (i.e. unpredictability with known probabilities) and “uncertainty” (i.e. unpredictability with unknown, imprecise probabilities or even not subject to probabilisation), although it is less often remembered how these early works already admitted the subtleties and limitations incorporated in such simplified interpretation regarding real physical systems. Indeed, economic and decision theory literature have generally restricted to simple decision settings, such as closed-form quantitative lotteries, without tackling in details the physical bases of industrial systems. Closer look evidences that various forms of uncertainty, imprecision, variability, randomness, model errors are mixed inside phenomenological data and modeling. Think for instance about riverbed roughness for which the topography at a time is generally incomplete, varies significafficuin time -enviroor after flood events, and is incompletely modeled within simplified hydrodynamics and associated numerical riverbed mesh. Knowledge of the systems is seldom as binary as incorporated in such statements as “probabilities are known” vs. “unknown” so that modern risk analysis would not generally stick to such historical distinctions.

A debate on the natures of uncertainty and associated quantification methods

10Indeed, tackling real-scale physical systems has been further discussed on that issue within the risk assessment community, taking a large extent in the 1980s and 1990s in close link to the development of US nuclear safety reviews or environmental impact assessments (Granger Morgan & Henrion 1990; Apostolakis 1990; Helton 1993). As already mentioned, debate has notably concerned the classification of the large variety of uncertainties encountered in large industrial systems into two salient categories, and the rationale for separatviroor not those two categories : namely the epistemic or reductible type with respect to the injection of more data, modelingphysical knowledge or model runs and the aleatory type,  that proves irreducible. To a certain extent, this epistemic/aleatory distinction may be viewed as refinement of the early economic distinction between uncertainty and risk. Note in fact that the practical reducibility of some of the uncertainty sources does not practically equate their “epistemic” (or theoretical reducibility) nature : reducibility also involves some industrial/practical constraints, and even a cost-benefit point of view, and there can be a continuum between “stricticuirreducible” or “reducible” : in some cases, “epistemic” uncertainty cannot be reduced : e.g. very expensive measurements, or even non-feasible without damaging the equipment …

11Finer distinctions include (Oberkampf et al 2002 ; Frey & Rhodes 2005; de Rocquigny 2006):

  • "variability" vs. "uncertainty" : this distinction, not far but not completely equivalent to the preceding one, is used more specifically when the system inputs mix a population of objects (or scenarios), a spatial distribution of properties within a system, or even a temporal distribution of properties affecting a system

  • "epistemic uncertainty" vs. "error", is a finer distinction depending on whether the ignorance or subjective uncertainty is “inevitable” or "deliberate" in spite of the availability of knowledge.

  • uncertainty that is "parametric" (associated with model inputs according to the level of information available on those inputs) vs. "modeling uncertainty" affecting the adequacy of the model itself to reality (structure, equations, discretisation, numerical resolution etc.)

12Accordingly, some authors link the choice of particular mathematical settings to such different natures of uncertainty involved in the system studied. On the one hand, the key aleatory/epistemic distinction has motivated the use of double probabilistic settings (Helton 1993; Apostolakis, 1999). This means a probabilistic description of the uncertain (aleatory) states of the system, for instance failure probabilities for uncertain structural failures or gaussian standard deviation for the variability of an environmental feature, upon which a second level of (epistemic) probabilistic description represents the lack of knowledge or expert discrepancies in the proper parameters of such aleatory phenomena. Conversely, such distinction is considered to be impractical to others. Measurement uncertainty is an example of practical source within which it is hard to discern epistemic and aleatory components : the (ISO 1995) international standard has in fact cancelled the mentioning of epistemic/aleatory structure as a mandatory distinction in the analyses and treatments, which receive therefore a common probabilistic sampling approach. Similarly, major developments have approached the impact of uncertainty onto complex numerical models with the help functional and numerical analysis, statistical computer science (design of experiment) under the name of sensitivity analysis : (Cacuci 1980) initiated the exploration of the impact, and major developments occurred since with an extension to global analysis (Sobol, 1993; Saltelli et al, 2004), within which single probabilistic settings are systematically used whatever the nature of uncertainty involved. Eventually, probabilistic treatment of uncertainty has been criticized by a number of authors, particularly for those sources of uncertainty affecting physical systems not amenable to be characterized as aleatory (or randomness) or as variability, hence renewing the early distinctions made by the economic literature (Granger Morgan & Henrion, 1990).

A Generic methodological framework in the industrial world

13Yet, when moving into supporting practical decision-making, industrial experience evidences that the variety of settings used to represent uncertainty does preserve a core of essential features, amenable to a unified framework approach. This section will summarise the generic methodology that has emerged as the product of a number of recent papers, and particularly with the ESReDA European industrial group (de Rocquigny, 2008).

Industrial system models considered

14Quantitative uncertainty assessment in the industrial practice typically involves:

  • a pre-existing physical or industrial system or component lying at the heart of the study, that is represented by a pre-existing model,

  • inputs, affected by a variety of sources of uncertainty,

  • a variety of actions, i.e. design, operation or maintenance options or any type of controlled variables or events that enable to modify or control to some extent the system performance or impact

  • an amount of data and expertise available to calibrate the model and/or assess its uncertainty,

  • industrial stakes and decision-making that motivate the uncertainty assessment more or less explicitly. They include: safety and security, environmental control, financial and economic optimization etc. They are generally the purpose of the pre-existing model, the output and input of which help handling in a quantitative way those stakes within the decision-making process.

Figure 2 – The pre-existing or system model and its inputs / outputs

Figure 2 – The pre-existing or system model and its inputs / outputs

15The pre-existing system may refer to a very wide variety of situations, such as: a metrological chain, a dam or hydraulic facility, a maintenance process, an industrial or domestic site threatened by flood risk, etc. Talking about quantitative studies of uncertainties, that system will generally be modeled by a single numerical or a chain of models referred to as the system model or pre-existing model: anything from the straightforward analytical formulae to coupled 3D finite element hydrodynamic or comprehensive hydrological models etc. standing as a numerical function linking output variables z = G(x , d) to a number of continuous or discrete input variables or parameters, where:

  • some input variables (noted x = (xi)i=1…p, underlining denoting vectors throughout the paper) impacting the state of the system are uncertain, subject to randomness, variability, lack of knowledge, errors or any sources of uncertainty,

  • while other inputs (noted d ) are known or fixed, either being well known (e.g. actions or design choices, controlled experimental factors for calibration …) or being considered to be of secondary importance with respect to the output variables of interest.

16Note that computation of the pre-existing model z = G(x , d) for a point value (x ,d) (i.e. not uncertain at all)may require a very variable CPU time: depending on the case, from 10-4s to several days for a single run. Within the potentially large number of raw outputs of the models, it is useful to sub-distinguish the model output variables of interest that are eventually important for the decision criteria are included formally within the vector z = (zv)v=1..r: most of the time, z is a scalar (z) or a small-size vector (less than 5 components). Although the state of the system may be of high dimension, the decision-making process involves essentially one or few variables of interest such as: a physical margin to failure, a net cost, an accumulated environmental dose, or a failure rate in risk analysis.

Probabilistic uncertainty modeling

Rationale for uncertainty modeling

17It is worth insisting on the fact that the undertaking of an uncertainty study is linked to two underlying facts: (i) the state of the system considered, conditional to taking some given actions (d), is imperfectly known at a given time; (ii) some of the characteristics of the state of the system, incorporated in a given type of “performance” or “consequence” represented by the variables of interest (z), are at stake for the decision-maker. Think of :

  • the occurrence of flooding over a given dike height within the next year, essential for safety, cannot be predicted exactly because of meteorological unpredictability or lack of knowledge of flooding phenomena,

  • the true length of a device at the time it is measured within a given experimental setting, an essential feature to secure the fit with the assembling specifications, is imperfectly known due to internal noise or uncontrolled and influential environmental factors affecting the sensor,

  • etc.

18Although a pre-existing model may be defined as a theoretically-causal or deterministic relationship between precise quantities characterising the state of the system, the consideration of uncertainty about the system means of course that we do not have the ambition of predicting exactly the realization of z based on precise-enough information on predicted values for x and e (i.e. representing the true state of the system). By the above definition, there will always be lack of knowledge to some extent of the state of the system. Modeling under uncertainty means that the information available – be it observations or expertise on those inputs - could merely characterise the possible or likely values for x. Hence, the model should help to infer possible or likely ranges for z deduced from that characterisation of the inputs.

19Whatever the paradigm chosen, there is a need for defining a distribution or measure of uncertainty describing those possible or likely values, which will be referred to as the (input) uncertainty model. The uncertainty model will encode the lack of knowledge remaining after incorporation of all information available to describe the inputs, either observed data or expertise. For instance :

  • deterministic (interval computation) : uncertain inputs are varying within physically-maximal or merely plausible ranges

  • probabilistic : a joint continuous or discrete distribution describing the input uncertainty within random vector X

  • Dempster-Shafer : set of belief and plausibility distribution function describing the input uncertainty (Dempster 1967; Helton & Oberkampf 2004)

Standard probabilistic setting

20While literature has discussed to a large extent the pros and cons of various probabilistic or non-probabilistic paradigms (for instance Helton & Oberkampf 2004), it is assumed later on that probabilistic uncertainty treatment is acceptable for the cases considered : meaning that, whatever the final goals, decision criteria will generally be built upon the probabilistic distribution Z representing uncertainty on the output variables of interest or some of its characteristics, such as : standard deviation, variance or coefficient of variation (i.e. easily interpreted as percentages of relative error/uncertainty) on one or more outputs, confidence intervals on the variable(s) of interest, probabilities of exceedance of a safety threshold etc.

21Formally, consider thus a probabilistic setting, involving a sample space X  representing the space of possible values for the input variables x in which X = (Xi)i=1…p ~ FX(xθX) whereby FX indicates a joint cumulative probability distribution function (cdf) associated to the vector of uncertain inputs X (capital letters denoting random variables), defining the uncertainty model. Parameters of FX are grouped within the vector θX: they include for example statistical parameters of position, scale and shape, or moments of marginal laws, coefficients of correlations (or parameters of the copula function), or even extended parameters of a non-parametric kernel model. Consequently, the output variablesZ inherit also a random model, the distribution of which being generally unknown and costly to compute if G is a complex system model : quantifying the uncertain behaviour of such outputs is in fact the purpose of uncertainty propagation methods which will receive more attention hereafter.

22Note also that multiple epistemological interpretations of such a probabilistic setting are possible, while still resulting in the same mathematical objects and computation needs. Indeed, frequent uses are made of such a standard probabilistic setting in regulated practice such as metrology (ISO, 2005), pollutant discharge control or nuclear licensing without positively choosing a single theoretical interpretation. This discussion is closely linked to that of the various natures of uncertainty about the state of the system which may be represented, such as natural time or space variability, lack of knowledge etc. : Section 3 will come back to that discussion. At this stage, only preliminary comments will be given. The frequentist interpretation considers x and z=G(x,d) as observable realizations of an uncertain model standing for an underlying population of truly variable systems (in time, space, pieces …) and the “true” uncertainty model can hence be theoretically inferred from a very large set of data. The subjective interpretation considers probability distributions as subjective preferences of the decision-maker, without the need for an underlying repeatable phenomenon with observable frequencies as appropriate when dealing with a unique system.

Double probabilistic settings representing separate aleatory/epistemic components

23As discussed in the §2, it is usual in some cases to explicitly distinguish between the aleatory and epistemic components of uncertainty. Well beyond the epistemological debate, such a distinction is crucial because of its significaff analytical, computational and decision-making consequences in the sense that the study process could highlight where uncertainty reduction, be it in data collection or model refinement, is most instrumental, yet with a corresponding cost in complexity. Mathematically, key differences regard the probabilistic definition of the uncertainty model, which may be summarised as the potential distinction of two levels requiring simple or double probabilistic settings :

  • level – 1  (mandatory) : an uncertainty model is built upon the inputs X, representing through a given probabilistic distribution FX (x  ‌  dX) the extent of uncertainty. This distribution is characterized by given parameters θX,

  • level – 2 (optional) : an additional uncertainty model is built to represent the variability or lack of knowledge on the precise values of parameters θX that prescribe the extent of uncertainty : it takes the form of a supplementary distribution encoding the random behaviour of those parameters X(X| ), as a function of “hyper-parameters” .

24It is often referred to in the literature as the aleatory-epistemic distinction (Helton & Burmaster, 1996), the level-1 modeling the aleatory (or “risk”) component while the level-2 models the epistemic (or “uncertainty”) component. Any type of risk or uncertainty analysis would always develop the first level as a model of the uncertainty about the variables or events of interest. Such an uncertainty model (namely its parameters θX) needs to be estimated or inferred on the basis of all available pieces of information, such as data or expertise. More elaborate interpretations and controversies come up when considering the issue of the lack of knowledge regarding such uncertainty description, as generated for instance by small data sets, discrepancies between experts or even uncertainty in the system model. This is particularly the case in the field of risk analysis or reliability, but happens also in natural risk assessment, particularly flood risk. The incorporation of observed data to infer or validate the level-1 probabilistic model, when done through classical statistical estimation, generates statistical fluctuation in the parameter estimates: this is made obvious by confidence intervals around e.g. high-level flow quantiles or return periods.

25This is even more so when considering the choice of the distribution for an input xi for which traditional hypothesis-testing techniques give at most only incomplete answers : is the Gaussian model appropriate for the distribution of xi, as opposed to, for instance, a lognormal or beta model ? This results in a sort of “uncertainty about the uncertainty (or probabilities)”, although this formulation is controversial in itself and may be more specifically referred to as epistemic uncertainty about the aleatory (or variability) characteristics. This is where a potential level-2 uncertainty model may be contemplated to represent the uncertainty about the parameters or even the laws themselves through an extended parameterisation. It is sometimes referred to as estimation uncertainty, in order to mark its strong link with the step of the process where the uncertainty model is estimated.

The central concept of the quantity of interest or risk measure

26As introduced earlier, the combination of the system model and the uncertainty model should help to infer possible or likely ranges for z. Within a probabilistic framework, dedicated probabilistic quantities are wanted to measure the likelihood of values for z. Such a quantity used for the inference of the outputs of interest under uncertainty will be called a quantity of interest (de Rocquigny Devictor Tarantola, 2008), otherwise referred to as a performance measure (Aven, 2003) or risk measure in the finance and economics. Some examples :

  • percentages of error/uncertainty on the variable(s) of interest (i.e. coefficient of variation)

  • confidence intervals on the variable(s) of interest

  • quantile of the variable of interest (such as the value at risk or VaR in finance), possibly conditional on penalised inputs

  • probabilities of exceedance of a safety threshold or of an event of interest (sometimes termed “assurance” when considering conditional non-exceedance probabilities)

  • expected value (cost, utility, fatalities …) of the consequences

  • confidence intervals on the probability of an event

27In any case, the quantity of interest or risk measure will be subsequently noted cz (d) since it is computed to represent the likely values of the outputs of interest z and hence crucially depends on the choice of actions d that modifies the anticipated state of the system, albeit to an uncertain extent. A closer mathematical look evidences that one could always write these quantities in the form of a functional of Z,cz = F[Z] typically requiring a computational process including multiple integration (hence not being a straightforward function). Assuming a double probabilistic setting, quantities of interest become formally more complex as they may extend to both levels, such as the range of possible values or confidence interval for the coefficient of variation or the exceedance probability. Mathematically, the quantity of interest c(2)z is then a functional of the extended uncertainty model {FX (. ‌ X ,d), }, again involving multiple integrals because a probabilistic second-level introduces typically a second layer of integration on top of the level-1 quantities of interest. In other words, computing such a level-2 expectation involves a double integration, meaning “averaging the risk measure over the uncertainties in the risk components”. Such quantity requires generally a highly-costly double loop of Monte-Carlo sampling (such as 100 * 10 000 runs). It is typically integrated in a decision criterion requires that “frequency of the undesired event of interest is less than 1/1000-yr at an -level of confidence”, encountered in protection against natural risk or integrated probabilistic risk assessment.

28The quantity of interest or risk measure is the key quantitative tool used in make decisions under uncertainty. From a theoretical standpoint, risk and uncertainty studies should eventually help in making decisions about appropriate actions in the context of a relative lack of knowledge about the state of the system. Decision-making is always about relative preference used to select within the scope of available actions. Hence the quantity of interest cz (d) should be viewed as an appropriate means to rank the options in the face of uncertainty. Classical literature (Savage 1974 ; Bedford & Cooke 2001) evidences for instance the expected utility as a central quantity of interest, the utility being a scalar function computed on the output of interest u(z) that represents the decision-maker preferences (e.g. his risk aversion) ; “expected” meaning the expectation over the probabilistic uncertainty model on X, and hence Z=G(X,d), a random vector which is interpreted to represent the decision-maker’s subjective belief about the possible states of the system.

Generic goals of uncertainty studies

29Yet, in more practical terms, uncertainty studies are not always implemented up to the completion of that ultimate theoretical goal, select within a scope of actions. In many cases, regulation specifies the compliance with an “absolute” criterion. Far-stretching decision-making would weigh the relative preference between building a nuclear power plaff (e.g. under seismic risk) vs. operating a hydro-electric facility (e.g. under flood risk and climate change impacts); or between accepting a standard device that has been controlled with a cheap (medium-accuracy) vs. an expensive (high-accuracy) metrological protocol. In fact, operational regulation or practices would often prefer to specify a fixed decision criterion (or risk acceptance criterion, tolerance specification etc.) to comply with, such as :

  • “design should ensure that the mechanical failure margin should remain positive for the reference seism in spite of uncertainty, with a probability less than 10-bto be negative“

  • “system redundancy design and maintenance should guarantee that frequency of plaff failure should be less than 10-b per year, at a 95% confidence level covering the uncertainties”

  •  “there should be less than 3% uncertainty on the declared measurement value for the output of interest”

  • “up to the - initiator at leasf (e.g. 1000-yr flood), installation is safe under designd

  • “range of the output variable of interest should be always less than 20%” or “maximal value of the variable of interest should stay below a given absolute threshold”

30Such decision criteria mostly appear as tests derived from the quantity of interest, comparing the quantity of interest to a fixed threshold cz (d) < cs. Note that this type of decision criterion may be differently interpreted whether one considers the probabilistic modeling as a representation of subjective degrees of belief over the imperfectly known states of the system  (such as in the theory of expected utility) or modeling observable objective frequencies of possible states within a sequence of time of the system or a population of similar systems perform (such as in the frequentist approach), or even both (as in a Bayesian approach).

31To complete the picture it is necessary to consider also that many risk and uncertainty studies are undertaken in relatively upstream stages, not yet facing explicit decision-making processes or even less regulatory criterion. This is the case in upstream development of phenomenological models or numerical code qualification. Even when these models and codes are designed to ultimately help decision-making or industrial certification, the goal of upstream uncertainty studies may be firstly to understand the extent of uncertainty affecting the model prediction, particularly the relative importance of input uncertainties in order to intensify measurement or modeling efforts on the most important ones. Once a better understanding of the uncertainties and of the behaviour of the system model is available so that it is considered to have the potential of a later operational use, it may be necessary to establish formally its range of validity and to control its residual prediction uncertainty : a more or less sophisticated calibration, validation or qualification process aims to accredit its validity. Although those two goals may be understood as only indirectly linked with decision-making, it may be seen that the quantity of interest is still the key quantitative tool involved. For instance, importance ranking of the uncertain inputs or the proper model calibration is different whether variance or a failure probability on the output of interest is selected as the quantity of interest, as discussed in (de Rocquigny et al., 2008). To summarise, industrial practice shows that, most of the time, the goals of any quantitative risk/uncertainty assessment belong to the following four categories:

  • U (Understand): To understand the influence or rank importance of uncertainties, thereby to guide any additional measurement, modeling or R&D efforts.

  • A (Accredit): To give credit to a model or a method of measurement, i.e. to reach an acceptable quality level for its use. This may involve calibrating sensors, estimating the parameters of the model inputs, simplifying the system model physics or structure, fixing some model inputs, and finally validating according to a context-dependent level. In a sequential process it may also refer to the updating of the uncertainty model, through dynamic data assimilation.

  • S (Select): To compare relative performance and optimize the choice of maintenance policy, operation or design of the system.

  • C (Comply): To demonstrate compliance of the system with an explicit criterion or regulatory threshold (e.g. flood control, dam safety, nuclear or environmental licensing, etc.).

32Moreover, the feed-back process proves an essential item in practical risk/uncertainty studies : for instance, failure to secure compliance for a given design may lead to change the actions in order to fulfill the criterion : this may involve changing the design itself in order to cover more conservatively the uncertainties, or investing more on information in order to reduce the sources of uncertainties, provided that those are truly reducible on the basis of additional scientific evidence. This would obviously rely on ranking the importance of uncertain inputs or events, and possibly require a new accreditation of the altered measurement or modeling chain before being able to use it in order to demonstrate compliance.

33Remark also that quantities of interest may differ as to the extent to whichthe quantity computed on the output of interest z correspond to probabilistic or deterministic uncertainty models on the various uncertain inputs x. Indeed, it should be reckoned that in most studies, besides describing some of the uncertain inputs by probabilistic distributions, at leasf some other uncertain inputs (variables or events) are fixed. This is because :

  • for some model inputs, the decision-process will conventionally fix the values despite of the acknowledgement of uncertainties: for comparative purpose, by a conventional “penalisation” i.e. the choice of a fixed “pessimistic” scenario etc.

  • uncertainties affecting some model inputs are considered to be negligible or of secondary importance with respect to the outputs variables of interest

34Noting xpn these inputs, a more precise specification of the quantity of interest should involve the explicit conditioning on those fixed inputs cz (d ) = cz (d | xpn). As an essential point to build a generic description or risk and uncertainty modeling, such a notation appears to be a convenient mean to unify the deterministic or probabilistic descriptions of uncertainty.

Generic conceptual framework and key steps in any uncertainty study

35At this stage a risk/uncertainty study can be formally summarized as involving a number of generic tasks that are summarized within the following main steps (see also Figure 3):

36(A) specify the system model, variables of interest, the setting and quantity of interest (risk measure) :

  • choose variables z according to decision-making, as well as a system model G(.) predicting z as a function of decision variables (d) and all input characteristics (x) defined as to retrieve information and represent uncertainty affecting the system

  • choose a representation of uncertainty, e.g. associating probabilistic and deterministic choices

  • FX (xX ,xpn ,d)

  • choose a quantity of interest in accordance with the decision-making process

37(B) identify (i.e. estimate) the uncertainty model FX (. ‌ X ,d) on the basis of available data and expertise

38(C) compute the quantity of interest cZ (d), i.e. propagate the input uncertainty model through G(.) to estimate the output distribution FZ (zd) and/or the associated quantities of interest and sensitivity indices 

Figure 3 – generic conceptual framework

Figure 3 – generic conceptual framework

39Step A has been introduced in the previous paragraph; subsequent step B (data modeling) and C (computational propagation and sensitivity analysis) will be briefly described hereafter.

Step B - uncertainty modeling

40Once the sources of uncertainty and corresponding input variables have been identified, there is inevitably a step of uncertainty modeling (or quantification, characterisation etc. of the sources of uncertainty) that depends on the type of quantities of interest chosen .

  • in a probabilistic framework, the uncertainty model will be theoretically a joint cdf on the vector of uncertain inputs (x), although it may be more simply specified as a set of simple parametric laws (e.g. Gaussian) on the components with some independence hypotheses or approximate rank correlations.

  • in a non-probabilistic framework, the uncertainty model would be for instance a Dempster-Shafer couple of plausibility / belief functions on x

  • in a deterministic framework, the maximal range on each component of  x

  • Whatever the framework, there is always however the same need to take into account the largest amount of information :

  • direct observations on the uncertain inputs, potentially treated in a statistical way to estimate statistical models

  • expert judgment, under a more or less elaborate elicitation process and mathematical modeling : from the straightforward choice of intervals to more elaborate Bayesian statistical modeling, expert consensus building …

  • physical arguments : e.g. however uncertain, the input remains positive, or below a known threshold for physical reasons …

  • indirect observations : this is the case when the model is calibrated / validated and may involve some inverse methods under uncertainty

41in order to build a satisfactorily “uncertainty model”, that is to say a measure of uncertainty on the inputs. Uncertainty modeling may be a resource-consuming step for data collection ; it appears in fact to be a key step that is very sensitive to the result, depending on the final goal and the quantities of interest involved : for instance, choice of upper bounds or distribution tails become very sensitive if the quantity of interest is an exceedance probability.

Step C/C’ - propagation and sensitivity analysis processes

42Once an uncertainty model is developed, the computation of the quantities of interest involves the well-known uncertainty propagation step (also known as uncertainty analysis). The uncertainty propagation step is needed to transform the measure of uncertainty on the inputs onto a measure of uncertainty on the outputs of the pre-existing model. In a probabilistic setting, this means estimating the cdf of z = G(x , d) knowing the cdf of x and given values of d, G(.) being a numerical model. According to the quantity of interest, and to the system model characteristics it may be a more or less difficult numerical step, involving a large variety of methods such as Monte-Carlo Sampling, accelerated sampling techniques (Rubinstein 1981), simple quadratic sum of variances (ISO 1995), Form-Sorm or derived reliability approximations (Madsen et al, 1986), deterministic interval computations etc. Prior to undertaking one of these propagation methods, it may also be desirable to develop a surrogate model (equivalently referred to as response surface, meta-model …), i.e. to replace the pre-existing system model by another one that leads to comparable results with respect to the outputs variables and quantities of interest, but that is however much quicker / easier to compute.

43The sensitivity analysis step (or importance ranking) refers to the computation of so-called sensitivity or importance indices of the components of the uncertain inputs variables x with respect to a given quantity of interest on the output z. In fact, this involves a propagation step, e.g. with sampling techniques, but also a post-treatment specific to the sensitivity indices considered, involving typically some statistical treatment of the input/output relations that generate quantities of interest involving both the measure of uncertainty on the outputs and inputs. A large variety of probabilistic sensitivity indices (Saltelli et al 2004) include for instance the: Graphical methods (Scatter plots, cobweb…), Screening (Morris, sequential bifurcations…), Regression based techniques (Pearson, Spearman, SRC, PRCC, PCC, PRCC, etc.), Non-parametric statistics (Mann-Whitney test, Smirnov test, Kruskal-Wallis test), Variance-based decomposition (FAST, Sobol, Correlation ratios), or local sensitivity indices on exceedance probabilities (FORM). Note that the expression “sensitivity analysis” is taken here in its comprehensive meaning encountered in the specialised uncertainty & sensitivity literature : in the industrial practice, the same expression may refer more approximately to some elementary treatments, such as the one-at-a-time variations of the inputs of a deterministic model or the partial derivatives. As such, these two kinds of indices are usually not suitable for a consistent importance ranking although they may be a starting point.

44(de Rocquigny et al 2008) evidences that the best choices for such challenging computational steps do not depend on the particular specificities of a physical or industrial context as such, but onto the generic features identified hereabove : the computing cost and regularity of the system model, the dominant final goal, the quantities of interest involved, the dimensions of vectors x and z etc. Recommendations drawn from the long-standing experience of other fields of engineering and risk assessment may hence be retrieved in order to guide the emerging uncertainty applications in hydrological and hydraulic modeling. Indeed, the methodology has now enjoyed considerable industrial dissemination through tutorials and supports the open source development platform Open TURNS.

Examples

45Three typical industrial examples are provided to illustrate the framework, while a much larger spectrum may be found in (de Rocquigny, 2008). Remember that those examples were indeed more an inspiration than a basis to the global methodology introduced hereabove: it was by removing the historic barriers between the sectoral approaches to uncertainty propagation and analysis – barriers that have been observed in recent industrial applications (metrology, reliability, risk analysis …) - that generic uncertainty problems could be formulated and placed in a consistent mathematical setting.

Metrology of environmental emissions

46The first industrial example comes from the domain of metrology. In the domain of metrology or quality control, uncertainty analysis or control is quite a basic requirement associated to the qualification of any measurement device, chain of devices or production process ; through sensitivity analysis, it is also a forcible means to optimise costs and productivity of monitoring and quality control. It may sometimes even be mandatory in the official regulations, such as in nuclear maintenance or radiological protection, or environmental control : for instance, in application of European undertakings relating to the section of the Kyoto protocol concerning industrial emissions, a system of CO2 emission permits has been established requiring uncertainty declaration in order to secure the fairness between real emissions and emission rights and stability of the financial markets for emissions permits, through the variance.

  • 3 Including the noises affecting the operation of the sensor, local fluctuation of the measurement (...)

47Probabilistic approaches are traditional in the field of metrology, whether environmental or not. It is already embodied in international standards such as the "Guide to the expression of uncertainty in measurement (GUM)" (ISO GUM, 1995): rather than aggregate elementary uncertainties in a deterministic manner, it is based on the standard hypothesis that the uncertainty associated with each sensor or measurement operation3, in comparison with a reference value, has an aleatory character that is efficiently modelled by probability calculations. To be precise about the reference value with respect to which uncertain deviations are considered, the standard recommends referring for preference to "the best value available" rather than the physical "true value". On the one hand, the specification or definition of the magnitude to be measured is never infinitely precise: at a micro-physical scale, complex temperature or external artefacts (such as background radiations, physical-chemical surface exchanges, …) constantly impact the length of a given rod at a small time scale or even blur the notion of frontier basing the definition of length. On the other hand, should the quantity be very precisely defined, the true value is never observable, but only approached by sensors based on metrological benchmarks that are themselves uncertain. Zooming further down into quantum mechanics, Heisenberg uncertainty principle establishes that any physical system is intrinsically uncertain, and perturbed by observational devices. The GUM thus recommends not working on "error", defined as the absolute variance between measurement result and (unobservable) true value, but rather on "uncertainty" representing the fluctuation of the result of a measurement supposed to have reasonably corrected all the systematic effects compared to a benchmark. The aleatory variable representing this uncertainty may include a "bias" (which therefore remains relative) if its expectation value is not null, even if some people reserve the term "uncertainty" to the aleatory variable of expectation value null, restricted to fluctuation after deduction of any bias.

48Supposing that this framework is accepted, a probabilistic quantity of interest is required in regulatory studies of uncertainties. This consists in determining "an enlarged uncertainty" cz=%uncZ defined by a confidence interval of = 95% around the measured mass tCO2, the ratio of which should not exceed a given threshold e.g. 3% max. of uncertainties around the declared value. According to a classical Gaussian linear approximation, this criterion may be linked to a given multiple of the coefficient of variation, i.e. the ratio of standard deviation to the measured mass.

49Physically, in the case of the CO2 emissions from a coal-fired power plaff (Figure 4), direct measurement of emissions by a single sensor has proved unreliable. Several options are possible to measure the number of tons of CO2 emitted (tCO2) : (a) flow-meters with integrated weighing mechanisms, (b) inference from the electric power produced for a specific consumption, (c) inventory of stocks and inputs. They all imply the aggregation, through simple analytical operations, of several elementary measurements such as the tonnage of coal consumed, the lower calorific power and the corresponding emission factor or oxidation factors. In the example of option (c), coal consumption Cc is itself deducted from the balance of tonnage supplied and variation in stocks is measured in cubature.

Figure 4 – Thermal power plaff emissions (left) and metrological steps (right)

Figure 4 – Thermal power plaff emissions (left) and metrological steps (right)

50Mathematically, it is therefore a question of aggregating the uncertainties associated with each variable xi measured by each i-th operation or elementary sensor (generally evaluated by the suppliers during design) within the function z = G(x1,…xp) = G(x) expressing the variable of interest z, for example the annual tonnage of CO2. In such context, the system model referred to earlier is simply a closed-form chain of elementary relations representing the metrological operations. Metrological sources of uncertainty Xiare classically modeled as Gaussian: this hypothesis is sometimes supported by argument based on the symmetry of errors and the existence of multiple underlying additive physical phenomena. Yet, when uncertainties are bounded, for example because of a command and control mechanism, the choice of a uniform or even triangular distribution may be preferred. The expected value is generally taken to be equal to the result of the calibrated measurement and the variance supplied by the characteristics of the sensor. Linear or rank correlation coefficients account for potential non-independence of uncertain inputs.

51Moreover the epistemic (or level-2) uncertainty in characterising the aleatory behavior of elementary devices may be explicitly included in metrological uncertainty modeling. For instance, repeatability data-based empiric variances can be multiplied by a factor greater than 1, decreasing with the number of measures n used to estimate it, instead of simply taking the gross estimated value: this accounts for the possibility, through the effect of sampling fluctuation, of under-estimating the unknown true variance. Yet, such practice is heterogeneous, because n is not always known at the industrial end-user. This question becomes particularly important when the cost of determining sources of uncertainty is high, because of the necessary tests involving burdensome basic operations. For instance in the nuclear field, this includes the reception of fuel assemblies before reloading: the measurement process is very demanding, but the risk associated with too great a level of uncertainty is serious, since mistaken acceptance of a fuel assembly too large for the core grid would result in the unavailability of the plaff.

52Propagation (or aggregation, combination etc.) of elementary uncertainties is undertaken by one of the two methods accepted by international standards: either an approximate linear Gaussian combination, or a more accurate Monte-Carlo sampling. Both also enable ranking the relative importance of sources of uncertainty, an output of key value to the industrial process. For the final result, the i-th input is only important insofar as both input uncertainty uncXi is strong and/or the physical model is sensitive to it (via ∂G / ∂Xi). In the CO2 example, the choice of a measurement chain for subsequent checking of CO2 emission was based on a minimised criterion %uncZ., but also used the importance ranking of the sources, ensuring that the heaviest contributions corresponded to sources that could not only be tracked for quality assurance, but were if possible reducible.

Structural reliability and design margins of a mechanical structure

53The second example comes from the field of structural reliability analysis and design margins. Design of industrial structures, be it in the nuclear, aerospace, offshore, transport etc., has generated the need for some rules or codes (Ellingwood, 1994) to prevent failure mechanisms, and secure reliability levels in the face of highly diverse sources of uncertainty affecting the operating systems (variability of material properties, of operational loads, fabrication tolerances, …).

54Beyond empiric design margins, a whole fare of methods are usually referred to in structural reliability analysis (Madsen et al, 1986) to tackle a structure characterized by an event of interest, or more generally a group of events, leading to failure. Failure is a matter of structural definition : it may include a number of so-called failure modes, or physical phenomena such as brutal collapse, or just crack initiation : they may happen under a certain number of conditions on two types variables affecting the system behavior, the design variables (again noted d) and the physical variables (noted x=(xi)i=1…p). A failure function G(x,d), i.e. the system model of the example, encapsulates this knowledge of the different phenomena leading to failure.

55Consider for instance the simplified example of the reactor vessel in nuclear safety : undesired failure could theoretically occur under the effect of abnormal pressure-temperature of the primary fluid, itself subjected to an internal initiator such as the drop in pressure following a pipe break elsewhere in the circuit. Stress upon a pre-existing flaw inside the vessel width could then exceed the resilience margin of the material resulting in a failure event either defined as the sudden rupture or, more conservatively, in the initiation or flaw propagation. Failure modeling involves firstly a complex finite-element thermo-mechanical model y= M(x,d) predicting stress and temperature fields as a function of numerous variables (properties of materials, flaw characteristics, the thermodynamics of accidental transients, the radiation received over time and the resulting fragilisation, etc.) as well as design or operational conditions d (such as temperature and pressure limits, recovery times etc.). Hence the failure margin z, representing the variable of interest, is computed by subtracting a stress intensity factor (noted K1(y,x,d)) from a toughness function (noted K1c(y,x,d)) defining altogether the failure function G(x,d)= K1c (y,x,d)-K1(y,x,d). Hence G(.) depends on numerous parameters (x,…xy) = x subject to multiple uncertainties, such as: properties of materials, flaw characteristics, the thermodynamics of accidental transients, the radiation received over time and the resulting fragilisation, etc..

56The field of structural reliability has a long history of standards and regulations in risk industries such as the nuclear or aerospace. The criteria specify reliability requirements according to the components concerned, that is to say an absence of failure during a given period and for a given scenario, e.g. a conventional accident with hypotheses d concerning the structure. This generally means ensuring that the failure function remains positive over a wide range of possible input values. The traditional method consists in "penalising" with safety margins those variables x that are sources of uncertainty, by applying coefficients or safety factors f i to the "best-estimate" valuesand verifying reliability zpn=G(xpn,d) > 0 by a deterministic calculation using the penalised parameters .

57This approach is generally referred to in the industry as a deterministic approach or if one prefers, as an approach through "penalised scenarios" : it involves an elementary form of deterministic treatment of sources of uncertainty pre-supposing again that x -> G(x,.) considered component per component xi is monotonous, a mostly intuitive situation in fracture mechanics, albeit less straightforward in fluid mechanics. It has the considerable advantage of limiting the number of computations to one or a few runs on potentially complex mechanical models. It however requires agreeing on reasonable upper limit of uncertainty for every component xi (or a lower one, depending on the direction of the monotony). As the identification of physically-realistic maximal values may be intractable or lead to controversial expert debates, it can happen that certain limits correspond to approximate quantiles αi for each component implicitly modelled as a random variable. For instance, a resistance property would be taken at its lower 95% value accounting for variability of materials, while a loading variable would be taken at its upper 95% value accounting for lack of knowledge or randomness of the operating conditions. This is also referred as a “partial safety factor” approach (Ellingwood, 1994), whereby safety factors are defined for each partial component xi of the vector x conditioning overall safety. An important literature continues to discuss the interest and inherent conservatism of this kind of approach which in fact aggregates potentially heterogeneous margins reflected by the quantiles, in a barely controllable manner. Experience shows it is quite difficult to translate this into an overall probabilistic risk level that could be comparable to other risk situations.

58For some years now the debate has been fuelled by comparison with approaches known as "probabilistic", explicitly modeling the sources of uncertainty by random variables X i, focusing on the probabilities of threshold exceedance and failure in the scenario defined by Pf(d) = P[G(X,d) < 0], and seeking to ensure that it is very close to 0. In fact it is frequently found that one only "probabilises" part of the uncertain inputs, which in fact comes down to a partial conditioning with the penalised values of other sources. Hence Pf will be compared to a threshold, or at least the probability of a reference scenario d0, considered as reliable enough, if an absolute threshold in 10-b is too delicate to specify. Formally, these approaches known as "probabilistic" should rather be seen as yet another kind of mixed deterministic-probabilistic uncertainty assessments.

59Faced with this criterion, can uncertainty studies make use of either of the methods of propagation already mentioned in metrology? Note that historic barriers between metrology and the reliability of industrial structures has prevented this question from easily emerging. More deeply, the problem arises essentially because we are interested, in our study of structural reliability, in rare "failure events", that is to say the tail-end of the Z distribution. In addition, such events are often associated with a pre-existing system G(.) that takes far longer to calculate than a closed-form metrology formula. In this context the Gaussian and linear hypotheses associated with Taylor quadratic approximations often prove false, while MCS requires too large a number of simulations to stabilise the estimator of Pf.

60Aside from the use of methods to reduce the variance of MCS (by reduction of the conditional dimension of MCS, importance sampling etc.), calculation strategies were historically developed specifically to evaluate the probability of exceeding a threshold, and were known in reliability research as First (or Second) Order Reliability Methods, (FORM or SORM). If the underlying approximations allow, FORM and SORM considerably reduce the computational load required to evaluate a very weak Pf. These methods also generate importance factors that rank sources of uncertainty: unlike the metrological example, they are ranked with regard to a quantity of interest of threshold exceedance instead of with regard to the average uncertainty (the coefficient of variation) of z. Yet, handling quantities of interest such as rare probabilities on complex system models remains a domain of challenging mathematical research both from the points of view of uncertainty propagation and sensitivity analysis.

An educational example – flood risk

  • 4 No conclusions may be drawn out of this paper regarding the real flood prediction methods nor the f (...)

61The final example has been developed as a realistic benchmarking and educational example for uncertainty modeling throughout the French industry (de Rocquigny, 2006). While not being used as such in real studies in the particular domain of flood risk4, it was designed to incorporate several aspects that are typical for risk analyses in various domains. The quantity of interest is the probability to surpass a threshold and information availability is typical for industrial studies in the sense that input variables are affected both by aleatory and epistemic uncertainty; although virtual and data available only on some of the uncertain inputs, with heterogeneous amount and quality. The pre-existing system is in this case a residential area bordered by a dyke protecting from uncertain flood events.

62Two main output variable of interest are involved. Safety-wise, the key variable is the overspill (Figure 5) caused by the rise s=zc-zd in water-level (zc) above the dyke crest (zd). A hydraulic model relates it to a number of features undergoing both natural meteorological and geo-morphological variability and lack of knowledge: river flow (q), riverbed elevations (up- and down-stream: zm and zv) and the state of the river, typically represented through Strickler's friction coefficient ks. Another variable is important in an economic perspective: the complete cost cc aggregating both the investment cost ci depending on the control variable d = zd (dike height) and the cost of damage cd if there is an overspill. A related financial model may itself incorporate a new source of uncertainties representing the cost of any em> apoverspill c's)ficient k.oertainty

ctors &)fierm eliabili38 an of any em an becocks isultors industrnng fra(bmarginwre ahis papm edamfriction exareates an elage c kind e on<

Figure 4 – Thermal power p5h2> Fotnotecalorporae/image/782/img-4.jpg" rel="iconSet">5img src="docannexe/image/782/img-4.jpg" alt="Figure 4 5 Thermal power p5h2> FotnotecalorporaextIconAccess">Zoom Original (jpeg5 44k)

2

50Mathematical6ample coE are int salculahis ure. Ths of /p> ffeceading to ities of interest, but probability t:

o nd e1WW8Num37

indirect obn:underthe qu-l st:0cm;the qu-ass=":0cm;line;tainn":0cm;ss="paranumbCo esti2> y close to 0tth too gem>'sard d for pf ancrind d. Qs the probability to surpanility.

s (againcice sam>indirect obn:underthe qu-l st:0cm;the qu-ass=":0cm;line;tainn":0cm;ss="paranumbSed foh2> it (again notee fairness boductizem>cc aggregating both theerest is the probability toel. Ac of d<(ntional accid/sub>
aggregating both the y.

ormashe kncyation avai)entialutthresces canis the probability to of any em> aous dos, itoned

50Mathematical6usions mossible inty propagation and analybe retrieveain. Zuss. The tras inpotentiof propase ities of int(ntioseo whichTli> 1)t, the i Reltingm>prof intare typie accurate ients, uncertainty; although highlyation and sensi expecteursue is genX iciently mm> verbsoel. Ace invalueckon absk leinty propagshain. Zof suon reasonacertaintyof the input/output relan somities clnaceial procder yead of simplyaiverbe orpora: eption of fueln true variu nd e1WW8Num29

indirect ob="paranumbTinty , or phy"listic u"aional in: eptiond criterion ntainty for evld result indirect ob="paranumbTintrobabiicitly modelincpicy contro"aional in Allion in ord iof clie 5) ; although virtual anoass tCOcld resulmi resna dy is genoximations often prove false, basic opernsors lly cr a few runs iwhile she possireferred toin the ncountered lyevel that coul,mhallenging anothe remainsdoumbe>cte

indirect ob="paranumbTint" emissicitly modelingaional in: ntial arlf a based o,afectsing tesenting the cosal or not.he compuf the resurberating con,, with heterogeypically representatives. h heteroge sulates tlnaceial proreferred. Thecrickler's faintya meateadnction ble mved unrc , abilthe of ong burdenolof simplyaimi resna dyy contro ld r/sub>tandard devtng wic guurvteriocex<1000 has nt(ble he coe)rd to the av devtng wict of o "the bee resur verbc guurvteriocex<1000 has n. Mem>crecise aboutee ficrstudies ofassoclf a based o,aortard inaance o mved un>cis the probability to(mashnputs< “the coefootnotecal”age prof iy faSELECT”

Figure 4 – Thermal Tt-decoration:undererif;">'T failNy cRoman'/em>'T failNy cRoman'/em>6-ven b580img src="docannexe/image/782/img-4.jpg" alt="Figure 4 6-ven b480img srrmal Tli> 1: Vuantityi The traional in thation and analyt/output relaootnot highlextIconAccess">Zoom Original (jpeg6 44k)

108

50Mathematical6instanceItat the irent conntin fes. Ankler'real flst and inherentionalp>. Ye conditipotenti 5) ction ireferredood predicootnotecal oge od risklue tissugain "foe funcel flrof in- (c) invesk industri effie computeferredHA rela 0um chain onriers beugh ethods to redeading tld, ths were histog fragilisati remain that inputthe keown as "probabiork, while a partieading to at chanput variis the probability to(in thial essicitly modelinage i Reltwo"probabihe sources seMathcr repreating condiabi faecal”agle of izprob),ativfromon etcm>prtex system mod(2 too nd stoo "prob)hod conhiertireferredrobasre function em>ci uncertanent althoughtingmbilisticfullarvel that couldthe coeitiesffects reeiork, while r, tyfer risk led hereabovgical referred it was by ut §3

60Aside from thstructuradrawn oudl probastal conte res elsewherTli> 2esummcause wn of anytitye sensor. Linearork, whilez = G(xFigure 4 – Thermal Tt-decoration:undererif;">'T failNy cRoman'/em>4<

i=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…pi=1…p4<

An1n5ucational e1n5u>Ots ccal resead< 1p class="texte" dir="ltr">60Aside from th is generallylf a stal coa threand inheundergoing bots ccal reseadtainty (hainrminis>etcm>pron and analyt/output ,riocexaz = own to. A relateetrolog ctorsquite difinron and analyt/output ,rf z =kuts, witYucca M preandesired fawf ashfac s e beown to. Aen sy is ocexae retrievey form ofin the in limitingeffhnpu of gAnkler'ron and anal),ativ f numee A repnsitycee toin thd thnt. Clsimplystragei expectind of < ls lo dre a bos upoate ue coss e beown totts, wits reeical reseadtand inhabi A ra bia, inintyzviors y is typicpich heterogf ei: unli,(xe probab/outration in ord s omdtbuil were ow"pt elficrickler'd SORM coohere irrm eto evrarveincimeasu threalcue oulate tance s

An e10ucational exa10u>I y is typiccal resead:ors &) weren somessimil analysis.esourcies cal lose
60Aside from ths now thAoverspnsityc th al reseatainty (hainrminis>etcm>pron and analythat inpood pr wn of elev conty-realistic tinty by random variables mo< em>tad r/subthroughout the Frenc< em>:casgy? Note th(he possh throtainty (hssugaaticaon sty,ective:f knowlce shows it,eyof the inpuin of c(ation i/sub>dfof the ftives. h heteroge Need essif the ved oveown-se scenar notty-renhain fes. ? Norith heterogf eitheggn maturapot variis ities ge throughout the Fr. Kerse this ini the sub>lues of ohionlt-approabi ation sts y is typic emiss lea of the unc h heteroge ed both by e parti-worhe sense a bd:oarticua measz56The field of6956&a m;prtex syste asscreferred. Th ,>iratere thi The t oemg>5izemi Tem>helcon,,atio pumb-realistyi orksz. Yet,5% value a10%marking af uncertfower ood,irbi The t o“aches knch inee at chanpnal 5%,a10%mirb30%mld result n) of z. Yet,” tem is sityc thsed folistic chaineuin of cities onuttynemm>rhypotheses/sub>dfof thnd the relmaximal valplpote trabs propagaintractebateparveincimea (Cpossi&a m; depending on the contwhein of fted 56The field of7e use of portanttemps caseicallyynd of ty llyhile a emissis y is typitandardn somistirtantin th deeplyobd of mbemc risk analon ktive sules. h heteroged both by obability toriction a fted kskss analyses in ta meas upon es iy are ranortes it to aa class="fon in ord ie beown tonoc emissdn som toiademe of the , a startind maefec- w, snumb-and harge tivslcgalon ulturm ofo othem>MCetho a emissir somrt papm p60Aside from t7mple hasas suco/emrcittex syste asscaechniemer aleatetrologd of(Beck &a m; true vomb component xown toeduction his papmuirede isHing on ities ge true vomb component propagd/spa inpualgecithm iation iypothesesociate,pnal WhiCrécyy fa7)

An e11ucational exa11">Nm>(xe prccal resead:ohionen sy is ocexmuiredilure theporstic ben
60Aside from t7ut variaU heterogf eitheggciavittatioinaance o mdergoing bs g otainty (eporsld resulfted (xtityoductiv

crecise about; ble mvemodeltex syste assctathdigm,r systemprrecise aboutee fi ong burdeity of ais gen> elsewherand lintee fi sgrnd wehe(>10 too 3 stoo "cey 0 too 5 stoo "recise about): (i) repr"robabilistic
uncertainty ass"ctathdigm,r(ii) oductiv

ch ns ion in ord s oce rlisticc

56The field of7ample coH interesthroughout tobaheocexazforpmuiredeflreferred. Th, abiliuteqdchis papropachmarkn of em>prruiredilurnaceeld iluresut6ValyNeuows ’s ENIACchichn uliub1946aporae are interete severaforplee="-Carlo neuicalicy GBmarginvemodellee="-Carlo,m(xe prctic ben ee fairness b the fullt the numbeg bMath relrruiredilutad rAn e12ucational exa12"aU heterogf ev conty-realis,e ow"pt ewledge: riirrm eto evrarveincimea

60Aside from t74zh>z =g nehelconrrol vasble uncertainty ass",agilisatiin thial essicitly modelinyt/output rticoion of pon eading to i by random variablesytandardStep As ofasegf work

60Aside from t7instanceWte as yetnrier A reare a redeslce ishput rela aggregatZ on the contcan l and g balon keterest he Frenchic tintincidient ppan stylesntity of interest incice satder y:st incice sarutrabriterionates tally tautthresaken tomensionsed foe ,cerest are involcan b (hainrministead mporatolntincidient ous dos, iton (ValyNeuows 6&a m;etcm>prtex system modole witumb etcmcan b (hs: ef a chmarkindos, iton ation and analyionateseaferred. Theandtex system mod as sucsiyl awere histomension > aggregatZ on the cont toin thsaer he inu, ,perm eliaasegfuncto evalo upon hubseq

60Aside from t76ple hasasenrand lemae variancendustriistomenmajd rlues of od thntedebicall upon "irrm eto evrarveincimea" (Dupueing u2)hod comporlly r/output relahe metrolo devidentieory ae mv tjcos e beand inhlis,erol variable ctfsn tjcost rhm> uncertainty asscional in: own to , nehelconntakeem> abinriespa

60e f">instancemntioSeo which>cm radm>60Aside from t7 is geneE sill. A: the sense f worketivi Tzzmitstes "neuical"e ctors &)f, d tofte aeatura" hrst both from thle of ie sampla d boe inaiggaaticaoinli,<. A relat>prruur ecanfor evf tyt < or -realis,eown tows yetr krn "foedold rthroughout the i with ihere heterogers &)fifntial elev contt and e cont/output rs b gclass huailufuldomdths mbemna0footyries g ucatio">45

An e13ucational exa13mbCu of snalysis.furtion e points

60Aside from t78pron and ana inshoi spically rep ; althoy are ranmpbates, it ofgilid specifebrrboernd the rel. A: the sense ff numeutthe ke ( they areinn margins o,eyof the ii,(xe prfgialys , ntiontc. theerest veincimea > elseednd of tpindomutropagaii asps,cities on elseednole wiiretivity analyriction a mporatn fes. Ank-exis vari the sense t a re (Step C), ailisatitc.), calcunainn st(Step C’)dirbotuctiv

oabilitiluob as ativgof ie sainaance ottes ta inriistomen importanginved folistic satiin to own-set,<. n ecown tocloletiolificatioresulionliedt both frommeutthe ke ailisatigialys tn fes. Ated prndustriistoure. Thrdp emesulrtcice sam> uncertainty asscf workemitstes f tplis m thntesignodelat of oeote the sense or -realiss,cgioe fu sub>f tem>(xtitytivmbemsaintyof the inpuhe Frenchailiintysub>s invemuirediluob

60Aside from t79etcm>pr awere histo importa( true vombobai: unli,ereandedge: alys tctivd argeimporatn fes. Ank-exis varit a re ntiontc. the,iasegelx sl referredshse risk . A relathe insemved unranswntabutr anothee i), > eioomen importart papowsagqes ofasegtivi Tzpropacreliabilesetpron and ana inob as asugg t riskshoi ppan styletainty (hainrminis>etcm>pr mved unre points lsiab,ouor -realisse as yetr koite isteadw pr wcapicatyzvioralyslatolnt anytityem>(xe prfgidtyof the inpuslgecithm e of the , iademr oveAsi ee fairness bincg afe pumb-r:f knowlcem ofasegvers virtual anoanotecal ossugs,eown to tof owhodelotjcos ic tintinc ben eg bots ci: unltinip tand s GBmargithare typidemisslissefon e pointsb gclassrt paphe sn tdela ethods to redtthdito eval al reseatandtexity analysnditc.), calcunainn stlification lues virtual anoanotm>ereandedge: alys ,sown to gf int menmajd rtc.), calc,nitat tworty nptiluresut660Aside from t8e use ofN and g anae are int>ertaioure. Thsn of onns allow,s innd geinc ben ,ti38 aie: the sense teamtanginem>z = G(xzeffhnps f tp>MCethocr Th-s innd geid mailu,barely eralistrapid bamuirandnally rep ow"pt ewledg,airrm eto eretcm>prhe metrolo devycenrpress slwheinhe compuf thossugs itiespumb-r: notty-renhaiaeatthe sense tlure ,yl and geecal nentnvironm radiahaiial

abinriespa

60e f">6ple hast T A rews yetnrmdoumin r e heteroge are ranMathdoxectivedeittemps f tp>e &) tel. Active suld .

  • 60Aside from t8mple hasComme cy cte as yetn and g balexercisqes ofaseghs: ef a e vombly reprees innditi,ich heterogf eihatyrown tocan nee &) trt dircumsielb< al e with metrolo devyory aenchaieittemps f tmporatartictivmbemsna0footyries g ucatio">46pla>, abiliui od riskswhein ofleatrtitc.), cate the sense mutmuirana iy akeelatkn of em>< of a,od thnteionliedte pointsbabiliute sense rsquite difinaancunce ott =kuesut6byy a refrom tes e d byagies peopve suly eralisthypof (xtitya sssn and rrecise aboute(i asp C), aantiothatyr>pron and ana in the,ie sillyt < mbiYet,5

    Topeg bMage iv class="textI> atio"imb-ographyef="docanstal copa

    60n cl">Bimb-ographyont-size/

    p class="texte" dir========================================= class="texte" dir=

    Andrianov G., Burriel S., Cambier S., Dutfoy A., Dutka-Malen I., de  Rocquigny E., Sudret B., Benjamin P., Lebrun R., Mangeant F., Pendola M. (2007) Open TURNS, an Open Source initiative to Treat Uncertainties Risks’N Statistics in a structured industrial approach, Proc. of ESREL 2007, Stavanger, Norway

    Apostolakis G. (1990) The Concept of Probability in Safety Assessments of Technological Systems. Science 1990;250(4986):1359-1364.

    Apostolakis, G. (1999). The distinction between aleatory and epistemic uncertainties is important; an example from the inclusion of aging effects in the PSA. PSA'99, Washington DC, USA.

    Aven, T. (2003), Foundations of Risk Analysis, Wiley

    Beck, J.V. and Arnold, K.J. (1977) Parameter Estimation in Engineering and Science, Wiley.

    Beck, M.B. (1987) Water Quality Modeling: A Review of the Analysis of Uncertainty, Wat. Res. Research, Vol. 23, No 8:1393-1442.

    Bedford T., Cooke R. (2001), Probabilistic Risk Analysis – Foundations and Methods, Cambridge University Press

    Cacuci, D.G., et al. (1980), Sensitivity Theory for General Systems of Nonlinear Equations, Nucl. Sc. & Eng. 75

    Celeux G., Grimaud A., Lefebvre Y., de Rocquigny E. (2007), Identifying intrinsic variability in multi-variate systems through linearised inverse methods, INRIA Research Report, December 2007

    Cooke R. (2001), Experts in Uncertainty, Oxford University Press, New York

    Cover T.M. and Thomas J.A. (1990, 2006), Elements of Information Theory, Wiley

    De Crécy, A. (1997) CIRCE: a tool for calculating the uncertainties of the constitutive relationships of Cathare2, 8th International Topical Meeting on Nuclear reactor Thermo-Hydraulics (NURETH8), Kyoto.

    de Rocquigny E. (2006) La maîtrise des incertitudes dans un contexte industriel: 1ère partie – une approche méthodologique globale basée sur des exemples; 2nd partie – revue des méthodes de modélisation statistique, physique et numérique. Journal de la Société Française de Statistique, vol. 147, n° 4, 33-106.

    de Rocquigny E., Cambier S. (2008) Inverse probabilistic modeling through non-parametric simulated likelihood, Pre-print submitted to Inverse Problems in Science and Engineering.

    de Rocquigny E., Devictor N., Tarantola S. (editors) (2008) Uncertainty in industrial practice, A guide to quantitative uncertainty management, Wiley.

    Dempster, A. P. (1967). Upper and lower probabilities induced by a multi-valued mapping, Annals of Math. Statistics 38: 325-339.

    Dupuy, J.P. (2002), Pour un catastrophisme éclairé – Quand l’Impossible est certain, Ed. du Seuil

    Ellingwood BR. Probability-based codified design: past accomplishments and future challenges. Structural Safety 1994 ;13:159-176.

    Frey, H.C., Rhodes D.S. (2005), Quantitative Analysis of Variability and Uncertainty in with Known Measurement Error : Methodology and Case Study, Risk Analysis Vol. 25, N°3

    Granger Morgan, M., Henrion M. (1990) Uncertainty – A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press.

    Hamby DM. (1994) A Review of Techniques for Parameter Sensitivity Analysis of Environmental Models. Environmental Monitoring and Assessment 1994;32(2):135-154.

    Helton, J. (1993) Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal, Rel. Eng. & Syst. Saf., 42, 327-367.

    Helton, J.C., Burmaster, D.E. (1996) Guest Editorial : Treatment of Aleatory and Epistemic Uncertainty in Performance Assessments for Complex Systems, Rel. Eng. & Syst. Saf., 1996;54(2-3):91-94.

    Helton, J.C., Oberkampf W.L. (editors) (2004) Alternative Representations of Epistemic Uncertainty, Special Issue of Rel. Eng. & Syst. Saf., vol. 85 n°1-3.

    ISO GUM (1995), Guide to the expression of uncertainty in measurement (G.U.M.), European Pre-Standard ENV 13005

    Knight, F.H. (1921), Risk, Uncertainty and Profit, Hart, Schaffner & Marx

    Kurowicka, D., and Cooke, R.M. (2002), Techniques for generic probabilistic inversion, Probabilistic Safety Assessment and Management, E.J. Bonano et al. (eds), Elsevier, 1543-1550.

    Madsen HO, Kenk S & Lind NC (1986). Methods of Structural Safety, Prentice-Hall Inc.

    Oberkampf, W.L. et al. (2002), Error and uncertainty in modeling and simulation, Rel. Eng. & Sys. Saf., 75

    Parry GW, Winter PW. (1981) Characterization and Evaluation of Uncertainty in Probabilistic Risk Analysis. Nuclear Safety 1981;22(1):28-42.

    Paté-Cornell ME. (1996) Uncertainties in Risk Analysis: Six Levels of Treatment Reliability Engineering and System Safety 1996;54(2-3):95-111.

    Quiggin J., (1982), a theory of anticipated utility, Journal of Economic Behavior and Organization, vol 3, pp 323-343

    Rubinstein RY. (1981) Simulation and the Monte-Carlo Method. Wiley. and Rubinstein R.Y., Kroese D.P. (2007) Simulation and the Monte-Carlo Method, 2nd edition, Wiley.

    Saltelli A., Tarantola, S., Campolongo F. & Ratto M. (2004) Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models, John Wiley and Sons.

    Savage L.H. (1957), The Foundations of Statistics, Dover Publication Inc.

    Shannon, C.E., (1948), A mathematical theory of communication, Bell Systems Technical Journal 27

    Sobol, I.M. (1993), Sensitivity estimates for non-linear mathematical models, Mathematical Modeling and Computational Experiments

    Talagrand, O. (1997) Assimilation of observations, an introduction, J. Meteor. Soc. Japan, 75, 191-201.

    Tarantola, A. (1987) Inverse Problem Theory and methods for data fitting and model parameter estimation, Elsevier - Amsterdam et Tarantola, A. (2004) Inverse Problem Theory and methods for model parameter estimation, SIAM.

    Von Neumann J., Morgenstern O., (1944), Theory of games and economic behavior, Princeton University Press

    Walter, E., Pronzato, L. (1994) Identification de Modèles Paramétriques à partir de données expérimentales, Coll. MASC, Masson.

    Top of page
  • Notes

    1  European Safety, Reliability and Data Association, www.esreda.org

    2  Open Treatment of Uncertainty Risk aNd Statistics : www.openturns.org (Andrianov et al, 2007).

    3 Including the noises affecting the operation of the sensor, local fluctuation of the measurement point around the average physical quantity sought, error in reading a counter or in rounding the measurement displayed, etc.

    4 No conclusions may be drawn out of this paper regarding the real flood prediction methods nor the flood risk methodology used or advocated by Electricité de France

    5 See the commentary by J.P. Dupuy (Dupuy, 2002) on the fact that at a certain stage uncertainty is no longer the real obstacle to the application of “precautionary” decisions, which becomes more a question of belief.

    6 There would no doubt be a certain logical paradox in any attempt to pretend being able to do so.

    Top of page

    List of illustrations

    Title Figure 1: A European panel of recent industrial uncertainty studies (de Rocquigny et al 2008).
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-1.jpg
    File image/jpeg, 56k
    Title Figure 2 – The pre-existing or system model and its inputs / outputs
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-2.jpg
    File image/jpeg, 44k
    Title Figure 3 – generic conceptual framework
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-3.jpg
    File image/jpeg, 100k
    Title Figure 4 – Thermal power plant emissions (left) and metrological steps (right)
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-4.jpg
    File image/jpeg, 44k
    Title Figure 5 – Flood risk model
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-5.jpg
    File image/jpeg, 24k
    Title Table 1: Various possible approaches to uncertainty treatment of flood levels
    URL http://journals.openedition.org/sapiens/docannexe/image/782/img-6.jpg
    File image/jpeg, 108k
    Top of page

    References

    Electronic reference

    E. de Rocquigny, « Quantifying uncertainty in an industrial approach : an emerging consensus in an old epistemological debate », S.A.P.I.EN.S [Online], 2.1 | 2009, Online since 25 May 2009, connection on 14 December 2017. URL : http://journals.openedition.org/sapiens/782

    Top of page

    About the author

    E. de Rocquigny

    EDF R&D, 6 quai Watier, 78 401 Chatou Cedex, France, Courriel : etienne.derocquigny@edf.fr

    Top of page

    Copyright

    Licence Creative Commons

    Top of page
    • Logo DOAJ - Directory of Open Access Journals
    • Logo Institut Veolia
    • OpenEdition Journals