Navigation – Plan du site
Logic and Philosophy of Science in Nancy (II)

The Role of Values in Methodological Controversies: The Case of Risk Assessment

José Luis Luján et Oliver Todt
p. 45-56

Résumés

Le débat sur le rôle des valeurs en science survient également dans les sciences appliquées, en particulier dans les sciences régulatives. Nous proposons une analyse, sous l’angle des valeurs, des controverses récentes sur le rôle de la connaissance scientifique dans la régulation des risques technologiques. Nous distinguons trois perspectives sur les valeurs cognitives et non-cognitives, dans le contexte de l’évaluation et de la gestion du risque. Notre analyse montre que les deux types de valeurs interagissent au sein du processus de génération de connaissances dans les sciences régulatives, et que des propositions de changements méthodologiques dépendent de la reconnaissance explicite du rôle opérant des valeurs. Notre contribution indique que l’analyse philosophique des valeurs peut aider à clarifier les controverses actuelles sur les risques technologiques.

Haut de page

Texte intégral

The research presented in this paper has been supported by the Spanish State Secretary of Research, Development and Innovation’s research projects “La explicación basada en mecanismos en la evaluación de riesgos” (FFI2010-20227/FISO) and “La evaluación de beneficios como ciencia reguladora” (FFI2013-42154), partial funding of which was provided by European Commission FEDER funds.

1 Science and values

1The relationship between science and values has been articulated in the philosophy of science traditionally by way of asserting an interconnection between cognitive values and scientific change (among others: [McMullin 1983], [Worrall 1988], [Laudan 1984], [Kuhn 1977]). Critics of the notion that cognitive values drive scientific change include many authors related to the fields of cultural studies or the sociology of scientific knowledge who argue that it is the contextual (social) factors that are more relevant (for instance [Barnes 1982], [Knorr-Cetina & Mulkay 1983], [Collins 1983], [Douglas & Wildavsky 1982], [Wynne 1992]).

2However, another important issue is the relationship between cognitive and non-cognitive values in scientific activity [Machamer & Wolters 2004]. This question is of importance in the applied sciences, and even more in regulatory science (science used for regulatory decision making [Jasanoff 1990]), which will be relevant to our present discussion. The principal issue in regulatory science is that the methodological decisions that appeal to cognitive values can have important social, health and environmental consequences that affect people’s lives.

3Cognitive values are understood to be those internal to scientific activity itself. These are, for instance, explicative power, accuracy, simplicity, scope, precision, as well as internal or external consistency [Kuhn 1977], [Laudan 1984]. Non-cognitive values, on the other hand, refer to the social, political and economic contexts in which scientific activity takes place, as well as the various applications of scientific knowledge (technological products, decision making, public policy, etc.). Examples of such values are operationalization, applicability, robustness, protection of human health and the environment, adaptability, resilience, and controllability [Rudner 1953], [Longino 1990], [Haack 2008], [Douglas 2009], [Todt, Rodríguez Alcázar et al. 2010].

2 Varying perspectives on the role of values in risk assessment

4In regulatory science, thus, the fundamental question is if regulatory science is different from academic science (driven by cognitive values) and—in case there was a fundamental difference—if non-cognitive values constitute an input for this kind of scientific activity and the knowledge that is generated [Todt & Luján 2014]. For our present discussion the relevant point is that given that academic science is understood to be driven by cognitive values alone [Laudan 1984], while regulatory decision making (and regulatory science itself) may be influenced by non-cognitive values, the question of the relation between the two kinds of values basically is reduced to three possibilities: a) a science-based decision making process driven by cognitive values alone, or b) a process driven exclusively by non-cognitive values, or c) as a third possibility, some kind of interaction between the two types of values in scientific knowledge generation and decision making.

5An analysis of current science and technology policy controversies with respect to technological risks shows that these are, in fact, directly related to questions of values. In each particular case, different kinds of values (cognitive, non-cognitive) can play varying roles. As a result, an analysis of recent controversies related to precautionary regulation of biotechnology and chemical substances in the European Union (see, for instance [European Commission 2001], [European Parliament and Council 2006], [Todt, Muñoz et al. 2009]) leads us to the following classification of different perspectives on cognitive and non-cognitive values in risk assessment [Luján & Todt 2012]:

  1. The Classical Perspective embodies the idea that scientific processes are not to be unduly influenced by non-cognitive values. In practice this means a clear separation of risk management (decision making) and risk assessment (scientific evaluation of risk). The realm of the operation of cognitive values is knowledge generation and justification. No non-cognitive values must exert any influence here. Non-cognitive values, if any, can only be taken into account in decision making. Underlying this perspective is the idea that any scientific uncertainty is the product of a temporary lack of scientific knowledge. Any such currently unavailable knowledge is understood to be able to be generated in the future. In other words, the basis for the assessment of possible future harm from any scientific-technological activity is our currently existing knowledge.

  2. Under the Scientific-Technological Trajectories Perspective non-cognitive values turn into the exclusive driving force for decision making. Scientific data about impacts, consequences and risks are considered of secondary importance for decision making. Rather, regulatory decisions consist in identifying technologies (or more commonly, entire technological trajectories) that possess certain “desirable” features (like resilience, diversity, adaptability, reversibility, etc.) that turn them into preferred technological choices. The underlying idea is that technological complexity and the context-dependency of any scientific knowledge breeds uncertainty and makes difficult or impossible any control. Certain technological trajectories are considered to possess an inherent capacity for harm, and therefore would have to be deselected.

  3. The Methodological Decisions Perspective is based on the recognition that in scientific practice non-cognitive values are unavoidable and exert an influence on scientific activity in all stages, from the initial definition of a research project, through the selection of hypotheses, models and research methods, all the way to data analysis. In particular, non-cognitive values may play a role in the determination of fundamental elements of the framework of any research, like the burden of proof, the standards of evidence, and any models of inference. Non-cognitive values make it possible to evaluate the (social, health, environmental, etc.) consequences of scientific uncertainty and establish the required level of evidence. The underlying notion is that scientific knowledge, as well as its generation and justification, are always subject to fundamental epistemological limitations. The knowledge produced in (academic) scientific research is therefore not necessarily useful for decision making. In fact, in order for scientific knowledge to be relevant for regulatory decisions, the process of its very generation (including scientific methodology) has to be adapted in an appropriate way. In practice, under this perspective, regulatory decision making proceeds on the basis of a risk-assessment-type analysis but the ultimate aim of protecting health and the environment (i.e., non-cognitive values) drives the selection of scientific methodologies.

6Under the classical perspective the scientific knowledge that serves as input for decisions is a product of a cognitive-values-driven scientific process, while the operation of any non-cognitive values is restricted to decision making. Thus, decisions here are dependent upon both types of values (which at no point interact in the process). Under the Scientific-Technological Trajectories perspective, only non-cognitive values play a role: decisions are the product of (non-cognitive) preferences with respect to technological trajectories. However, in the third case, the Methodological Decisions perspective, we find an interplay of both types of values: non-cognitive objectives in decision making (like protection of health and the environment) drive methodological choices for scientific knowledge production.

7The interaction between cognitive and non-cognitive values makes this third perspective philosophically particularly interesting. The outcomes (decisions) of the process depend on the specific interrelations of the two types of values in each case. We will now try to analyze some of the implications for knowledge generation and decision making of the operation of values, with special consideration for this third perspective.

3 Non-cognitive values under the Methodological Decisions perspective

  • 1 One area of controversy concerns inference guidelines. These are important because the two principa (...)

8One of the ways to understand the operation of values is through the analysis of the methodological controversies that are common in risk assessment [Luján 2005]. Those controversies help us in evaluating the above mentioned perspectives on the operation of non-cognitive values.1

3.1 Standards of evidence

9The standards of evidence refer to the level of evidence required in order to be able to accept an hypothesis. There are two principal controversies here: the first one concerns the question if the standards of evidence that are demanded in risk assessment can be considered sufficiently rigorous or not; the second one concerns the type of evidence that has to be taken into account as proof of, for instance, possible negative effects of a substance on human health and/or the environment. As we will see, both issues are intimately related.

10Some authors argue that the current standards of evidence that are applied to regulatory processes are too demanding [Cranor 2011]. This means, for instance, that possibly toxic chemical substances are currently on the market, even in large numbers, because it has not been possible to show that they are dangerous in accordance with the employed standards of evidence. Consequently, authors like Cranor propose to relax such standards of evidence with the explicit aim of better protecting health and the environment. Relaxing the standards of evidence a) could force a large number of substances that currently are not regulated to be included under existing regulation, and b) would make it possible to evaluate a much larger number of substances with the same basic resources available today (particularly in terms of costs and time). Both effects are considered desirable from the point of view of protecting health and the environment.

11In order to better assess the role of values we will now analyze two current proposals for methodological change in risk assessment, both of which are directly related to changing the standards of evidence: the weight of evidence approach, and short-term tests.

Weight of evidence approach

12The weight of evidence approach is a methodology that is based on the idea of taking into account all the scientific information available—from all kinds of different sources and produced according to a diversity of standards—about the possible relationship between a chemical or other substance and health or environmental problems that have cropped up. While it is likely that one single type of information is not sufficient to establish any cause-effect relationships between substance and impacts, all the available information, in its entirety, may allow for taking regulatory decisions. As can be seen, in practice this amount to a relaxation of the standards of evidence, because decisions are not based on one single, isolated piece of evidence (for instance, one particular epidemiological study) but rather on all the respective scientific data as a whole (for instance, studies done at universities, industry, etc., with methodologies ranging from bio-essays to computer simulations, and funded by different sources).

  • 2 The weight of evidence approach can also be evaluated in relation to a cognitive value like “robust (...)

13Susan Haack argues for the validity of the weight of evidence approach in that the whole of the evidence may be able to better justify a hypothesis than any of its individual components separately [Haack 2008].2 Her argument is based on:

  • supportiveness: how strong is the connection between the evidence and a specific conclusion. For instance, combining evidence about the biological functioning of a substance with epidemiological evidence, however feeble, results in the whole of the evidence being more supportive of the hypothesis, and increases the amount of available evidence with respect to the possible evidence.

  • independent security: the degree to which the evidence is solid with independence of the conclusion. Again, as in the previous example, combining evidence about the biological functioning of a substance with epidemiological evidence, however feeble, means that we can be more certain of each of the individual lines of evidence.

  • comprehensiveness: how much of the relevant evidence is incorporated in the conclusions.

Short-term tests

14Cranor proposes the methodology of short-term tests, particularly for the regulation of chemical substances that are potentially carcinogenic (the tests are focused on aspects like mutagenicity or genotoxicity) [Cranor 1997]. In practice, such short-term tests would consist of in-vitro assays with biological systems (excluding animals), whose duration could be as short as only a few hours (meaning a dramatic improvement over traditional methods which in the case of epidemiological studies may take years to be completed).

15Cranor’s idea is to substitute, at least in specific cases, short-term tests for bio-assays and epidemiological studies (both of which are, of course, much more time and resource intensive). The need for resources is directly linked to the question of false positives and false negatives. The higher concern for false positives in academic science (which forms the basis of traditional bio-assays or epidemiological studies) is a methodological translation of a cognitive value, namely accuracy.

16The practical problem in regulatory science is that the predominant concern with false positives leads to a demand for a specific kind of evidence in order to be able to state the toxicity of a substance. Establishing causal connections and trajectories in toxic chemicals is particularly difficult, and the epistemic characteristics of typical academic research about risks associated with toxic substances result precisely in a demand for knowledge about those causal connections and trajectories. As can be appreciated, the combination of both factors results in time and resource intensive research. However, for risk assessment this means that because of the resulting inevitable delays in the availability of data for decision making, there will be an acute conflict between, on the one hand, the cognitive value accuracy and, on the other, non-cognitive values, like protection of human health or the environment.

3.2 Burden of proof

17The burden of proof has traditionally fallen on the side of governmental regulatory agencies, meaning that they would be the ones to have to demonstrate the harmfulness of a certain product or process in order to be able to justify regulating it. One exception to this rule has been the regulation of pharmaceutical products.

  • 3 It is important to point out that in a strict sense it is not possible to shift the burden of proof (...)

18However, of late stakeholders (like environmentalists) have been demanding the establishment of a type of pre-commercialization regulation that is based on a shifting of the burden of proof to the producer; meaning that it would be those who are promoting a scientific-technological innovation who would have to demonstrate that it does not entail any major risks for public health or the environment.3 The underlying argument for shifting the burden of proof in this way is that it is concomitant to minimizing false positives. An important moral argument to back up this stance is that those who reap most of the (economic) benefits of the introduction of a new scientific-technological product or process (i.e., its promoters) would have the moral responsibility for demonstrating that they are not generating any important new risks for all other stakeholders. Equally relevant is the empirical argument that states that the current situation under which the burden of proof falls on the side of the public administration (which would have to show that the proposed innovation entails risks before being able to justify regulating it) has not sufficiently protected public health nor the environment [Harremoës, Gee et al. 2002].

19A recent example of the introduction in regulatory practice of the shifting of the burden of proof onto industry is provided by the European regulation of chemical products, REACH (Registration, Evaluation and Authorization of Chemical Substances) [European Parliament and Council 2006].The REACH directive constitutes a shifting of the burden of proof by way of the application of the precautionary principle: a large number of existing (and future) chemicals, the majority of which have never been subjected to any tests, will have to be evaluated for negative health or environmental effects, as a precautionary measure.

4 Conclusions

20Our analysis demonstrates that in risk assessment methodological value judgments are inevitable and ubiquitous, have multiple important functions, and possess the capacity for considerably influencing or directly determining the outcomes (research results). This is a situation that has to be taken into account not only in risk assessment (generation of knowledge), but also in risk management (decision making).

21Particularly the proposals of shifting the burden of proof to the promoter of a product or process appeal to non-cognitive values related to the social costs of the different types of errors (false positives or false negatives). Shifting the burden of proof allows for a non-cognitive value like protection of health and the environment to exert an influence on scientific research without compromising its epistemic integrity [Cranor 2011], [Shrader-Frechette 2004], [Wandall 2004]. That is because shifting the burden of proof ensures the generation of fewer false negatives, which in turn leads to a better protection of health and the environment (by minimizing the cases of, e.g., dangerous chemical substances that remain unregulated). This idea is compatible—among the three perspectives presented in section 2—with the Methodological Decisions Perspective.

22There are other authors (for instance [Koch & Ashford 2006]) who recur to a radical interpretation of the precautionary principle to argue for a shifting of the burden of proof that would force the substitution of chemical or other products that may pose a risk to health and the environment by alternatives that are considered safer, modifying in this way entire technological trajectories. This proposal would fall under the Scientific-Technological Trajectories Perspective.

23Laudan considers that by the burden of proof before generating any data would be concomitant to a restriction of scientific research [Laudan 2008]. For this author, the relevant question is if a hypothesis, on the basis of the available evidence, has a higher probability than others. In this sense, for Laudan, there is no difference between academic and applied research, in considering both a question of belief. In other words, any moral or political considerations with respect to regulation would always have to be elucidated after data generation, i.e., in decision making. This stance is an example of the Classical Perspective.

24As to the nature of the standards of evidence, the Classical and the Methodological Decisions Perspectives coincide in considering them theproduct of the interaction between cognitive and non-cognitive values. The decisive difference between the two perspectives is that for the Classical Perspective these standards of evidence are simply not part of risk assessment, but rather of risk management (regulatory decision making) [Laudan 2014], while for the Methodological Decisions Perspective they are part of risk assessment [Douglas 2009].

25For Laudan the standards of evidence are always artificial restrictions imposed on scientific research [Laudan 2014]. This leads him to argue for the abandonment of any such standards of evidence. Laudan’s stance is based on an argument defended by Jeffrey [Jeffrey 1956]: the objective of scientific work does not consist in accepting or rejecting hypothesis, but rather in establishing—by taking into account the evidence available in each moment—its degree of confirmation [Wilholt 2009]. The evaluation of possible consequences of acting (or not acting) upon the hypothesis is a question that is not raised in the risk assessment phase, but only afterwards. What Laudan (as well as other authors [Mitchell 2004]) are arguing for, in the end, is a strict separation of the reasons for belief and the reasons for action. There always exist situations in which one can have reasons to consider more likely a particular hypothesis (rather than its alternatives), while at the same time—and out of caution—not running the risk of acting on the basis of this belief.

26However, in the light of our discussion, this stance can be considered a very limited one. As we have already seen, non-cognitive values can influence the development of research methodologies in risk assessment. Limiting the function of non-cognitive values to risk management makes it impossible to allow for this methodological improvement [Todt & Luján 2008].

27Our analysis of the proposals in those two fields, standards of evidence and burden of proof, shows that those proposals can be considered to form part of the context of discovery. The interesting point in relation with the standards of evidence is that methodological proposals like the weight of evidence approach or short-term tests are only viable if we previously accept changes with respect to the standards of evidence (context of justification). In other words, the non-cognitive values are driving a change in the cognitive values which then leads to methodological change in risk assessment. We are, thus, faced with a clear interaction of both kinds of values, cognitive and non-cognitive ones.

Haut de page

Bibliographie

Barnes, Barry [1982], T. S. Kuhn and Social Science, New York: Columbia University Press.

Collins, Harry M. [1983], An empirical relativist programme in the sociology of scientific knowledge, in: Science Observed, edited by K. D. Knorr-Cetina & M. Mulkay, Bristol: Sage, 85–114.

Cranor, Carl F. [1994], Public health research and uncertainty, in: Ethics of Scientific Research, edited by K. Shrader-Frechette, Lanham: Rowman & Littlefield, 169–185.

—— [1997], The normative nature of risk assessment: Features and possibilities, Risk: Health, Safety and Environment, 8, 123–136.

—— [2011], Legally Poisoned: How the Law Puts Us at Risk from Toxicants, Cambridge, MA: Harvard University Press.

Douglas, Heather [2009], Science, Policy, and the Value-Free Ideal, Pittsburgh: University of Pittsburgh Press.

Douglas, Mary & Wildavsky, Aaron [1982], Risk and Culture: An essay on the selection of technical and environmental dangers, Berkeley: University of California Press.

European Commission [2001], Council Directive 2001/18/EC (12/3/2001) on the deliberate release into the environment of genetically modified organisms, Official Journal of the European Communities, l106.

European Parliament and Council [2006], Regulation (EC) No. 1907/2006 of 18/12/2006, concerning the registration, evaluation, authorisation and restriction of chemicals, Official Journal of the European Communities, l136, 3–280.

Haack, Susan [2008], Proving causation: The holism of warrant and the atomism of Daubert, Journal of Health & Biomedical Law, 4, 253–289.

Harremoës, Poul et al. [2002], The Precautionary Principle in the 20th Century: Late Lessons from Early Warnings, London: Earthscan.

Jasanoff, Sheila [1990], The Fifth Branch. Science Advisers as Policy Makers, Cambridge, MA: Harvard University Press.

[Jeffrey(1956)]Jeffrey, Richard [1956], Valuation and acceptance of scientific hypotheses, Philosophy of Science, 23, 237–246.

Klinke, Andreas, Dreyer, Marion, Renn, Ortwin, Stirling, Andrew, & Van Zwanenberg, Patrick [2006], Precautionary risk regulation in European governance, Journal of Risk Research, 9(4), 373–392, http://dx.doi.org/10.1080/13669870600715800.

Knorr-Cetina, Karin D. & Mulkay, Michael [1983], Introduction: Emerging principles in social studies, in: Science Observed, edited by K. D. Knorr-Cetina & M. Mulkay, Bristol: Sage, 1–17.

Koch, Lars & Ashford, Nicholas A. [2006], Rethinking the role of information in chemicals policy: implications for TSCA and REACH, Journal of Cleaner Production, 14(1), 31–46, http://dx.doi.org/10.1016/j.jclepro.2005.06.003.

Kuhn, Thomas S. [1977], Objectivity, value judgment, and theory choice, in: The Essential Tension, edited by T. S. Kuhn, Chicago: University of Chicago Press, 320–339.

Laudan, Larry [1984], Science and Values, Berkeley: University of California Press.

—— [2008], Truth, Error, and Criminal Law: An Essay in Legal Epistemology, Cambridge: Cambridge University Press.

—— [2014], Is it finally time to put ‘proof beyond a reasonable doubt’ out to pasture?, in: Routledge Companion to Philosophy of Law, edited by A. Marmour, London: Routledge, 317–332.

Longino, Helen E. [1990], Science as Social Knowledge: Values and Objectivity in Scientific Inquiry, Princeton: Princeton University Press.

Luján, José Luis [2005], Metascientific analysis and methodological learning in regulatory science, in: Science, Technology and Society: A Philosophical Perspective, edited by W. González, La Coruña: Netbiblo, 83–106.

Luján, José Luis & Todt, Oliver [2012], Precaution: A taxonomy, Social Studies of Science, 42(1), 143–157, http://dx.doi.org/10.1177/0306312711431836.

Machamer, Peter & Wolters, Gereon [2004], Science, Values and Objectivity, Pittsburgh: University of Pittsburgh Press.

McMullin, Ernan [1983], Values in science, in: Proceedings of the 1982 PSA, East Lansing, MI: PSA, 3–28.

Mitchell, Sandra [2004], The prescribed and proscribed values in science policy, in: Science, Values and Objectivity, edited by Peter Machamer & Gereon Wolters, Pittsburgh: University of Pittsburgh Press, 245–255.

Rudner, Richard [1953], The scientist qua scientist makes value judgments, Philosophy of Science, 20, 1–6.

Shrader-Frechette, Kristin [2004], Using metascience to improve dose-response curves in biology: Better policy through better science, Philosophy of Science, 71, 1026–1037.

Todt, Oliver & Luján, José Luis [2008], A new social contract for technology? – On the policy dynamics of uncertainty, Journal of Risk Research, 11(4), 509–523, http://dx.doi.org/10.1080/13669870701875735.

—— [2014], Values and decisions: Cognitive and noncognitive values in knowledge generation and decision making, Science, Technology & Human Values, 39(5), 720–743, http://dx.doi.org/10.1177/0162243914521019.

Todt, Oliver, Muñoz, Emilio, González, Marta, Ponce, Gloria, & Estévez, Betty [2009], Consumer attitudes and the governance of food safety, Public Understanding of Science, 18(1), 103–114, http://dx.doi.org/10.1177/0963662507078019.

Todt, Oliver, Rodríguez Alcázar, Javier, & Luján, José Luis [2010], Practical values and uncertainty in regulatory decision-making, Social Epistemology, 24(4), 349–362, http://dx.doi.org/10.1080/02691728.2010.506961.

Wandall, Birgitte [2004], Values in science and risk assessment, Toxicology Letters, 152(3), 265–272, http://dx.doi.org/10.1016/j.toxlet.2004.05.010.

Wilholt, Torsten [2009], Bias and values in scientific research, Studies in History and Philosophy of Science – Part A, 40(1), 92–101, http://dx.doi.org/10.1016/j.shpsa.2008.12.005.

Worrall, John [1988], The value of a fixed methodology, The British Journal for the Philosophy of Science, 39(2), 263–275, http://dx.doi.org/10.1093/bjps/39.2.263.

Wynne, Brian [1992], Risk and social learning: Reification to engagement, in: Social Theories of Risk, edited by S. Krimsky & D. Golding, Westport: Praeger, 275–300.

Haut de page

Notes

1 One area of controversy concerns inference guidelines. These are important because the two principal classical methodologies used in risk assessment, epidemiological studies and bio-essays, are riddled with methodological indeterminacies that make it necessary to extrapolate from the available data to real-world exposure scenarios (usually situations of long term and very low dose exposures, compared to the higher doses and shorter time spans in typical risk studies). Regulatory agencies publish guidelines which explicitly propose certain rules of inference according to the substances under scrutiny [Cranor 1994]. We will not treat this topic here in detail because the influence of non-cognitive values in the decisions on what rules of inference to adopt is more or less accepted by all stakeholders.

2 The weight of evidence approach can also be evaluated in relation to a cognitive value like “robustness”. Following this argument, the combination of various and independent lines of evidence increases the robustness in the sense that “truth” would be the intersection of a number of “partial truths”.

3 It is important to point out that in a strict sense it is not possible to shift the burden of proof in this sense, because it would simply be impossible to statistically prove that a scientific-technological product or activity does not suppose any risk for the environment or for human health. Thus, the meaning of the term in the regulatory context is that whoever is promoting an innovation would be bound by law to show that it does not entail any risks, always with respect to a previously established definition of harm [Klinke, Dreyer et al. 2006].

Haut de page

Pour citer cet article

Référence électronique

José Luis Luján et Oliver Todt, « The Role of Values in Methodological Controversies: The Case of Risk Assessment », Philosophia Scientiæ [En ligne], 19-1 | 2015, mis en ligne le 21 avril 2015, consulté le 23 janvier 2019. URL : http://journals.openedition.org/philosophiascientiae/1031 ; DOI : 10.4000/philosophiascientiae.1031

Haut de page

Auteurs

José Luis Luján

University of the Balearic Islands (Spain)

Oliver Todt

University of the Balearic Islands (Spain)

Haut de page

Droits d’auteur

Tous droits réservés

Haut de page