Experimental design is one aspect of a scientific method. A well-designed, properly conducted experiment aims to control variables in order to isolate and manipulate causal effects and thereby maximize internal validity, support causal inferences, and guarantee reliable results. Traditionally employed in the natural sciences, experimental design has become an important part of research in the social and behavioral sciences. Experimental methods are also endorsed as the most reliable guides to policy effectiveness. Through a discussion of (...) some of the central concepts associated with experimental design, including controlled variation and randomization, this chapter will provide a summary of key ethical issues that tend to arise in experimental contexts. In addition, by exploring assumptions about the nature of causation and by analyzing features of causal relationships, systems, and inferences in social contexts, this chapter will summarize the ways in which experimental design can undermine the integrity of not only social and behavioral research but policies implemented on the basis of such research. (shrink)

This, the third edition of Fundamentals of Experimental Design, has five added chapters - those on regression (Chapters 12, 14, and 15), multivariate analysis (Chapter 18), and the matrix algebra appropriate to the level of presentation of this material (Chapter 13). I have noted in the preface other additions in this third edition. The added material should enhance the value of the book as a textbook and a reference. Given these additions, however, alternative approaches in using the current (...) edition as a textbook may merit consideration. It may help to note that Chapters 16 and 17 (analysis of covariance, trend analysis) do not depend on the material in Chapters 12 through 15, although the student should know something about simple linear regression to be able to understand fully the material in Chapters 16 and 17. In any event, the instructor who wants to teach only the material in the first two editions can do so by dropping the added chapters - 12 through 15, and 18 - from the syllabus. (shrink)

Computational modeling plays an important role in prediction and optimization of real systems and processes. Models usually have some parameters which should be set up to the proper value. Therefore, parameter estimation is known as an important part of the modeling and system identification. It usually refers to the process of using sampled data to estimate the optimum values of parameters. The accuracy of model can be increased by adjusting its parameters to the optimum value which need a richer dataset. (...) One simple solution for having a richer dataset is increasing the amount of data, but that can be costly and time consuming. When using data from animals or people, it is especially important to have a proper plan. There are several available methods for parameter estimation in dynamical systems; however there are some basic differences in chaotic systems due to their sensitivity to initial condition. Accordingly, in this paper, a new cost function which is proper for chaotic systems is applied to the chaotic one-dimensional map. Then the efficiency of a newly introduced intelligent method experimental design in extracting proper data is investigated. The results show the success of the proposed method. (shrink)

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a (...) grounded theory; generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

I review three answers to the question: How can phenomenology contribute to the experimental cognitive neurosciences? The first approach, neurophenomenology, employs phenomenological method and training, and uses first-person reports not just as more data for analysis, but to generate descriptive categories that are intersubjectively and scientifically validated, and are then used to interpret results that correlate with objective measurements of behaviour and brain activity. A second approach, indirect phenomenology, is shown to be problematic in a number of ways. Indirect (...) phenomenology is generally put to work after the experiment, in critical or creative interpretations of the scientific evidence. Ultimately, however, proposals for the indirect use of phenomenology lead back to methodological questions about the direct use of phenomenology in experimental design. The third approach, 'front-loaded' phenomenology, suggests that the results of phenomenological investigations can be used in the design of empirical ones. Concepts or clarifications that have been worked out phenomenologically may operate as a partial framework for experimentation. (shrink)

We applaud the authors' basic message. We note that the negative research emphasis is not special solely to social psychology and judgment and decision-making. We argue that the proposed integration of null hypothesis significance testing (NHST) and Bayesian analysis is promising but will ultimately succeed only if more attention is paid to proper experimental design and implementation.

In many areas of the social and behavioral sciences, the nature of the experiments and theories that best capture the underlying constructs are themselves areas of active inquiry. Integrative experiment design risks being prematurely exploitative, hindering exploration of experimental paradigms and of diverse theoretical accounts for target phenomena.

Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. Yet their metascientific approach has one glaring omission (and misinterpretation of) – the role of sample diversity in generalisability. In this commentary, we challenge false notions of subsumed duality between contexts, population, and diversity, and propose modifications to their design space to accommodate sample diversity.

During the twentieth century statistical methods have transformed research in the experimental and social sciences. Qualitative evidence has largely been replaced by quantitative results and the tools of statistical inference have helped foster a new ideal of objectivity in scientific knowledge. The paper will investigate this transformation by considering the genesis of analysis of variance and experimental design, statistical methods nowadays taught in every elementary course of statistics for the experimental and social sciences. These methods were (...) developed by the mathematician and geneticist R. A. Fisher during the 1920s, while he was working at Rothamsted Experimental Station, where agricultural research was in turn reshaped by Fisher’s methods. Analysis of variance and experimental design required new practices and instruments in field and laboratory research, and imposed a redistribution of expertise among statisticians, experimental scientists and the farm staff. On the other hand the use of statistical methods in agricultural science called for a systematization of information management and made computing an activity integral to the experimental research done at Rothamsted, permanently integrating the statisticians’ tools and expertise into the station research programme. Fisher’s statistical methods did not remain confined within agricultural research and by the end of the 1950s they had come to stay in psychology, sociology, education, chemistry, medicine, engineering, economics, quality control, just to mention a few of the disciplines which adopted them. (shrink)

To what extent does the design of statistical experiments, in particular sequential trials, affect their interpretation? Should postexperimental decisions depend on the observed data alone, or should they account for the used stopping rule? Bayesians and frequentists are apparently deadlocked in their controversy over these questions. To resolve the deadlock, I suggest a three‐part strategy that combines conceptual, methodological, and decision‐theoretic arguments. This approach maintains the pre‐experimental relevance of experimental design and stopping rules but vindicates their (...) evidential, postexperimental irrelevance. †To contact the author, please write to: Tilburg Center for Logic and Philosophy of Science, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands; e‐mail: [email protected]. (shrink)

This paper considers ways that experimental design can affect judgments about informally presented context shifting experiments. Reasons are given to think that judgments about informal context shifting experiments are affected by an exclusive reliance on binary truth value judgments and by experimenter bias. Exclusive reliance on binary truth value judgments may produce experimental artifacts by obscuring important differences of degree between the phenomena being investigated. Experimenter bias is an effect generated when, for example, experimenters disclose (even unconsciously) (...) their own beliefs about the outcome of an experiment. Eliminating experimenter bias from context shifting experiments makes it far less obvious what the “intuitive” responses to those experiments are. After it is shown how those different kinds of bias can affect judgments about informal context shifting experiments, those experiments are revised to control for those forms of bias. The upshot of these investigations is that participants in the contextualist debate who employ informal experiments should pay just as much attention to the design of their experiments as those who employ more formal experimental techniques if they want to avoid obscuring the phenomena they aim to uncover. (shrink)

Almaatouq et al.'s prescription for more integrative experimental designs is welcome but does not address an equally important problem: Lack of adequate theories. We highlight two features theories ought to satisfy: “Well-specified” and “grounded.” We discuss the importance of these features, some positive exemplars, and the complementarity between the target article's prescriptions and improved theorizing.

Can you think of a meme that made you laugh or a political post that made you angry? These examples illustrate how social media use (SMU) impacts how people feel. Similarly, how people feel when they initiate SMU may impact the emotional effects of SMU. Someone feeling happy may feel more positively during SMU, whereas someone feeling sad may feel more negatively. Using an experimental design, we examined whether following SMU, those in a happy mood would experience increases (...) in positive affect (PA) and those in a sad mood would experience increases in negative affect (NA). A large sample of college students (N = 703) were randomly assigned to a happy, sad, or neutral mood induction before SMU. PA and NA were assessed at baseline, post-mood induction, and after SMU. Contrary to hypotheses, after SMU, people in happy moods experienced decreases in PA, and those in sad moods experienced decreases in NA, reflecting SMU having a dampening effect on emotions. PA and NA were significantly lower after SMU compared to baseline and did not vary by condition. How young adults feel when they log onto SMU matters in understanding how SMU impacts PA and NA, but on average, emotional experiences are dampened. (shrink)

Peter Spirtes, Clark Glymour, Richard Scheines, Christopher Meek, S. Fineberg, E. Slate. Prediction and Experimental Design with Graphical Causal Models.

This article reports an experimental study aimed at evaluating the change of cognitive processes in ethical decision making before and after business ethics training. An experimental design (Solomon Four-Group Design) was used to test the effectiveness of the training within a German university undergraduate business-oriented student sample. The cognitive processes in decision making (implicit and explicit moral attitudes, selective attention, moral awareness, moral judgment, moral intention, and moral behavior) were measured by using different direct instruments (e.g. (...) questionnaire items for moral judgments and explicit moral attitude scales) as well as indirect measures such as eye-tracking and the Implicit Association Test (IAT). The study yielded mixed results. On the one hand, significant changes in explicit attitudes, moral awareness, moral judgments, moral intention, and moral behavior in the pre-post-measurements of the training group have been revealed. On the other hand, there were no significant differences between the pre-post-measurements of the first training and control group as well as between the posttest-only-measures of the second training and control group. In addition, the implicit measures did not show any significant training effect. Implications with an emphasis on methodological aspects for future research on business ethics training are discussed. (shrink)

A “stopping rule” in a sequential experiment is a rule or procedure for deciding when that experiment should end. Accordingly, the “stopping rule principle” states that, in a sequential experiment, the evidential relationship between the final data and an hypothesis under consideration does not depend on the experiment’s stopping rule: the same data should yield the same evidence, regardless of which stopping rule was used. In this essay, I reconstruct and rebut five independent arguments for the SRP. Reminding oneself that (...) the stopping rule is a part of an experiment’s design and is no more mysterious than many other design aspects helps elucidate why some of these arguments for the SRP are unsound. (shrink)

Standard practices in experimental economics arise for different reasons. The “no deception” rule comes from a cost-benefit tradeoff; other practices have to do with the uses to which economists put experiments. Because experiments are part of scientific conversations that mostly go on within disciplines, differences in standard practices between disciplines are likely to persist.

As etiologic concepts of diseases gradually changed from humoral to microbial in the 19th century, syphilis presented particularly great challenges to doctors and scientists. Was “syphilis” merely a synonym for “venereal disease,” and did all manifestations attributed to it have the same cause? The discovery in 1879 of the gonococcus by Albert L. Neisser , and of the cause of chancroid, or soft chancre, in 1890 by Augusto Ducrey , established that venereal disease and syphilis were not synonymous, but syphilis (...) continued to mystify investigators. Symptoms could disappear for years and then—in most but not all persons—reappear with entirely different “tertiary” manifestations. .. (shrink)

Much of the early history of developmental and physiological genetics in Germany remains to be written. Together with Carl Correns and Richard Goldschmidt, Alfred Kühn occupies a special place in this history. Trained as a zoologist in Freiburg im Breisgau, he set out to integrate physiology, development and genetics in a particular experimental system based on the flour moth Ephestia kühniella Zeller. This paper is meant to reconstruct the crucial steps in the experimental pathway that led Kühn and (...) his collaborators at the University of Göttingen, and later at the Kaiser Wilhelm Institutes of Biology and Biochemistry in Berlin, to formulate, in their specific way, what later became known as the "one gene-one enzyme hypothesis." Special attention will be given to the interaction of the different parts of Kühn's Ephestia-based project, which were rooted in different research traditions. The paper retraces how, roughly between 1925 and 1945, these elements came to form a mixed experimental setup composed of genetic, embryological, physiological and, finally, biochemical constituents. Accordingly, emphasis is laid on the development of the terminology in which the results were cast, and how it reflected the hybrid state of an experimental system successively acquiring new epistemic layers. (shrink)

Stroke continues to be a leading cause of disability. Basic neurorehabilitation research is necessary to inform the neuropathophysiology of impaired motor control, and to develop targeted interventions with potential to remediate disability post-stroke. Despite knowledge gained from basic research studies, the effectiveness of research-based interventions for reducing motor impairment has been no greater than standard of practice interventions. In this perspective, we offer suggestions for overcoming translational barriers integral to experimental design, to augment traditional protocols, and re-route the (...) rehabilitation trajectory toward recovery and away from compensation. First, we suggest that researchers consider modifying task practice schedules to focus on key aspects of movement quality, while minimizing the appearance of compensatory behaviors. Second, we suggest that researchers supplement primary outcome measures with secondary measures that capture emerging maladaptive compensations at other segments or joints. Third, we offer suggestions about how to maximize participant engagement, self-direction, and motivation, by embedding the task into a meaningful context, a strategy more likely to enable goal-action coupling, associated with improved neuro-motor control and learning. Finally, we remind the reader that motor impairment post-stroke is a multidimensional problem that involves central and peripheral sensorimotor systems, likely influenced by chronicity of stroke. Thus, stroke chronicity should be given special consideration for both participant recruitment and subsequent data analyses. We hope that future research endeavors will consider these suggestions in the design of the next generation of intervention studies in neurorehabilitation, to improve translation of research advances to improved participation and quality of life for stroke survivors. (shrink)

Context: The integration of data measured in first- and third-person frameworks is a challenge that becomes more prominent as we attempt to refine the ties between the dimensions we assume to be objective and our experience itself. As a result, cognitive science has been a target for criticism from the epistemological and methodological point of view, which has resulted in the emergence of new approaches. Neurophenomenology has been proposed as a means to address these limitations. The methodological application of this (...) discipline, even in its mildest form, enriches the methodology typically used in cognitive sciences. Problem: Nowadays psychological studies are difficult to replicate. As a way to achieve replication of results published in a previous study in order to develop a methodological adaptation suitable for electroencephalographic measurements in a subsequent experiment, first-person accounts from the participants in our pilot study were included in the experiment construction. This study’s objective is to show the benefit of including a mild-neurophenomenology-inspired approach in the adaptation from an original paradigm, which requires, foremost, the ability to replicate the original results. Method: Interviews with open and semi-structured questions were carried out at the end of an Approach-Avoidance Task (AAT. The first-person reports, together with the behavioral outcomes of each pilot, were taken into account for the development of the next piloting phase until replication of the original results was achieved, and the final experimental design was elaborated. Results: A sequence of four pilots, where the integration of third- and first-person information derived from subjects’ behavior and reported experiences while carrying them out rendered the behavioral replication we sought to achieve, providing support for a first-person enriched cognitive science paradigm. Implications: Including first-person accounts systematically during the development and performance of classic cognitive paradigms ensures that those paradigms are measuring what they claim to measure. This is the next logical step to improve replication rates, to refine the explanation of the results and avoid confounding third-person data interpretation. Constructivist content: Including first-person experiences and acknowledging the active role that participants’ experiences regarding the paradigm had in the modeling of its final version is in concordance with a constructivist standing. (shrink)

The Hooks et al. review of microbiota-gut-brain literature provides a constructive criticism of the general approaches encompassing MGB research. This commentary extends their review by: highlighting capabilities of advanced systems-biology “-omics” techniques for microbiome research and recommending that combining these high-resolution techniques with intervention-based experimental design may be the path forward for future MGB research.

Asking and answering questions is the cornerstone of science yet formal training in understanding this key process is often overlooked. "Asking Questions in Biology" unpacks this crucial process of enquiry, from a biological perspective, at its various stages. It begins with an overview of scientific question-asking in general, before moving on to demonstrate how to derive hypotheses from unstructured observations. It then explains in the main sections of the book, how to use statistical tests as tools to analyse data and (...) answer those questions before, finally, showing the best practice in presenting scientific reports. As such, it is an indispensable companion to all students of biology, but particularly those enrolled in courses concerning experimental design; data analysis; hypothesis testing; research methods; or any practical project work. The late Chris Barnard was Professor of Animal Behaviour at Nottingham University. Francis Gilbert is Associate Professor of Ecology at Nottingham University. Peter McGregor isa Reader in Applied Zoologyat Cornwall College. ". (shrink)

PurposeThe purpose of this study was to introduce an orthogonal experimental design to improve the efficiency of building and optimizing models for freezing of gait prediction.MethodsA random forest model was developed to predict FOG by using acceleration signals and angular velocity signals to recognize possible precursor signs of FOG. An OED was introduced to optimize the feature extraction parameters.ResultsThe main effects and interaction among the feature extraction hyperparameters were analyzed. The false-positive rate, hit rate, and mean prediction time (...) were 27%, 68%, and 2.99 s, respectively.ConclusionThe OED was an effective method for analyzing the main effects and interactions among the feature extraction parameters. It was also beneficial for optimizing the feature extraction parameters of the FOG prediction model. (shrink)

Recently, there has been an increased interest in developing statistical methodologies for analyzing single case experimental design data to supplement visual analysis. Some of these are simulation-driven such as Bayesian methods because Bayesian methods can compensate for small sample sizes, which is a main challenge of SCEDs. Two simulation-driven approaches: Bayesian unknown change-point model and simulation modeling analysis were compared in the present study for three real datasets that exhibit “clear” immediacy, “unclear” immediacy, and delayed effects. Although SMA (...) estimates can be used to answer some aspects of functional relationship between the independent and the outcome variables, they cannot address immediacy or provide an effect size estimate that considers autocorrelation as required by the What Works Clearinghouse Standards. BUCP overcomes these drawbacks of SMA. In final analysis, it is recommended that both visual and statistical analyses be conducted for a thorough analysis of SCEDs. (shrink)

This argument map presents Paul Loewi’s crucial experiment in which he showed that neural transmissions of signals are chemical in nature, not electrical, in form of an argumentation. The map can be used in science education to show how the formulation of hypotheses should be related to a corresponding determination of experimental designs.

This study explores the influence of ethics education on accounting students’ level of ethical reasoning in Tunisia. Based on cognitive developmental theory, we tested the effectiveness of an ethics intervention before and after ethics education with a control group. A triangulated research design was incorporated. Experimental and qualitative methods were used to control for experimental bias. This study revealed that the progress of moral development was not significant between the pre-test and the post-test. Data analysis revealed the (...) primary challenges in teaching ethics in the Tunisian audit context. Through a confirmatory qualitative survey, stating “the consequences of unethical behaviour in the long term” demonstrates that the intervention meets the primary interest of this study, and it may influence auditors’ decisions and their perceptions of the potential consequences of their decisions. (shrink)

Since modern medicine is based substantially in clinical medical research, the flaws and ethical problems that arise in this research as it is conceived and practiced in the United States are likely to be reflected to some extent in current medicine and its practice. This paper explores some of the ways in which clinical research has suffered from an androcentric focus in its choice and definition of problems studied, approaches and methods used in design and interpretation of experiments, and (...) theories and conclusions drawn from the research. Some examples of re-visioned research hint at solutions to the ethical dilemmas created by this biased focus; an increased number of feminists involved in clinical research may provide avenues for additional changes that would lead to improved health care for all. (shrink)

This paper is motivated by the increasing popularity of efficient designs for stated choice experiments. The objective in efficient designs is to create a stated choice experiment that minimizes the standard errors of the estimated parameters. In order to do so, such designs require specifying prior values for the parameters to be estimated. While there is significant literature demonstrating the efficiency improvements of employing efficient designs, the bulk of the literature tests conditions where the priors used to generate the efficient (...) design are assumed to be accurate. However, there is substantially less literature that compares how different design types perform under varying degree of error of the prior. The literature that does exist assumes small fractions are used, which is in contrast to computer-aided surveys that readily allow for large fractions. Further, the results in the literature are abstract in that there is no reference point to provide clear insight on the magnitude of any issue. Our objective is to analyze the robustness of different designs within a typical stated choice experiment context of a trade-off between price and quality. We use as an example transportation mode choice, where the key parameter to estimate is the value of time. Within this context, we test many designs to examine how robust efficient designs are against a misspecification of the prior parameters. The simple mode choice setting allows for insightful visualizations of the designs themselves and also an interpretable reference point for the range in which each design is robust. Not surprisingly, the D-efficient design is most efficient in the region where the true population VOT is near the prior used to generate the design: the prior is $20/h and the efficient range is $10–$30/h. However, the D-efficient design quickly becomes the most inefficient outside of this range, and the estimation significantly degrades above $50/h. The orthogonal and random designs are robust for a much larger range of VOT. The robustness of Bayesian efficient designs varies depending on the variance that the prior assumes. Implementing two-stage designs that first use a small sample to estimate priors are also not robust relative to uninformative designs. Arguably, the random design performs as well as any design, and it will perform even better if data cleaning is done to remove choice tasks where one alternative dominates the other. (shrink)

Research on expert‐novice differences has mainly focused on how experts solve familiar problems. We know far less about the skills and knowledge used by experts when they are confronted with novel problems within their area of expertise. This article discusses a study in which verbal protocols were taken from subjects of various expertise designing an experiment in an area with which they were unfamiliar. The results showed that even when domain knowledge is lacking, experts solve a novel problem within their (...) area of expertise by dividing the problem into a number of subproblems that are solved in a specified order. The lack of domain knowledge is compensated for by using abstract knowledge structures and domain‐specific heuristic strategies. However, the quality of their solutions is considerably lower than the quality attained by experts who were familiar with the type of problem to be solved. The results suggest that when experts are confronted with novel problems as compared with familiar problems, their form of reasoning remains intact, but the content of their reasoning suffers due to lack of domain knowledge. (shrink)