Europe PMC

This website requires cookies, and the limited processing of your personal data in order to function. By using the site you are agreeing to this as outlined in our privacy notice and cookie policy.

Abstract 


While research has examined factors influencing understanding of informed consent in biomedical and forensic research, less is known about participants' attention to details in consent documents in psychological survey research. The present study used a randomized experimental design and found the majority of participants were unable to recall information from the consent form in both in-person and online formats. Participants were also relatively poor at recognizing important aspects of the consent form including risks to participants and confidentiality procedures. Memory effects and individual difference characteristics also appeared to influence recall and recognition of consent form information.

Free full text 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Ethics Behav. Author manuscript; available in PMC 2013 Feb 27.
Published in final edited form as:
Ethics Behav. 2011; 21(4): 332–350.
Published online 2011 Jul 14. https://doi.org/10.1080/10508422.2011.585601
PMCID: PMC3583369
NIHMSID: NIHMS443188
PMID: 23459667

Do undergraduate student research participants read psychological research consent forms? Examining memory effects, condition effects, and individual differences

Abstract

While research has examined factors influencing understanding of informed consent in biomedical and forensic research, less is known about participants’ attention to details in consent documents in psychological survey research. The present study used a randomized experimental design and found the majority of participants were unable to recall information from the consent form in both in-person and online formats. Participants were also relatively poor at recognizing important aspects of the consent form including risks to participants and confidentiality procedures. Memory effects and individual difference characteristics also appeared to influence recall and recognition of consent form information.

Keywords: consent form, self-determination, recall, recognition, research informed consent

For psychological research, the American Psychological Association (2010) states participants in research studies must be informed of the purpose and procedures of the study, the right to decline to participate, the right to withdraw from the study and the potential penalties for doing so, the risks and benefits of participating in the study, limits to confidentiality, incentives for participation, and contact information for questions and concerns about participation. These aspects of research studies are generally explained to participants within informed consent documents that participants read and sign; agreeing that they understand all the aforementioned components of the research study. In order to give informed consent, an individual must possess the capacity the make a decision (Appelbaum & Roth 1982; Christensen, Haroun, Schneiderman, & Jeste, 1995), which is evidenced by the ability to understand the information presented, the ability to appreciate the likely consequences of participation, the ability to weigh options using reasoning capacities, and the ability to communicate a final decision regarding whether or not to participate (Appelbaum & Grisso, 1988; Grisso & Appelbaum 1998; Pincus, Lieberman, & Ferris, 1999).

Functionally, documentation of informed consent is typically defined by a signature indicating that one has read and agreed to information presented in a consent document. However, in reality, documentation of informed consent may not actually indicate informed consent if a participant did not thoroughly read the consent document or if they did not fully understand what they read. A lack of understanding of consent information may be exhibited in a number of ways. Some individuals may not realize that they are participating in a research study to begin with, or they may misunderstand the nature of randomization process and the placebo treatments. Others may not fully understand the risks involved with their participation, or that they have the ability to withdraw at any time. For example, in medical settings, some individuals may misinterpret the role of the researchers and/or care providers, and may even fall prey to the “therapeutic misconception,” which is an assumption made by the patient that research and/or treatment decisions are being made solely with the individual’s best interest in mind (Appelbaum, Roth, & Lidz, 1982; Cassileth, Zupkis, Sutton-Smith, & March, 1980; Silva & Sorrell, 1988). In addition, a number of studies have shown that participants in clinical trials lack a full understanding of the risks and benefits of their participation even after efforts have been made to inform them (Daugherty, 1999; Waggoner & Mayo, 1995). In a review of more than 20 studies, Stanley and Guido (1996) found that comprehension, which was measured by recall questions, ranged from 35% to 80% for medical research studies.

Factors associated with informed consent comprehension

While a number of factors may contribute to understanding of consent form material, such as reading level, age, education, memory, gender, psychiatric diagnoses, and mental state (Arscott, Dagnan, & Kroese, 1999; Bruzzese & Fisher, 2003; Grisso & Appelbaum 1998; Hopper et al., 1994; Moser et al., 2002; Sugarman, McCrory, & Hubal, 1998), participants simply may lack understanding because they failed to read the documentation closely. In one study, fewer than half of the participants read all of the information given to them in the consent document, and only half recalled the number of drugs they would be taking, the length of their treatment, or even its purpose (Olver et.al, 2009). A number of studies have also addressed recall of information presented in informed consent documents and related legal documents in legal settings. Participants either did not read or simply skimmed these materials and were subsequently not able to recall much, if any, of the information presented (e.g., Estey, Wilkin, & Dossetor, 1994; Stanley & Guido, 1996; Wogalter, Howe, Sifuentes, & Luginbuhl, 1999).

As the process of reading and understanding consent forms may be most critical for biomedical research or in forensic settings, research exists in these areas looking at participants’ understanding of informed consent. Despite the lower risks involved in non-medical studies, informed consent is a necessary and important first step to psychological survey research as well. Thus, understanding if and how individuals pay attention to important information presented within informed consent forms warrants more research attention among non-medical studies. One small study found that 5% of participants self-reported not reading consent documents while 30% reported only skimming the material (Varnhagen et al., 2005). Additional research with experimental designs beyond self-report are needed to continue exploring participants’ attention to details within consent forms. Further, few studies have considered consent reading from an existing theoretical motivational framework. One theoretical framework that may be particularly relevant is self-determination theory.

Self-determination Theory

Autonomy orientation

Self-determination Theory (Deci & Ryan, 1985; 2000) posits that individuals vary in the extent to which they engage in behaviors based on autonomy and controlled orientations. These aspects of personality are understood to be relatively enduring within individuals, but are also suggested to be influenced by environmental and situational factors (Deci, Koestner, & Ryan, 1999; Deci, Schwartz, Sheinman, & Ryan, 1981). Individuals who are more autonomously oriented tend to regulate their behaviors based on intrinsic interests and integrated values (Ryan & Deci, 2002). Autonomy orientation is associated with self-actualization, openness, lack of defensiveness, and greater consistency between one’s values and behaviors (Bober & Grollnick, 1995; Hodgins, Koestner, & Duncan, 1996; Koestner, Bernieri, & Zuckerman, 1992). In general, autonomous individuals are presumed to be more intrinsically motivated and more likely to engage in behaviors out of genuine interest and choice. We would expect that individuals who are higher in autonomy would be more intrinsically interested in participating in research and more motivated to be informed in the consent process (i.e., more likely to read the consent forms).

Controlled orientation

In contrast, controlled orientation represents a tendency to regulate behaviors based on contingencies (i.e, rewards or extra credit) and expectations (Ryan & Deci, 2002). Controlled orientation has been associated with type-A coronary prone behavior, hostility, defensiveness, and emotional reactivity (Deci & Ryan, 1985, Goldstein & Iso-Ahola, 2008; Hodgins & Knee, 2002; Knee & Neighbors, 2002; Neighbors, Vietor, & Knee, 2002; Weinstein & Hodgins, 2009). In general, controlled individuals are more likely to engage in behaviors in order to receive some external reinforcement (i.e. payment or credit), or because they are required. We would therefore expect individuals who are more controlled to be less interested in participation for its own sake and thus less likely to read consent forms.

Alternative to Self-determination Theory

There are at least two constructs that would serve as plausible rival alternatives to self-determination as motivational explanations for failure to read consent forms: conscientiousness and social desirability. Conscientiousness can be defined as a tendency to exhibit much effort and care in following the dictates of one’s conscience. Functionally, it represents a tendency to care about the quality of one’s work (McCrea & Costa, 2003) and therefore those higher in conscientiousness may pay particular attention to the consent form prior to participation. Social desirability is a tendency to want to present the self in a positive light (Crowne & Marlowe, 1960). For example, those participants higher in social desirability may intentionally pay particular attention to the details within the consent form at the forefront in order to help researchers form a favorable impression of them (i.e., perhaps as a “good participant”). In addition, a person high in social desirability might want to appear to have read a consent form even if he or she had not. For example, participants who desire to express themselves as having desirable characteristics may respond to questions such as “On how many questions did you guess?” with a response such as “zero,” when in actuality the participant guessed on quite a few questions. Both conscientious and social desirability would seem to be important factors to control for in considering the association between self-determination and informed consent.

Online consent

Benefits of online research are numerous and include ease and expanded time of access, reduced cost and paperwork burden, ability to recruit a large and representative sample through email and survey link invitation, and perhaps more accurate data due to greater anonymity for stigmatizing attitudes and behaviors (Gosling, Vazire, Srivastava & John, 2004; Simon Rosser et al., 2009; Stretcher, 2007). As more research is conducted over the Internet, the importance of examining attention to details in online statements outside of the laboratory is apparent (Kraut et al., 2004). With online research comes the need for online consent documents. A systematic review of research designed to enhance informed consent concluded that the use of multimedia enhanced consent process, including web-based consent only marginally improved participant’s comprehension (Flory & Emanuel, 2004). Varnhagen and colleagues (2005) examined the paper versus online consent forms and found little differences between recalled information from both documents. In fact, participants in both conditions recalled only about 10% of the information presented. While this study represents an important first step, Internet-based research is more cost effective and practical if participants do not need to attend a scheduled lab meeting to read a consent form. Thus, we also designed the present study to examine how individual difference factors relate to retention of information within consents forms delivered in-person or via the web.

The present study

Undergraduate students are an important population to explore questions about informed consent in psychological research. For one, the majority of social psychology research utilizes undergraduate samples and generalizes these findings to the population as a whole. While these methods have recently been critiqued (Arnett, 2008; Henrich, Heine, & Norenzayan, 2010), the convenience, cost-effectiveness, and practicality of using undergraduate research participants (mostly North American students from psychology courses) has long been an acceptable factor in psychological research. In addition, recent census data indicates that more than 50% of the current U.S. population aged 18 to 24 has attended at least some college (U.S. Census Bureau, 2009). If students experience psychological research and accompanying consent forms for the first time in college, patterns of behavior initiated during this first contact (e.g., reading the information carefully) may translate into future behavior when taking part in research studies in the future. Thus, utilizing a sample of college student participants, the present study was designed to extend previous research in evaluating the extent to which participants fully read informed consent documents and to explore factors associated with recognition and recall of informed consent material. We were specifically interested in method of presentation (i.e., web vs. paper), memory effects, and individual differences in self-determination as factors associated with informed consent comprehension. First, using a between-subjects randomized design, we compared in-person paper-and-pencil consent documents with online consent documents to examine differences in recall and recognition effects between individuals from the two conditions. Contrary to prior studies examining recall of consent form material, our study involved no deception or surprise test following the consent forms; that is participants were told in consent forms that they would be asked questions about the material within the consent documents. In addition, this study used two forms of questions to examine both recall and recognition effects (i.e., one question where participants were asked an open-ended question to recall an embedded word from the consent form; eight questions where participants were asked multiple choice questions to recognize content information from the consent form).

Secondly, the individual differences (i.e., personality characteristics) that impact attention to detail within consent forms have not been assessed for both written and online mediums. In addition, most of the research on attention to consent forms has been at the more qualitative and exploratory levels, in that the studies did not control for a number of important factors (e.g., time effects on memory) or used a subjective coding method to assess degree of recall of information. Thus, we examined the within-subjects effects of personality characteristics on recall and recognition of consent form material after controlling for condition (in-person versus online) and potential memory effects. We assessed retention of information using a standardized open-ended and multiple choice quiz covering aspects of the consent as specified by the APA (2010). Specific individual difference factors considered included social desirability, conscientiousness, controlled orientation, and autonomy orientation. We hypothesized that those with high levels of social desirability, conscientiousness, and autonomy orientation would score higher on the quiz inquiring about consent form material, while those with higher controlled orientation would perform worse on the quiz.

Method

Participants

Participants were college students recruited through the Psychology subject pool at a large research-focused university in the northwestern United States. During one academic quarter, introductory and higher level Psychology students had an opportunity to obtain extra course credit if they chose to participate in this study or approximately 15 other research studies. Other options for extra credit were also available. Two-hundred and sixty participants who chose to participate in this study ranged in age from 18 to 39, with a mean age of 19.63 (SD = 2.63). Forty-five percent of participants identified as Asian/Asian-American/Hawaiian/Pacific Islander, with 32% identifying as Caucasian/White, 4% as “Hispanic/Latino(a), 2% as African-American/Black, 1% as Native Alaskan/American Indian, 11% as “mixed, and 5% as “other.”

Materials and Procedure

All procedures were approved by the Human Subjects Review Board at the university. During nine weeks of the spring quarter, participants were able to sign up via the university’s online system for one of thirty-five 30-minute timeslots. Approximately four to 10 students attended each scheduled session. Participants who chose to participate in this study were randomized via a coin flip to one of two conditions upon arrival at their scheduled session: (1) an in-person condition and (2) an online condition.

In-person condition

Those participants in the in-person condition indicated their name on a sign-up sheet and were given a consent form with the instructions “Please read and, if you choose to participate, sign the consent form and then hand it back to me” by an undergraduate research assistant. Participants were given the consent form as they entered the session and sat at one long table with other in-person condition participants as they reviewed the consent form. Upon signing the form, the participants handed the signed consent form to the research assistant and were given a survey packet containing a nine-item quiz about the consent form and three individual differences measures. As participants finished the survey, they handed in the survey packet and were offered a copy of the consent form for their records. No identifying information was written on the survey packets and signed consent forms and the sign-up sheets were kept separate from the packets at all times. Credit was assigned via the university’s online system by the principal investigator based on the names from the sign-up sheet.

Online condition

Upon arrival at their scheduled meeting, participants randomly assigned to the online condition were told they had been randomized to the online condition and were given a sheet of paper with instructions for how to access the online survey from an Internet-accessible computer of their choice. These instructions informed participants that they had one week to fill out and submit the online questionnaire. Participants were assigned a random Personalized Identification Number (PIN) to use for logging into the survey, which is a necessary component of the survey design program used in this study. These PINs were not linked to any identifiable information. Once they logged into the survey, participants were presented with the consent form on the computer screen, followed by the survey packet containing the quiz and individual differences measures. After completing the survey items, participants were instructed to send an email to the principal investigator at a researcher-created email name to indicate they had finished the study. Participants were able to request a copy of the consent form from this email address as well. This method was used to keep contact information separate from survey responses (i.e., anonymous).

Consent form

The consent form was presented as one double-sided sheet during the in-person condition or as one page of scrollable text within the online condition. Participants in the online condition needed to scroll down to the end of the text to electronically indicate “I accept. I want to participate in this study” before entering the online survey. The consent form was divided into nine sections. Eight headings included (1) “Researchers” (five researchers’ names, titles, and contact information; 74 words), (2) “Research Statement” (standard and required text specified by the Human Subjects Review Board (HSRB) at the university to introduce research in general; 120 words, (3) “Purpose of the Study” (47 words), (4) “Study Procedures” (247 words), (5) “Risks, Stress, or Discomfort” (55 words), (6) “Alternatives to Taking Part in this Study” (47 words), (7) “Benefits of the Study” (30 words), and (8) “Other Information” (specifying anonymity procedure, incentive for participation, voluntary nature of the study, right to withdraw, and freedom to skip questions without penalty; 250 words). The consent form ended with the researchers’ signature and a (9) participant’s statement indicating understanding of the consent form and study procedures, as well as information on how to contact the researchers or HSRB at the university with questions or concerns (121 words). To avoid deception, information to be asked about later was presented clearly with the first page containing the statement “[t]he purpose of this study is to examine personality characteristics that associate with individuals’ attention to details on forms such as this one” under the “Purpose of the Study” section. In the section “Study Procedures” on the first side of the in-person consent form and within the visible text presented to participants prior to scrolling through the online consent form was the statement, “In the questionnaire, you will be asked to recall information from this form. For example, you will be asked to recall the phrase lucky charms when completing the questionnaire.”

Recall and recognition quiz

After reading the consent form, all participants were presented with a quiz asking them to recall (open-ended response) or recognize (multiple choice responses) information from the consent form.1 At least one recognition quiz question was drawn from each of the nine consent form sections, with the exception of the “Research Statement” section which contained general information about research and was not specific to this study. All consent forms used at the university contain this statement; thus, information from this statement may have been recognized from other studies that participants had participated in. At the end of the quiz, participants were asked to rate on a seven-point Likert scale how confident they felt their answers to the nine quiz items were correct (“1 = not at all confident” to “7 = very confident’). Participants also indicated how many of the nine items (one recall, eight recognition) they guessed the answer to (response option zero to nine). In addition to the two randomized assessment conditions (in-person and online), participants were further randomized to assess for “memory effects,” such that approximately half of participants were presented with the quiz at the beginning of the survey packet, with the remaining participants receiving the quiz following the measures of individual differences.

Measures of individual differences

Autonomy and controlled orientations were assessed with the General Causality Orientations Scale (GCOS; Deci and Ryan, 1985; Hodgins, Koestner, & Duncan, 1996). The GCOS includes 17 scenarios, each of which is followed by an autonomous and a controlled response. For example, one of the scenarios is: “You are embarking on a new career. The most important consideration is likely to be:” The autonomous orientation is assessed by the likelihood of responding by “How interested you are in that kind of work” whereas the controlled orientation is assessed by the likelihood of responding by “Whether there are good possibilities for advancement.” Participants rate how likely they would make each response on a seven-point Likert-type scale from “very unlikely” to “very likely”. Reliability estimates of both subscales were adequate (α = 0.87 for autonomy, α = 0.77 for controlled).

Conscientiousness was assessed using two items from the Ten Item Personality Inventory (TIPI; Gosling Rentfrow, & Swann, 2003). The two items for the conscientiousness factor were specified as “Disorganized, careless” (reverse coded) and “dependable, self-disciplined” and participants rated from “1 = disagree strongly” to “7 = agree strongly” how much they agreed the personality characteristics applied to them. Conscientiousness was scored as the mean of the two items, which correlated with each other (r = 0.31, p < .05). This subscale of the TIPI has displayed adequate convergent and discriminate validity with the conscientious construct assessed with longer measures of the Big Five personality domains (Furnham, 2008; Gosling et al., 2003).

Social desirability bias was measured with the Social Desirability Scale-17 (SDS-17;Stober, 1999). As suggested by Stober (2001), item #4, “I have tried illegal drugs (for example, marijuana, cocaine, etc.),” was removed to yield 16-items. The reliability estimate for this collection of items was slightly less than preferred (α = 0.65).

Results

Analytic plan

Descriptive analyses included frequencies, correlations, and means of targeted dependent and independent variables. Main effects of memory, condition, and individual differences effects on recall information were examined through logistic regression analyses predicting correct (1) or incorrect (0) recall of the word “lucky charms” from the consent form. Main effects of these factors on recognition of information in the consent form were examined through linear regression analyses predicting the summed score of the eight recognition quiz items. Two other linear regression models predicting confidence in responses and number of quiz answers guesses were also included. For analyses, a dichotomous “condition” variable was created to determine assessment condition effects (“0 = in-person,” “1 = online”) and a dichotomous “version” variable was created and specified as “0 = quiz last” (i.e., after the individual differences questionnaire, and “1 = quiz first” (i.e., immediately after the consent form. Randomization appeared to be successful, with 51% specified as “in-person” and 51% with the “quiz first.” A non-significant chi-square test revealed version did not differ with within or between conditions. A correlation matrix, as well as means and standard deviations, of all variables examined in regression analyses can be found in Table 1.

Table 1

Correlation matrix of variables with means and standard deviations

12345678910
1 Recall word correct a--
2 Sum of 8 recognition items0.47**--
3 Confidence in answers0.48**0.45**--
4 Number of items guessed−0.45**−0.39**−0.66**--
5 Version b0.100.16*0.11−0.04--
6 Condition c−0.13*−0.04−0.04−0.020.03--
7 Autonomy orientation0.18**0.18**0.15*−0.14*0.010.02--
8 Controlled orientation−0.17**−0.16*0.010.10−0.020.000.27**--
9 Conscientiousness0.12*0.080.08−0.09−0.03−0.090.23**0.07--
10 Social Desirability0.060.010.14*−0.16*0.13*−0.060.04−0.070.11--

Mean (Standard deviation)0.26 (0.24)4.58 (1.72)3.45 (1.60)4.79 (2.32)0.51 (0.50)0.48 (0.50)5.47 (0.77)4.18 (0.71)10.68 (2.45)8.61 (2.97)

Note:

*p < .05,
**p < .01;
a“0 = no, 1 = yes;”
b“0 = quiz last, 1 = quiz first;”
c“0 = in person, 1 = online”

Descriptive analyses

The percentages of participants who reported correct answers for the recall word question and the eight recognition quiz questions are represented in Figure 1. Only 26% of participants accurately recalled the phrase “lucky charms” from the consent form. Participants received a mean score of 4.58 (SD = 1.72) on the recognition quiz, indicating that participants missed slightly more than three questions on average (range = 0 to 8). One participant received a score of zero on the quiz and 4% answered all questions correctly. As indicated in Figure 1, over 60% correctly answered multiple choice questions about the study’s purpose, ability to skip questions without penalty, what to do if they had questions/concerns, and the risks. Less than 60% were able to recognize information about researchers’ names, number of participants recruited, confidentiality, and benefits of the study. Participants reported guessing on nearly five of the nine quiz questions (M = 4.79, SD = 2.32) and reported moderate confidence in their answers to the items (M = 3.45, SD = 1.60).

An external file that holds a picture, illustration, etc.
Object name is nihms443188f1.jpg
Percentages of participants with correct answers to the recall (1 item) and recognition (8 items) quiz questions

Note: Recall Q1 = recall word “lucky charms,” Recognition Q1 = researchers names, Recognition Q2 = number of participants, Recognition Q3 = confidentiality, Recognition Q4 = purpose, Recognition Q5 = able to skip items, Recognition Q6 = questions/concerns, Recognition Q7 = benefits, Recognition Q8 = risks

Predicting recall and recognition of quiz items

Recall word question

Logistic regression analyses evaluated the likelihood of recalling the recall word quiz question (“lucky charms”). Randomization (version, condition) and the four individual differences factors (conscientiousness, social desirability, autonomy orientation, controlled orientation) were included in the model on Step 1, with a version x condition interaction term entered on Step 2 to determine if recall effects were more pronounced among certain versions within condition after controlling for all other variables. Continuous predictors were centered prior to this and all of the following regression analyses. Results of this logistic regression analysis can be found in Table 2. Controlling for other factors in the model on Step 1, a significant condition effect was found such that those in the online condition were approximately half as likely to accurately recall the word “lucky charms” from the consent form than those in the in-person condition (odds ratio = 0.49). Follow-up descriptive analyses revealed that participants in the online condition were significantly worse at recalling the word from the consent form compared with those in the in-person condition (80% versus 69% incorrect responses), χ2 (1, N = 260) = 4.49, p < .05. There were no differences in recall of the quiz word between quiz first and quiz last versions of the survey. While conscientiousness and social desirability displayed no unique effects on recall of the consent form word, autonomy orientation and controlled orientation significantly predicted recall. Those higher in autonomy orientation were approximately two times more likely to recall the word from the consent form (odds ratio = 2.12), while those with higher controlled orientation scores were approximately half as likely to recall the word (odds ratio = 0.46).

Table 2

Logistic regression predicting recall word quiz question from randomization and individual differences factors

Dependent variable:WaldExp (B)
Recall quizBSEChi-Square (df =1)Odds Ratiop
Step 1
Versiona0.470.312.371.600.12
Conditionb−0.710.315.150.490.02
Conscientiousness0.100.072.001.100.16
Social desirability0.000.050.001.000.99
Autonomy orientation0.750.2310.382.120.00
Controlled orientation−0.780.2311.340.460.00
Step 2
Version x condition1.510.675.144.540.02
a“0 = quiz last, 1 = quiz first;”
b“0 = in person, 1 = online”

On Step 2, there was a significant version x condition effect. Figure 2 presents the graph of this interaction. It was revealed that those in the online condition who received the quiz last (i.e., after the individual differences questionnaire) were significantly worse at recalling the phrase from the consent form than those in other conditions. Based on these findings and to provide more descriptive analyses, we created a contrast comparing participants who received the quiz last in the online condition versus all other conditions (i.e., online quiz first, in-person quiz first, and in-person quiz last). Only 10% of those who received the quiz last in the online condition were able to correctly recall the phrase “lucky charms” in comparison to 35% of participants in all other conditions, χ2 (1, N = 260) = 9.71, p < .001.

An external file that holds a picture, illustration, etc.
Object name is nihms443188f2.jpg

Interaction graph of version x condition predicting recall item correct response

Recognition quiz composite score

On Step 1 of a linear regression analysis, we predicted composite scores obtained on the eight-item recognition quiz from the randomization factors (version, condition) and from the four individual differences characteristics of conscientiousness, social desirability, autonomy orientation, and controlled orientation. On Step 2, we entered the version x condition interaction term. Table 3 contains the regression coefficients, t-statistics, and p-values from this analysis. After controlling for all other variables in the model on Step 1, a significant version effect was found. Those participants who received the quiz before filling out the measures of individual differences performed significantly better on the quiz. Follow-up reports of means and independent samples t-tests revealed that those who received the quiz first (M = 4.84, SD = 1.75) scored better on the eight-item recognition quiz than those who received the quiz after the individual differences measures (M = 4.30, SD = 1.65), t (258) = 2.53, p < .05. In addition, after controlling for other factors in the model, both autonomy orientation and controlled orientation predicted composite quiz scores. Similar to the recall question effects, those higher in autonomy orientation scored better on the quiz. In contrast, those higher in controlled orientation scored worse on the quiz. Neither conscientiousness nor social desirability displayed unique effects on recognition quiz scores. There were no observable differences in recognition effects by condition. To stay consistent with analyses reported above, we again compared participants in the online condition who received the quiz last with all other conditions. Online participants who received the quiz last reported lower scores on the eight-item recognition quiz (M = 4.19, SD = 1.86) than those in all other conditions (M = 5.02, SD = 1.95), t (258) = 2.93, p < .01. On Step 2, the version x condition effect was non-significant.

Table 3

Linear regression predicting recognition eight-item quiz composite score from randomization and individual differences factors

Dependent variable:
Recognition quizBSEβtp
Step 1
Versiona0.540.210.162.640.01
Conditionb−0.170.21−0.05−0.850.40
Conscientiousness.035.043.049.8020.42
Social desirability−.027.035−.047−.7730.44
Autonomy orientation.507.141.2273.594.000
Controlled orientation−.532.150−.220−3.553.000
Step 2
Version x condition0.510.410.131.250.21
a“0 = quiz last, 1 = quiz first;”
b“0 = in person, 1 = online”

Predicting confidence and guessing of responses

Confidence in responses

Table 4 contains results from regression analysis predicting confidence in all nine (one recall, eight recognition) quiz responses from the two randomization and four individual differences factors on Step 1. Autonomy orientation was found to be the only significant predictor of confidence in responses and was associated with great confidence. A marginally significant (p = 0.06) effect was observed for social desirability, such that those higher in social desirability were more likely to report more confidence in their responses. On Step 2, there was a significant version x condition interaction effect. A graphical representation of this interaction suggested that those in the online condition who received the quiz last reported the least amount of confidence in their responses to the quiz items (see Figure 3).

An external file that holds a picture, illustration, etc.
Object name is nihms443188f3.jpg

Interaction graph of version x condition predicting confidence in responses to quiz items

Table 4

Linear regression predicting confidence in quiz responses from randomization and individual differences factors

Dependent variable:
Confidence in responsesBSEβtp
Step 1
Versiona0.290.200.091.440.15
Conditionb−0.120.20−0.04−0.610.54
Conscientiousness0.020.040.040.560.58
Social desirability0.060.030.121.860.06
Autonomy orientation0.290.140.142.130.03
Controlled orientation−0.050.15−0.02−0.360.72
Step 2
Version x condition0.870.400.242.210.03
a“0 = quiz last, 1 = quiz first;”
b“0 = in person, 1 = online”

Number of quiz items guessed

The number of the nine quiz items the participant reported guessing at was predicted by the two randomization and four individual differences factors on Step 1. Table 5 contains results from regression analysis. Individuals higher in controlled orientation reported guessing at more answers to the quiz. Participants who were higher in autonomy reported guessing on significantly fewer items. Those higher in social desirability reported fewer number of quiz answers guessed at. There was no significant version x condition effect on Step 2.

Table 5

Linear regression predicting number of quiz answers guessed from randomization and individual differences factors

Dependent variable:
Number of items guessedBSEβtp
Step 1
Versiona−0.070.29−0.02−0.260.80
Conditionb−0.120.29−0.03−0.430.67
Conscientiousness−0.050.06−0.05−0.860.39
Social desirability−0.110.05−0.14−2.160.03
Autonomy orientation−0.460.20−0.15−2.350.02
Controlled0.430.210.132.090.04
Step 2
Version x condition−0.330.57−0.06−0.570.57
a“0 = quiz last, 1 = quiz first;”
b“0 = in person, 1 = online”

Discussion

The present experimental study used a between-subjects design to target two research questions: (1) do undergraduate participants read psychological research consent forms and accurately recall and recognize information from these forms and (2) what factors relate to recall and recognition of information from consent forms. In general, approximately ¼ of all participants recalled the embedded phrase “lucky charms” from the consent form. Those who received online consent forms to read were half as likely to recall the phrase compared to those who received paper versions of consent forms in-person from a research assistant. Participants in the online condition who received the consent form quiz after the measures of individual differences were especially poor at recalling the phrase from the consent form. In addition, participants were relatively poor at recognizing information from the consent forms in both in-person and online conditions. Participants reported guessing on more than half of the recall and recognition quiz questions and reported moderate confidence in their responses. Interestingly, despite over half (62%) of participants recognizing the purpose of the study to be “To examine personality characteristics that associate with individuals’ attention to details within forms such as consent forms,” the majority of participants apparently did not pay enough attention to the consent form to recall the word “lucky charms.” Given that participants were told in the consent form that they would be asked to recall the specific phrase they were later asked about (i.e., there was a no deception in the study), it is possible that the majority of participants in this research study neglected to read the consent form.

Memory effects may help explain some of the findings from the study. Participants who received the quiz after the individual differences questionnaire scored approximately one-half of a point less than those who received the quiz immediately after the consent form. This may suggest that recognition of important information may wane over the course of the study (possibly more for longer studies). It may have been that participants read the consent form but after completing the 10–15 minute questionnaire were no longer able to recall the specific phrase. However, it is unclear why this memory effect was only evident in the online condition. Perhaps the online questionnaire took longer to complete than the in-person assessment (e.g., participants could have potentially taken a break during the survey and returned to find the quiz several minutes or hours later).

There were a number of significant findings related to Self-determination. Autonomy orientation was associated with better recall, higher confidence in responses, and less self-reported guessing. In contrast, controlled orientation was associated with less recall, less recognition, and more self-reported guessing. These findings are consistent with previous research and theory regarding individual differences in self-determination (Deci & Ryan, 1985; Ryan & Deci, 2002). Moreover, autonomous individuals tend to be more intrinsically and genuinely interested in activities in which they engage. Thus, they appear to be more interested in the research process and more engaged in the consent process. Controlled students’ participation may be more motivated by contingencies (i.e., credit), resulting in less genuine interest in the process, and less engagement in the consent process. These effects were present even after controlling for other individual differences factors that could have affected participants’ engagement in the consent form (i.e., social desirability, conscientiousness). Consistent with what we might expect, those higher in social desirability reported greater confidence in their answers and reported guessing at fewer questions, yet no main effects were found for social desirability and recall or recognition.

In the present study, over one-third of participants were not able to recognize the information from the consent forms regarding the risks of the study. Approximately 15% did not know they were able to skip items without penalty. Moreover, more than 50% of participants failed to recognize that their research data would be kept anonymous. Although psychological survey research is generally benign, risks may still be present. For example, while the literature generally suggests that assessments of traumatic events and symptoms generally cause minimal long-term distress to participants, some participants do report distress after these assessments (Newman & Kaloupek, 2009). While this research is important, it is essential for participants to understand the potential risks they may encounter from the research. Indeed, loss of confidentiality is an important risk in many psychological research studies (e.g., a file with PIN codes connected to participant identifiers is left open and a research participant recognizes a friend’s name in a study of sexual minority participants) causing participants to have their personal information revealed. If participants do not read the consents forms, they may be unaware of the potential risks they may encounter from participation and therefore experience unwanted distress, unintended participation, or even loss of a soul.2

HSRBs require certain materials to be present in consent forms and if participants do not read these forms the information contained may be irrelevant. While anecdotal experiences suggest that many HSRBs at institutions of higher learning have made careful efforts to improve consent procedures, perhaps it is possible to make consent form materials more accessible by working with psychological researchers to better understand how best to present important information regarding risks in a straightforward and understandable manner in both in-person and online formats. In one experimental study, Varnhagen and colleagues (2005) found that participants recalled more information from consent forms in brief formats as opposed to longer ones. Perhaps bulleted key information (e.g., “chunking,” Simon Rosser et al., 2009) can aid in recall and recognition of important information about the study prior to participation. While the Internet appears to be a comparable medium to traditional paper-and-pencil questionnaires for obtaining personality measures (Buchanan & Smith, 1999), we found differences via presentation medium, such that those reading paper consent forms in-person appeared to recall greater details within the form compared to those reading the consent form online (as evidenced of remembering the embedded term lucky charms). Thus, it may be especially important for researchers to take measures to promote reading of consent forms during online studies when researchers may not be present. Many researchers have also manipulated various presentational aspects of consent documents in an attempt to discover if some presentational methods are more effective than others. Aspects that have been manipulated include (but are not limited to): font type, font color, use of bolding and italics, inclusion of cartoons and other visual aids, variations in length, variations in writing level, and other format manipulations. Having simultaneous visual and auditory cues was found to be associated with better recall (Pavio & Sadoski, 2001). Perhaps adding a brief quiz to pass before entering the study is another option.

Limitations

Despite efforts to minimize confounds and enhance generalizability in the present study, limitations exist. While undergraduate students are populations often used in psychological research studies Arnett, 2008; Henrich et al., 2010), the college student sample composed primarily of Asian- and White-American participants limits the generalizability of these findings to more diverse populations in the community. In addition, students were part of courses that gave extra credit for research participation in multiple studies. Thus, motivation to obtain credit (rather than some intrinsic reward) may have affected participants’ engagement in this study and thus limited their attention to details in the consent forms. It is important to note that the recall phrase “lucky charms” may have had relatively little salience to participants. While they were informed they would be asked to recall the phrase later on, there were no incentives to remember the phrase. For example, if participants were told in the consent form that they needed to remember the phrase in order to receive their extra credit, findings may have differed. Also unknown is whether participants completed other studies accessible through the psychology subject pool and perhaps familiarity with or perceptions of these types of “low-risk” studies may have influenced their attention to consent form content. Prior research partly supports this idea as participants’ reasons for not reading the consent documents thoroughly include trust that researchers (be it a doctor in a medical setting or a scientist at a university) would not expose them to undue risk (Varnhagen et.al, 2005; Wogalter et al., 1999) and consideration that these documents are similar enough to one another and thus not necessary to read in detail (Varnhagen et.al, 2005).

The current study had potentially minimal risks in answering psychological questions compared to studies in biomedical field that could potentially pose greater risks including even potentially lethal risks. If participants understood the study to involve only one brief survey (which they likely knew from recruitment materials), the current results may not be generalizable to other studies beyond psychological survey research. In particular, these findings may not generalize to medical research or psychological intervention involving risks of potential physical or psychological harm. In order to standardize approaches across condition, the in-person researcher was available for participants if they had questions, but did not offer an additional explanation as is performed in many other research studies with in-person consent. Some items asked about in the quiz were more important than others (e.g., confidentiality procedures versus research assistant names) and it is unknown which items participants reported guessing at. Finally, the brief two-item measure of conscientiousness, although empirically supported, may be a limiting measure of the construct. This is especially evident given in the context of the other individual differences items, which contained more than two items when assessing factors of Self-determinism and social desirability.

In conclusion, the present study suggested that the majority of participants in psychological survey research may neglect to read consent forms thoroughly and therefore may not be fully aware of the risks, benefits, confidentiality procedures, and other important information about research studies prior to participation. The present results in combination with previous studies raise a number of important issues for consideration. First, how should we think about data, which has not been acquired following consent that is truly informed? Moreover, if participants do not read consent forms, how can they provide informed consent? This is especially relevant to online data collection where researchers are not present to provide highlights from the form or answer questions. Another issue worthy of discussion is the extent to which the lack of truly informed consent is an issue for most online studies, which tend to involve less risk in comparison to biomedical clinical trials. Simon Rosser and colleagues (2009) suggest that for highly sensitive material (e.g., sex research), online consent procedures may increase recruitment rates and promote more accurate reporting by giving participants perceived anonymity (i.e., they do not have to meet the researchers face-to-face) and reduced researchers demand characteristics. However, these researchers also suggested that participants may be less inclined to dedicate their full attention to study materials if participating online (e.g., multitasking while filling out a survey online, clicking through the consent material quickly to avoid others who may be around from seeing what the content refers to). These findings also have implications for researchers. In addition, if researchers can facilitate autonomy it may be possible to increase involvement in the consent process. Additional research will need to be conducted to determine which aspects of autonomy are most influential in the process and whether they can be enhanced as part of the consent process. Some work has shown that both autonomy and controlled motivations can be influenced with relatively simply priming tasks (e.g., Hodgins, Brown, & Carver, 2007; Weinstein & Hodgins, 2009). Conversely, if researchers can find ways to diminish controlled motivation in the consent process, it may enhance involvement. Again, additional research will need to determine specifically what dimensions will be most relevant.

Footnotes

1A copy of the quiz can be requested of the first author at ude.notgnihsaw.u@redepe

2On April 1, 2010, the online video game company, GameStation, added an “immortal soul clause” to the online buyer’s agreement. The clause, included as an April Fool’s Day joke, lead approximately 7,500 online shoppers to “releases their soul” to the U.K.-based company. It was estimated that only 12% of shoppers read the purchase agreement before indicating their agreement to the condition. See http://www.foxnews.com/scitech/2010/04/15/online-shoppers-unknowingly-sold-souls/

Contributor Information

Eric R. Pedersen, Department of Psychology, University of Washington.

Clayton Neighbors, Department of Psychology, University of Houston.

Judy Tidwell, Department of Psychology, University of Houston.

Ty Lostutter, Department of Psychiatry and Behavioral Sciences, University of Washington.

References

  • American Psychological Association. Ethical principles of psychologists and code of conduct. 2010 Retrieved February 2, 2011 from http://www.apa.org/ethics/code/index.aspx.
  • Appelbaum PS, Grisso T. Assessing the patient’s capacities to consent to treatment. New England Journal of Medicine. 1988;319:1635–1638. [Abstract] [Google Scholar]
  • Appelbaum PS, Roth LH. Competency to consent to research: A psychiatric overview. Archives of General Psychiatry. 1982;39:951–958. [Abstract] [Google Scholar]
  • Appelbaum PS, Roth LH, Lidz C. The therapeutic misconception: Informed consent in psychiatric research. International Journal of Law & Psychiatry. 1982;5:319–329. [Abstract] [Google Scholar]
  • Arnett JJ. The neglected 95%: why American psychology needs to become less American. American Psychologist. 2008;63:602–614. [Abstract] [Google Scholar]
  • Arscott K, Dagnan D, Kroese BS. Assessing the ability of people with a learning disability to give informed consent to treatment. Psychological Medicine. 1999;29:1367–1375. [Abstract] [Google Scholar]
  • Bober S, Grolnick W. Motivational factors related to differences in self-schemas. Motivation and Emotion. 1995;19:307–327. [Google Scholar]
  • Bruzzese JM, Fisher CB. Assessing and enhancing the research consent capacity of children and youth. Applied Developmental Science. 2003;7:13–26. [Google Scholar]
  • Buchanan T, Smith JL. Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology. 1999;90:125–144. [Abstract] [Google Scholar]
  • Cassileth BR, Zupkis RV, Sutton-Smith K, March V. Informed consent—why are its goals imperfectly realized? New England Journal of Medicine. 1980;302:896–900. [Abstract] [Google Scholar]
  • Christensen K, Haroun A, Schneiderman LJ, Jeste DV. Decision-making capacity for informed consent in the older population. The Bulletin of the American Academy of Psychiatry and the Law. 1995;23:353–365. [Abstract] [Google Scholar]
  • Crowne DP, Marlowe D. A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology. 1960;24:349–354. [Abstract] [Google Scholar]
  • Deci EL, Ryan RM. The general causality orientations scale: Self-determination in personality. Journal of Research in Personality. 1985;19:109–134. [Google Scholar]
  • Deci EL, Ryan RM. The “what” and “why” of goal pursuits: Human needs and the self determination of behavior. Psychological Inquiry. 2000;11:227–268. [Google Scholar]
  • Deci EL, Koestner R, Ryan RM. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin. 1999;125:627–668. [Abstract] [Google Scholar]
  • Deci EL, Schwartz AJ, Sheinman L, Ryan RM. An instrument to assess adults orientations toward control versus autonomy with children: Reflections on intrinsic motivation and perceived competence. Journal of Educational Psychology. 1981b;73:642–650. [Google Scholar]
  • Estey A, Wilkin G, Dossetor J. Are research subjects able to retain the information they are given during the consent process. Health Law Review. 1994;3:37–41. [Google Scholar]
  • Flory J, Emanuel E. Interventions to improve research participant’s understanding in informed consent for research: A systematic review. JAMA. 2004;292:1593–1601. [Abstract] [Google Scholar]
  • Furnham A. Relationship among four Big Five measures of different length. Psychological Reports. 2008;102:312–316. [Abstract] [Google Scholar]
  • Goldstein JD, Iso-Ahola SE. Determinants of parents’ “sideline rage” emotions and behaviors at youth soccer games. Journal of Applied Social Psychology. 2008;38:1442–1462. [Google Scholar]
  • Gosling SD, Vazire S, Srivastava S, John OP. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist. 2004;59:93–104. [Abstract] [Google Scholar]
  • Gosling SD, Rentfrow PJ, Swann WB., Jr A Very Brief Measure of the Big Five Personality Domains. Journal of Research in Personality. 2003;37:504–528. [Google Scholar]
  • Grisso T, Appelbaum PS. Assessing competence to consent to treatment: A guide for physicians and other health professionals. New York: Oxford University Press; 1998. [Google Scholar]
  • Henrich J, Heine SJ, Norenzayan A. The weirdest people in the world? Behavioral and Brain Sciences. 2010;33(2–3):61–83. [Abstract] [Google Scholar]
  • Hodgins GS, Koestner R, Duncan N. On the compatibility of autonomy and relatedness. Personality and Social Psychology Bulletin. 1996;22:227–237. [Google Scholar]
  • Hodgins HS, Knee CR. The integrating self and conscious experience. In: Deci EL, Ryan RM, editors. The motivation and self-determination of behavior: Theoretical and applied issues. University of Rochester Press; 2002. pp. 87–100. [Google Scholar]
  • Hodgins HS, Brown AB, Carver B. Autonomy and control motivation and self-esteem. Self and Identity. 2007;6:189–208. [Google Scholar]
  • Hopper KD, Zajdel M, Hulse SF, Yoanidis NR, TenHave TR, Labuski MR, et al. Interactive method of informing patients of the risks of intravenous contrast media. Radiology. 1994;192:67–71. [Abstract] [Google Scholar]
  • Knee CR, Neighbors C. Self-determination, perception of peer pressure, and drinking among college students. Journal of Applied Social Psychology. 2002;32:522–543. [Google Scholar]
  • Koestner R, Bernieri F, Zuckerman M. Self-regulation and consistency between attitudes, traits, and behaviors. Personality and Social Psychology Bulletin. 1992;18:52–59. [Google Scholar]
  • Kraut R, Olson J, Benaji M, Bruckman A, Cohen J, Couper M. Psychological research online: Report of the Board of Scientific Affairs’ Advisory Group on the Conduct of Research on the Internet. American Psychologist. 2004;59:105–117. [Abstract] [Google Scholar]
  • McCrae RR, Costa PT. Personality in adulthood: A five-factor theory perspective. 2. New York: Guilford Press; 2003. [Google Scholar]
  • Moser DJ, Schultz SK, Arndt S, Benjamin ML, Fleming FW, Brems CS, et al. Capacity to provide informed consent for participation in schizophrenia and HIV research. American Journal of Psychiatry. 2002;159:1201–1207. [Abstract] [Google Scholar]
  • Neighbors C, Vietor NA, Knee CR. A motivational model of driving anger and aggression. Personality and Social Psychology Bulletin. 2002;28:324–335. [Google Scholar]
  • Newman E, Kaloupek D. Overview of research addressing ethical dimensions of participation in traumatic stress studies: autonomy and beneficence. Journal of Traumatic Stress. 2009;22:595–602. [Abstract] [Google Scholar]
  • Olver IN, Whitford HS, Denson LA, Peterson MJ, Olver SI. Improving informed consent to chemotherapy: A randomized controlled trial of written information versus an interactive multimedia CD-ROM. Patient Education and Counseling. 2009;74:197–204. [Abstract] [Google Scholar]
  • Pavio A, Sadoski M. Imagery and text: a dual coding theory of reading and writing. Mahwah, NJ: Erlbaum; 2001. [Google Scholar]
  • Pincus HA, Lieberman JA, Ferris S. Ethics in Psychiatric Research: A Resource Manual for Human Subjects Protection. Washington, DC: American Psychiatric Association; 1999. [Google Scholar]
  • Ryan RM, Deci EL. Overview of self-determination theory: An organismic dialectical perspective. In: Deci EL, Ryan RM, editors. Handbook of self-determination research. University of Rochester Press; 2002. pp. 87–100. [Google Scholar]
  • Silva MC, Sorrell JM. Enhancing comprehension of information for informed consent: A review of empirical research. IRB: A Review of Human Subjects Research. 1988;10:1–5. [Abstract] [Google Scholar]
  • Simon Rosser BR, Gurak L, Horvath KJ, Oakes JM, Konstan J, Danilenko GP. The challenges of ensuring participant consent in Internet-based sex studies: A case study of the Men’s INTernet Sex (MINTS-I and II) studies. Journal of Computer-Mediated Communication. 2009;14:602–626. [Europe PMC free article] [Abstract] [Google Scholar]
  • Stanley BH, Guido JR. In: Research ethics: A psychological approach. Stanley BH, Sieber JE, Melton GB, editors. Lincoln: University of Nebraska Press; 1996. pp. 105–128. [Google Scholar]
  • Stober J. The Social Desirability Scale-17 (SDS-17): Development and first results on reliability and validity. Diagnostica. 1999;45:173–177. [Google Scholar]
  • Stöber J. The Social Desirability Scale-17 (SDS17): Convergent validity, discriminant validity, and relationship with age. European Journal of Psychological Assessment. 2001;17:222–232. [Google Scholar]
  • Stretcher V. Internet methods for delivering behavioral and helath-related interventions (e-Health) Annual Review of Clinical Psychology. 2007;3:53–76. [Abstract] [Google Scholar]
  • Sugarman J, McCrory DC, Hubal RC. Getting meaningful informed consent from older adults: A structured literature review of empirical research. Journal of American Geriatrics Society. 1998;46:517–524. [Abstract] [Google Scholar]
  • United States Census Bureau. [Accessed February 2, 2011];2005–2009 American Community Survey 5-Year Estimates. 2009 from http://factfinder.census.gov/
  • Varnhagen MG, Daniels J, Peters TC, Parmar N, Law D, Hirsch R, et al. How informed is online informed consent? Ethics & Behavior. 2005;15:37–48. [Abstract] [Google Scholar]
  • Verheggen FW, van Wijmen FC. Informed consent in clinical trials. Health Policy. 1996;36:131–153. [Abstract] [Google Scholar]
  • Waggoner WC, Mayo DM. Who understands? A survey of 25 words or phrases commonly used in proposed clinical research consent forms. IRB: A Review of Human Subjects Research. 1995;17:6–9. [Abstract] [Google Scholar]
  • Weinstein N, Hodgins HS. The moderating role of autonomy and control on the benefits of written emotional expression. Personality and Social Psychology Bulletin. 2009;35:351–364. [Abstract] [Google Scholar]
  • Wogalter MS, Howe JE, Sifuentes AH, Luginbuhl J. On the adequacy of legal documents: Factors that influence informed consent. Ergonomics. 1999;42:593–613. [Google Scholar]

Citations & impact 


Impact metrics

Jump to Citations

Citations of article over time

Alternative metrics

Altmetric item for https://www.altmetric.com/details/2227039
Altmetric
Discover the attention surrounding your research
https://www.altmetric.com/details/2227039

Smart citations by scite.ai
Smart citations by scite.ai include citation statements extracted from the full text of the citing article. The number of the statements may be higher than the number of citations provided by EuropePMC if one paper cites another multiple times or lower if scite has not yet processed some of the citing articles.
Explore citation contexts and check if this article has been supported or disputed.
https://scite.ai/reports/10.1080/10508422.2011.585601

Supporting
Mentioning
Contrasting
3
26
1

Article citations


Go to all (8) article citations

Similar Articles 


To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.


Funding 


Funders who supported this work.

NIAAA NIH HHS (1)