Navigation – Plan du site

Questionnaire Pretesting Methods

Do Different Techniques and Different Organizations Produce Similar Results?*
Méthodes de pré-test de questionnaire. Les différentes techniques et organisations produisent-elles des résultats semblables ?
Jennifer M. Rothgeb, Gordon Willis et Barbara Forsyth
p. 5-31


Depuis 20 ans, il y a eu un accroissement significatif dans l’utilisation de formes nouvelles de pré-tests de questionnaires. Diverses techniques d’évaluations ont été utilisées pour signaler les points forts et faibles de ces méthodes. Des études limitées ont été faites sur l’efficacité des interviews cognitifs dans la résolution des problèmes de questionnaire. Mais ces études n’ont pas distingué les effets des méthodes des effets de l’organisation. L’objectif de cette recherche est de déterminer comment des méthodes diverses utilisées par trois instituts d’enquête par questionnaire (U.S. Census Bureau, Westat Inc. et Research Triangle Institute) arrivent à prédire les problèmes. Pour cette recherche, plusieurs chercheurs dans chacun des instituts ont utilisé trois méthodes de pré-test (revue informelle par experts, un système formel d’évaluation cognitive, et l’interview cognitif) sur trois questionnaires dans une expérience tridimensionnelle de tris multiples. Une classification est faite des problèmes identifiés par les méthodes et ces problèmes sont comparés les uns aux autres. Nous calculons le degré d’accord entre les instituts et entre les méthodes. Nous concluons sur l’utilité et l’adéquation de chaque méthode, et le degré d’accord et le caractère systématique de ces résultats.

Haut de page

Texte intégral


  • * This paper reports the results of research and analysis undertaken by Census Bureau staff and colle (...)

1During the past 20 years, in an effort to improve survey data quality, researchers and survey practitioners have significantly increased their use of an evolving set of questionnaire pretesting methods, including review by experts, cognitive interviewing, behavior coding, and the use of respondent debriefing. Several researchers have addressed issues related to questionnaire evaluation, and have attempted to determine the potential strengths and weaknesses of each (Campanelli, 1997; DeMaio, Mathiowetz, Rothgeb, Beach, and Durant, 1993; Esposito and Rothgeb, 1997; Forsyth and Lessler, 1991; Oksenberg, Cannell, and Kalton, 1991; Presser and Blair, 1994; Schaeffer and Dykema 2004; Willis, 2005). Further, several empirical investigations have evaluated the effectiveness of core features of these techniques, (Conrad and Blair, 1997; Davis and DeMaio,1993; DeMaio and Landreth, 2004; Foddy, 1996; Fowler, 1992) and several evaluative studies have attempted to assess the effectiveness of cognitive interviews in ameliorating questionnaire problems (Beatty, Willis, and Schechter, 1997; Forsyth, Rothgeb, and Willis, 2004; Fowler and Cosenza, 2000; Lessler, Tourangeau, and Salter, 1989; Presser and Blair, 1994; Willis and Schechter, 1997; Willis, Schechter, and Whitaker, 1999); these are reviewed in detail by Willis (2005). In the last ten years much has been written specifically about the cognitive interviewing technique regarding its theoretical underpinnings, merits, application of probing methods, and interpretation, utility and quality of results (Beatty, 2004; Beatty and Willis, 2007; Conrad and Blair, 1997; Gerber and Wellens, 1997; Tucker, 1997; and Willis, 2005.)

2Increasingly, evaluations of pretesting methods have focused on the side-by-side comparison of techniques, in order to determine the degree to which the results obtained through use of these techniques agree, even if they cannot be directly validated. However, this research is complex, as evaluation in practice must take into account the multi-faceted nature of each of the pretesting techniques, and of questionnaire design in general (see Willis, DeMaio, and Harris-Kojetin, 1999). Although two studies (Presser and Blair, 1994; Willis, 2005) have specifically compared the results of cognitive interviewing, expert evaluation, and behavior coding, when these have been applied to the same questionnaire, this research has generally not been conducted in a way that allows for the separation of the effects of pretesting method from those of the organization applying these methods.

3For example, Presser and Blair used expert panels whose members were different individuals than those conducting cognitive interviews, and who were in turn different from the coders who applied behavior coding. Thus, their finding that the expert panel discovered the greatest absolute number of problems, and cognitive interviewing the least, cannot be uniquely attributed to either pretesting technique or the individuals applying them. Similarly, Willis (2005) assessed cognitive interviewing at two survey organizations, as well as behavior coding, and individual-level (as opposed to group-based) expert review. Although this study obtained relatively good correspondence between pretesting techniques, in terms of identifying candidate questions that appeared problematic and in identifying the same qualitative categories of problems, the particular techniques were again confounded with the individuals using them.

  • 1 Throughout this paper we refer to the detection of “problems” in tested questions by the pretesting (...)

4The overall objective of the current study was to rectify this limitation, and to avoid an “apples and oranges” type of comparison. Overall the selected design balanced technique with organization, for the same set of questionnaires (see Lessler and Rothgeb, 1999; Rothgeb and Willis, 1999), to determine level of agreement among three question pretesting techniques, when applied by each of three survey research organizations (The Census Bureau, Westat, and Research Triangle Institute). Therefore, we would be able to investigate the independent effects of organization, and or techniques, under conditions of controlled questionnaire content. For this research, multiple staff members within each of these organizations utilized three pretesting methods: informal expert review, formal cognitive appraisal, and cognitiveiInterviewing. A classification scheme was then developed to code problems identified through any of the three methods, and by each organization1.


5The experimental design was developed in order to balance each major experimental factor, so as to render the analysis as unambiguous as possible. In particular, the overall requirement was to provide a form of balancing sufficient to enable a factorial combination of Technique, Organization, and Questionnaire; that is, each technique was applied by each of the organizations to each tested questionnaire. Further, it was decided that the use of three questionnaires on varied topics would, as well as making a Latin Square design possible, also increase generalizability of the results, with respect to the range of survey questions to which the results would meaningfully apply. The Latin Square design developed is represented in Table 1. Each organization selected three researchers, and each of these researchers applied one of the depicted sequences. It was decided that each of the three researchers would evaluate all three questionnaires, and each would use all three techniques. Further, the established sequences could be replicated across each of the three organizations, so that the design table was simply repeated a total of three times.

Table 1. Latin Square-based Experimental Design: Procedure used in each of the three organizations

Within each

Expert review



Researcher 1

(Questionnaire A)

(Questionnaire B)

(Questionnaire C)

Researcher 2

(Questionnaire C)

(Questionnaire A)

(Questionnaire B)

Researcher 3

(Questionnaire B)

(Questionnaire C)

(Questionnaire A)

6Finally, each researcher applied an invariant ordering of techniques, starting with expert review, then forms appraisal, and finally, cognitive interviewing, rather than varying this ordering. This was done partly to reflect the ordering of techniques within usual survey pretesting practice. Further, we chose not to vary the ordering of pretesting techniques because this would, in some cases, present the forms appraisal system prior to expert review, producing a source of an undesirable carryover effect, as learning the (formal) forms appraisal system would very likely influence the evaluator’s (informal) expert review activities, even when applied to a different questionnaire. On the other hand, this design resulted in the switching of the questionnaire content (between A, B, and C) for each evaluation trial, from the perspective of each evaluator, and therefore did not take advantage of the natural progression across techniques that evaluators normally experience as they apply these techniques to a single questionnaire. However, this limitation was viewed as an acceptable compromise, as the design selected allowed for the control of Pretesting Technique and Organization as the main factors of interest, and in particular, retained an uncontaminated factorial combination of Technique, Organization, and Questionnaire in a relatively efficient manner.


7Staff participating in the research consisted of a lead senior methodologist at each organization along with two other researchers at each. All participating staff had previously conducted expert reviews and cognitive interviews for other questionnaire-design projects.

Survey Instruments

8We selected a total of 83 items which were distributed among three questionnaire modules on different survey topics, deliberately choosing subject matter with which none of the participating researchers had substantial experience. A subset of questions about expenses for telephones and owned automobiles was extracted from the U.S. Census Bureau’s 1998 Consumer Expenditure Survey. Questions on transportation were extracted from the U.S. Department of Transportation’s 1995 National Public Transportation Survey. Finally, questions pertaining to attitudes about environmental issues were extracted from the U.S. Environmental Protection Agency’s 1999 Urban Environmental Issues Survey. We selected topics which could be administered to the general population by telephone and which contained very few skip patterns so as to maximize the number of sample cases receiving each question.

Pretesting Techniques

9We chose to evaluate questionnaire pretesting techniques that are commonly used following initial questionnaire drafting. Expert review and cognitive interviewing are very frequently applied in Federal cognitive laboratories, and we decided to also include the forms appraisal method, which is more systematic than an expert review, but less labor intensive than cognitive interviewing.

Expert Review

10The first method used in evaluating the questionnaires was informal, individually-based expert review. Participating researchers each independently conducted an expert review on an assigned questionnaire (A, B, or C in Table 1), and determined whether he/she thought each questionnaire item was problematic. The questionnaire review form was designed so that each item was accompanied by a ’problem indicator box’ which the researcher marked if he/she perceived a potential problem with the item, for either the interviewer or the respondent. Space was also provided under each question for the researcher to write specific notes about the suspected problem. No other specific instructions were provided to the researchers conducting the expert review, except for a short description of overall questionnaire goals. Each of the three researchers at each of the three organizations completed one expert review on one assigned questionnaire module.

Forms Appraisal

11For the forms appraisal, we utilized the Questionnaire Appraisal System (QAS) developed by Research Triangle Institute (RTI) for evaluation of draft questions for the CDC Behavioral Risk Factor Surveillance System (BRFSS). The QAS is intended mainly as a teaching tool for relatively novice questionnaire designers, and as a resource to be used by more experienced individuals. Overall, it provides a guided, checklist-based means of identifying potential flaws in survey questions (See Attachment A for a copy of the QAS.) For each survey question to be evaluated, the researcher completes a QAS form that leads the user to consider specific characteristics of the question and the researcher decides whether the item may be problematic with respect to that characteristic. There are eight general dimensions on which each item is evaluated: Reading, Instructions, Clarity, Assumptions, Knowledge/ Memory, Sensitivity/Bias, Response Categories, and Other. Within each of the eight dimensions there are several sub-dimensions for which the researcher evaluates the item, for a total of 26 separate “checks” for each survey question. For each check, the researcher circles a Yes/No box indicating whether the item is perceived to be problematic. In addition, when a “yes” is marked, the researcher also enters notes about the nature of the potential problem. The QAS was developed in order to provide multiple means for detecting problems, rather than minimizing overlap between coding categories.

12Because most of the participating researchers did not have prior experience with the QAS forms appraisal, we provided each researcher with a self-study manual. In addition, researchers completed a few practice exercise test questions using the forms appraisal, and their completed work was reviewed by the project manager at that organization. Then researchers were given their assigned module, additional instructions, and QAS forms to complete. Each of the three researchers at the three organizations completed a QAS for each questionnaire item in their assigned module.

Cognitive Interviews

13Our third pretesting method was cognitive interviewing. Each organization independently developed a cognitive interview protocol, after expert reviews and forms appraisals had been completed. Because there is variation between organizations in the degree of use of scripted versus unscripted probing, and in the manner in which results are summarized, we did not attempt to standardize these aspects of the research, as such differences between organizations were of interest. Each organization independently recruited research subjects. Each interview was expected to last approximately one hour. Cognitive interviews were conducted both in the organizations’ cognitive laboratories and off-site at locations convenient to subjects. All laboratory subjects were either staff of the organizations, or members of the general population who were 18 years of age or older. Each of the three researchers within each organization conducted three cognitive interviews with their assigned modules. As with the other testing techniques, researchers marked a problem indicator box after each questionnaire item, for each interview, when they believed that a potential problem existed, and entered open-ended written comments for marked questions.

14After the three cognitive interviews at each organizations were completed, the head researcher from each organization reviewed and summarized these results, by making a determination of whether, for each tested item, significant problems had been detected for that item. We believed that this approach most closely replicates usual practice of cognitive interviewers, as results from all cognitive interviews (rather than each individual cognitive interview) conducted by a particular interviewer are typically evaluated in total to determine where in the questionnaire problems may exist. This practice also served to equate scores based on cognitive interview results with those from the expert review and forms appraisal, for which each questionnaire item was coded only once by each technique as a potential problem.


Item Summary Score computation

15The initial level of analysis involved only the number of problems identified as problematic, and not the qualitative nature of problems. In order to determine whether pretesting techniques were consistent in their identification of individual problems as problematic, each item was given a dichotomous score (Problem versus No-Problem) by each researcher, for each of the three pretesting techniques (expert review, forms appraisal, and cognitive interviews). Then, for each of the 83 items across the three questionnaires, a Summary Score consisting of the total number of times a problem was assigned was assessed. Summary scores were computed both by assessing: a) the number of organizations that identified a problem, under each technique (e.g., Census, RTI, and/or Westat under cognitive interviewing); and b) the number of Techniques that identified that item as problematic, within each Organization (e.g., whether Expert Review, Forms Appraisal, and/or Cognitive Interviewing identified the item, when tested at the Census Bureau). Each of these scores could therefore range between 0 and 3.

Analysis of Summary Scores

16The foundation of our analysis was based on the Summary Scores for each pretesting Technique, and for each research Organization. In our analysis we examined differences between mean item scores (through ANOVA), and correlations between item scores. Results of each are described below.

Analysis of differences between Pretesting Techniques

17The mean item scores (given a possible minimum of 0 and a maximum of 3) for each pretesting technique were as follows: a) Expert Review: 1.55; b) QAS: 2.93; c) Cognitive Interviews: 1.46. These results suggest that the Question Appraisal System was the most productive in identifying potential questionnaire problems (on average, it found a problem in 2.93 of 3 possible opportunities, or 97.7%). Although the forms appraisal is very sensitive in detecting potential problems, one might question the specificity of this method: The fact that there is very little variation (basically every item was found to have one or more problems) seems to represent the “promiscuous use” of coding with this method. On the other hand, the means of the items scores for the expert review and cognitive interviews indicate that they both identified potential problems about half the time, on average.

18To determine whether the pretesting Techniques found significantly different numbers of problems, and whether they found different number of problems in each of the three questionnaire modules, analysis of variance (ANOVA) was conducted. The unit of analysis, or “case,” was the questionnaire item; the independent variables were questionnaire module and pretesting technique; and the dependent variable was the Summary Score, or number of times each item was flagged as having a problem (0-3). The Questionnaire (A, B, or C) was equivalent to the ‘between-subject’ factor and pretesting technique the ‘within-subject’ or repeated measures factor. The ANOVA results indicated that questionnaire module had no overall effect on problem identification frequency, but there was a large difference by pretesting technique (F=92.8, p<.001). There was no significant interaction between questionnaire module and pretesting technique (F=1.8, p<.13).

19To determine where differences were within the overall pretesting technique effect, a two-way ANOVA was conducted among the pairs of pretesting techniques. ANOVA results for expert review versus cognitive interviewing indicated no significant differences, and a marginal interaction between questionnaire module and pretesting technique (F=2.78, p<.07). ANOVA results for expert review versus forms appraisal indicated a large difference (F=157.60, p<.001) between item scores for expert review and the forms appraisal, independent of the questionnaire module (F=1.98, p<.14). Similarly, ANOVA results comparing items scores between forms appraisal versus cognitive interviewing revealed a large difference (F=153.03, p<.001) between the two techniques, independent of questionnaire modules (F=.23, p<.4).

20Spearman correlation analyses were then conducted to determine the degree to which the different pretesting techniques determined the same questionnaire items to be problematic. Because of ceiling effects (and resultant restriction in range) of the item scores for the forms appraisal, only the expert review and cognitive interviews could be meaningfully correlated. The correlation coefficient for Spearman’s r between the summary scores for expert review and cognitive interviews was .27 (p<.02), demonstrating positive, but low correlation between the two methods in the items identified as problematic.

Analysis of Differences Between Research Organizations

21Similar to the test of differences as a function of Technique, the mean scores (range of 0-3) for each research organization were as follows: a) Census: 1.95; b) RTI: 2.02; c) Westat: 1.96. The similarity in the mean scores demonstrates that a comparable criterion level in identifying problems was adopted, overall, across organizations. Analysis of variance conducted to determine whether the research organizations obtained different number of problems and whether they found the same or different number in each of the three questionnaire modules revealed no significant effect of questionnaire module, organization, or interaction between module and organization.

22Spearman correlations between the item Summary Scores produced by different organizations (across all pretesting techniques) were very similar, and all lo0he same or,

dy one hour. Cog./spYizan theQestio questian>

dy.6oreethe horsticetween the ian>

dy.5d WibinsationalAian>

dy.7 of respon/a>22Our third prets organipretesto maBeatty, 2004or 9ray>dy bta quhcontr veryanizationsOcompleted a Qng, acombiiran expeir idean to that charactehcoBeatved thatcs which ut expenses.

VA),iing iniall lo0osting teel in identifyi eac limitatione teg thmber of estip cttitudch the to that characteferentterviewer arre diser. Because of ceTopresench theiciobmatic wunutel questechniqu which.Analysis oe Coding of Problems

16After the thre rms app chnigg ited whether the research anizationsree, even ifon.

differenmber of problems in each of theproblematattemartive inh organiey Willisr thehe item is percerms appraisrisos l:lang="en" lang="en"> revealed a ln each of thd faatest by the codree, even ifon.

dd, well lo0osh anizationceTop degree to whiundesirablary Sential probwhether t alsonitive intategories of applica. The Qow.cores dne ">n class)be administered t same questiongnitive inthniques. Fchniques. ANOVA results limitatiwould, ieatty,ndesirct a coinr stalso include ategories oan exdeert review15were distributed a problematic.Analysis os an scheme was thblems

4The initial leveln scheme was thblemscores d, in (othgeb, 1999essler, 199.)ty in28 l5 worst dd, wexpered opdentirhighenceosed techniqu whifatesiarher fetichniquesme or-saturaoratories rories, ap ctl1.95;s theehhich te intermmune was t,; c)hnier.vexpec) judghree qundand behavior codid) rories, acs whiy evend Salter, 1198, ANf the eight dimensie or satur iser. midposed tcd Other. Withinidentir, ecieosed teuctive inlis (2edof overall questraisndder t;eview aeceivi--erms appraischnif questOVA ringsegre;evaluatisequour.l q;evaluatisiewlwkwartheymiaxl:lang="en" lang="en"> revealed a lded that imvey. deth identirl5 worst dbeof questionnaire conta nd thf respider no sb sequences couy Scorelving setyworst dstribumatic wpossible.ysis etest muchores dh s,entirl5 w in order to provide multiandardize tnumber ofh org-tituring three t to the range of assessportatiognitive interd> notecall" id="bodyftn2" href="#ftn3">(...)<3li2

Analysis o7hre probing methol5 wSn devoof transport

4The foundation ofues. Flr from each orsiewedoblemgund thof assessrl5 worst dhores d15 cs which ere d detegnitivt reviewitten commeom each orghe natctechrmal) formach of thspidected for rough any of the thhree pretesting techniquehe items ization(...)<4li3

Analysis o8

th this meten devef probing m

21The mean item deter2 >for a tt there wreview,ree pretest27 l5 worst ddssessedross organihores d15 cs which ere dnterllh evalanpentir,r from each orsiproblem, undeeparate 338  whether thtestinte tohecks” (e mean sc XsOcompleted a)of pretestin of 26 span>

te .5 orst dpire-design . Ft re deter2,p chnigc(forrable crsnce oblems in eorst ddlti-fases apprence (F whvey.i methods, of theproblemat de xeorst rais (Dnizieweetfo are typicallyr meo the gen, Vaghin-ould/d f(forr-design , Uhnif que/vaghin-gre, Uhnif que detec3, p&ctormoreHighnlis (2was to pd/ividual.i meunav (2arease e oErr cole. Inowledge/) l:landlti-fases app69.9  e prrtatioce oSential probwhether

Table 1. Latin Squ2. Ftthere wrthol5Rhblet ddssessedyr m15wve in idhin each ere dng>

from3n5" doquelexe/ourge/348/oug-1.jpg"ffic="icspSetan oug src" doquelexe/ourge/348/oug-1-snce 480.jpg"falt="ltr" u2. Ftthere wrthol5Rhblet ddssessedyr m15wve in idhin each ere d" /> xtandnotes"> IcspAccthgocfrofic="noensus:e"> icspZoom"om3n5" doquelexe/ourge/348/oug-1.jpg">Ag/spdir frofic="noensus:e"> icspOrig"om3n5" doquelexe/ourge/348/oug-1.jpg">Origion b(jpeg, 56k

Icspanfrom3n5" doquelexe/ourge/348/oug-2.jpg"ffic="icspSetan oug src" doquelexe/ourge/348/oug-2-snce 480.jpg"falt="" /> xtandnotes"> IcspAccthgocfrofic="noensus:e"> icspZoom"om3n5" doquelexe/ourge/348/oug-2.jpg">Ag/spdir frofic="noensus:e"> icspOrig"om3n5" doquelexe/ourge/348/oug-2.jpg">Origion b(jpeg, 36k

21To determine Noboth ide of tiques, (orst dsboth ber"ject managl5 wto expeas;s theehhich t/ermmune was t restrier.vexd whether they articipatin, (orst dsbothrt reviewjudghree qsaturaeasurries, act/p> establishewo(orst d(vaghin-ould/d f(forr-design they uhnif que/vaghin-gre)ddlti-fas app31.4 e prrtatioce oorder to d suggest that a in their ideneviewthe expgnitivePs te thf reBlair (1994)1999), to dsed cog,years one ovaghintivenner ic mean swhetorial cevieirtategories of applicst that aing a

Analysis o9 Differences BMid-Lsed t15 wnd Other.

21To determine iling effecraisndue wasrd to ersunce oc a Lsr osnidentir, ecieosed th this mepentir28rl5 worst dsbothh qe by upyr m11was midposed ” problems, rovfor a copy le established s probleyrais Err cole. Knowledge/ l:lanepeateecks”aryutnitiveaisal SysSd QASurial prohecks” probleygn seleost clelts oupnd suf whiy evpecificfor evaa ategories oan hecks” probleyovice q cog seepecchniql was h oringssd QASuriaividupecC theehhich te intCrmmune was t,; are sumrrelog seedirwhiniquo pretermmune was trable carryql was sd QASuri.h organ3 >for a ttriction icd Other. Witinch of the organiPhniques. ANOVA resu

22Spearman corrFt re deter3,p chesent t ideund signifianizatin>coree, onestraisndder tthese troblems eck, thaeceivnterviewing, rather te prior ex clelt whetherctare typicallyanizieweer. Wilatiws: a)pfltsdean detecting potential problems, one might qneview-design tthis resePs te thf reBlair (1994)19 eque or difsticrviewing, rather te prior exate s whetherctare typicallyorder to detnistabliblematiss very and cognitiven the blog seedetecting potential probmight qneview-design tsd QASuriaividual versus cognitonterviewing); and b) sac limitationesngobate

dd, weomewveryranguce que; agf indhin-oo whiunce osr osnersuesent a r t alser. gntigate tdetermim efor exChi-sdeveloique

Table 1. Latin Squ3.BMid-osed t15 woproblems, technique (F=92.8, p&lng>

border:none;" dir="id #000000;border-right:none;border-top:1px solid #000000;border-bottom:none;" dir="l0000;border-bottom


tr" e">22Similar to th deter4 promist preterangbuas trablmid-osed t15 woproblems, te In our analysFewas a function clelt htestinndependent oso problem agf inlematiss very lieterangbuas trablificatiorisos theo, well lo0ozations. Analysis ofaividu chnigzationschnique (F=92.8, p&l

Table 1. Latin Squ4.BMid-osed t15 woproblems, tenizations obtained ding>

(Questionnair xml:lanas Midposed ”t15 woprobley

(Questionnair xml:lan

(Questionnair xml:lanQAS/span>

(Questionnair xml:lanerviews >

(Questionnair xml:lanTf times a problorst ddssessed

(QuestionnairIe typicallyanizieweer.




(Questionnairn xm

(QuestionnairQdesign tthis re





(QuestionnairQdesign tsd QASuri




(Questionnair39 xm

(QuestionnairRetec3, p&ctormo




(Questionnair3 xm






(QuestionnairJudgcopy/E pretesti




(Questionnairn xm

(QuestionnairReries, aTgree ology




(Questionnair xm

(QuestionnairReries, aUnit





(QuestionnairReries, aSd QASuri




(Questionnairn xm

(QuestionnairErr cole. Knowledge/




(Questionnair3 xm






(QuestionnairTf ti





(Questionnair(n) (Questionnair65 (Questionnair19 xm
(Questionnair33 xm

(Questionnair xml:lang="en" lang="en">Table 1. LatMidBMid-osed t15 woprobleyng>

(Questionnair xml:lang="en" lang="en">Table 1. Lat). Eac/span>

(Questionnair xml:lang="en" lang="en">Table 1. LatRTI/span>

(Questionnair xml:lang="en" lang="en">Table 1. Lat The s/span>

(Questionnair xml:lang="en" lang="en">Table 1. Latif times a problorst ddssessed

(Questionnair xml:lang="en" lang="en">Table 1. LatIe typicallyanizieweer. >

(Questionnair x"en" lang="en">Table 1. Lat6.6%

(Questionnair x"en" lang="en">Table 1. Lat10.8%

(Questionnair x"en" lang="en">Table 1. Lat7.8%

(Questionnair x"en" lang="en">Table 1. Latn xml:lang

(Questionnair xml:lang="en" lang="en">Table 1. LatQdesign tthis re >

(Questionnair x"en" lang="en">Table 1. Lpa3 .4%

(Questionnair x"en" lang="en">Table 1. Lat33.3%

(Questionnair x"en" lang="en">Table 1. Lat30.4%

(Questionnair x"en" lang="en">Table 1. Lat107xml:lang

(Questionnair xml:lanQdesign tsd QASuri





(Questionnair xml:lanRetec3, p&ctormo




(Questionnair3 xm

(Questionnair xml:lanReier.vex





(Questionnair xml:lanJudgcopy/E pretesti




(Questionnair1 xm

(Questionnair xml:lanReries, aTgree ology




(Questionnair xm

(Questionnair xml:lanReries, aUnit





(Questionnair xml:lanReries, aSd QASuri





(Questionnair xml:lan




(Questionnair3 xm

(Questionnair xml:lanOablil:





(Questionnair xml:lanTf ti





(Questionnair xml:lani (Questionnair121xm border:none;" dir="id #000000;border-right:none;border-top:1px solid #000000;border-bottom:none;" dir="l0000;border-bottom

(Questionnair102xm border:none;" dir="id #000000;border-right:none;border-top:1px solid #000000;border-bottom:none;" dir="l0000;border-bottom


tr" h3">(Questionnair a href="#tocfrom3n10"dir="tocto3n10">Analysisrobl15 wCproblems, at Hderest Cod (F=Lsed t(Cognitive Process (F=Model) e">22Similar to thFinally,y lietata were furablibcollapsed accdir (F=9o eaons fy liestagos n a four-stagobcognitive reries, amodel. Notes very lieificatis dentified as “err cole. anowledge/ ” and “someth (F=else” are excluded from ltr" s 5 and 6. In addidge/, due=9o small cell siznt o lie’judgcopy and e pretesti’, and ’reries, aselecesti’eificatis were collapsed. ltr" 5 showst preterangbuas trablificatis dentified accdir (F=9o chnique (F=T2.8, p&l Chi-square ique (F=drdenot reveal a he seraically sessificapy associtesti betweenwoprobleyrablificatio dentified and T2.8, p<(Chi-sq (4) =4.99, p<.29). However o liemost compell (F=rerultnlematiss o bes veryificatis relprods o comprehenssti and communicaesti are ilieoverwhelm (F=majlemtyrablificatis dentified, whionsisrces,eraopy with finr (Fs from atililibnizationsby Pressliband Blair (1994) and Will t o(2005)

eillustration">Table 1. Latin Squ5.l15 wHderest osed t1od (F=CprobleyrDerangbuas t,sby Preique (F=T2.8, p&l >

a href="docapnexe/imago/348/img-3-small580.jpg" rel="icspSetan img src="docapnexe/imago/348/img-3-small480.jpg" alt="ltr" u5.l15 wHderest osed t1od (F=CprobleyrDerangbuas t,sby Preique (F=T2.8, p&l" /> icspZoom" href="docapnexe/imago/348/img-3-small580.jpg">Ag/sp icspOrde" href="docapnexe/imago/348/img-3.jpg">Ordeinal (jpeg, 48k) div
e">22Similar to thFinally,y deter6eterplayst preterangbuas trablificatis dentified ato liemost general cognitive reries, amodel osed tby oIn our analysOverallo lire was no associtesti betweenw lielemlicaesti oblorst ddnd In our anal (chi-sq(4) =4.384, p<.357)

eillustration">Table 1. Latin Squ6.l15 wHderest Lsed t1od (F=CprobleyrDerangbuas t,sby Oobtained ding>

a href="docapnexe/imago/348/img-4-small580.jpg" rel="icspSetan img src="docapnexe/imago/348/img-4-small480.jpg" alt="ltr" u6.l15 wHderest Lsed t1od (F=CprobleyrDerangbuas t,sby Oobtained di" /> icspZoom" href="docapnexe/imago/348/img-4-small580.jpg">Ag/sp icspOrde" href="docapnexe/imago/348/img-4.jpg">Ordeinal (jpeg, 44k) div
h1">(Questionnair a href="#tocfrom1n5"dir="tocto1n5">Dercuss dinga h1 h2">(Questionnair a href="#tocfrom2n6"dir="tocto2n6">Assesscopy obl‘ificati’ehe suss o qdesign s: Quapyie seve Analysisnga h2 h3">(Questionnair a href="#tocfrom3n11"dir="tocto3n11">Comparisti oblndependent e">22Similar to thInw liecurropy study,y liethreeschnique (F=92.8, p&lsroblexpert review, qdesign elemraisal, and cognitive ie typica (F=revealed (us (F= pretechor="le. PficatioAssesscopy inr caeor)= preQdesign tAemraisal Syraoms o bes viemost “ifiductive” n dentify (F=qdesign eificatis. However ogivenw lielasremely hder freqdency with whionsth inndependen dniqcrodsificatis,u chnigvery posser" u verysuonsa elemraisal method, as weelemliodsit,=may encouragoba lowethreshold forlificatio dentificaesti,o lirefo, wifiduc (F=a largobes a problfalse positives=rerults. Tlirefo, wweesuspqcrs very lieQAS method, as used, has hder sensstivmtyrbua poorlspqcificmty. Ichnigposser" u very preQdesign tAemraisal methodhnigzacurpro n tso dentificaestirablificatis, and very preqdesign s used forlth innizationsdogpossesst premanylificatis verysyraomsrevealed. However o chnigzlso likely veryaysyraomsprobles dnsessedemainly assa elids o preqdesign naire dnsesser orprblibividuaschnique (F=92.8, p&l oreqdires a fair degreesabladdidge/al expert judgcopy o besmracaically effective dur (F= prechnique phasl

22Similar to thHowever oevenw liefinr (Fsablvastly greproibif timificatis ny lieQAS issa elmbigule. finr (F,sbecauseo chnig nyone sensesa elrtifacas fy lieanalysisrprocedures used. Forlcurropy purposnt oa=qdesign ewas scdied as ificatiaescsby lieQAS ifo chfailods o “ier"” eaons fy26 se222Similar to thInroique (Fly,yexpert review and cognitive ie typica (F=ifiducodsvery ell lo0orerults inw liecurropy study,yinw ermso fy liepan>(Questionnair a href="#tocfrom3n12"dir="tocto3n12">Cos,eraopcyozations. Analysis of e">22Similar to thOne ie tyque (F=finr (Fsfrom liecurropy study was very prererults amo(Fsa Analysis ofawere flt eo, well lo0oividuwere 9prererults atestinndependentysOurefinr (Fs suggeues very lietea functsa Analysis ofauseoell lo0ocrmteria n dniqree (F=iosnctially ificatiaescsqdesign naire ctis, atolelstyinw ermso fygeneral ifiports trabl ctis seleceod. However o liemo, welssificapy issue thwherblib lietea functsa Analysis ofaseleceods liesame= ctis as havi(F=ificatis; and itrwas found veryselecestirablificatiaescs ctis aations. Analysis ofawas only moderpro n magnitude, and lowlibividu losesifeviously reportods n a comparisti oblnwo . Analysis ofaby Will to(2005). However onotes very lise he seraicalererults were based n etata havi(F=only four=iosnctial preeto(0, 1, 2, 3), and verya pree obl0awas used n ly witi=atestinnlie83 ctis, roduc (F= lieeffective overallo/spgie fy lierepenropy measuros o prees ctis. Aumber"icalererangcesti-in-/spgieeffectscouldo lirefo, wbe reries,er" uforlthe relprive modestyrablilieobsf iodsrelprstiships, and masksa mwonsgreproibtegreesablllplicmtlagreecopy aations. Analysis of

(Questionnair a href="#tocfrom3n13"dir="tocto3n13">Assesscopy oblrisorablificatio(15 woode)s o 15 wosst qdesign s: Qualie seve analysisnga h3

e">22Similar to thCber"ificaestirabl preiisos ablificatis dentified proughy liethreeschnique (F=92.8, p&lsrifiducodsie tyque (F=rerults ato lie’middle-osed ’eablqualie seve cod (F=analysis: Pficatis associteodswith qdesign tthis rerces,ertueods lies (Fle largost cprobleyrablificatis dniqcrodsforlallo lreesndependentysForlcognitive ie typica (F, qdesign tthis rercemprised nver halfy fy liereiqcrodsificatis,uwhereas forlthe orblib wo 92.8, p&lsrth incprobleyraccdunrodsforla lietliemo, wividuasquart problthe dentified ificatis. Tlielemraisal sclime and cognitive ie typica (F=reiqcrodsificatisswith informaestirreier.vexs o asgreproiblasnctwividudrdeexpert review. Expert review and lieQAS tenrods o reiqcr ie typicallyanizieweer. ,uwhereas cognitive ie typica (F=tenrodsnot toysOverall,y lieiisos ablificatis dentified proughycognitive ie typica (F=were hderly clusaorodswith n a few ificatiorisos,y liecproblems, ablificatis dentified proughyexpert review were somiwverylesstclusaorod, and lieQAS rerults were lielelstyclusaorodsinw l innigard (as chfound a mweertuderablificatis)

22Similar to thExaee aestirabl preiisos ablificatis found ato liemost general,ycognitive process (F=model osed tzlso demes,erprods verythmprehenssti and communicaesti ificatis were dentified o pregreprost lasnctwby allo lreesndependent,well lo0oio ifevioussfinr (Fs (Pressliband Blair, 1994; Will toxml:langem>et al em>g="en" lang="en">Similar to th,o2000). Notes very n a senses l inmay not beesurpris (F, ellply givenw liees a problif timorst ddevoted o p ingeneral cprobleyrinw lie15 wsyraomsproblwas dned opod.

(Questionnair a href="#tocfrom1n6"dir="tocto1n6">Coscstrs ts and Caveatsnga h1

e">22Similar to thBased n e9prererults abl p innizationsifijqcr,seaons fy liethreeschnique (F=methodsythisrgbuat dsomiwverytea functly olthe dentificaestirablificatisrinwsurvey=qdesign s,yinw ermso fy lieiisos ablificatis dentified. However o lies a functionwieobslivodswere largoly=qdapyie seve orprblibividuqualie seve;swith lll eodsvariaesti,o lise=92.8, p&lsr clelted o bes viemost useful inw fuhnii(F=ouryificatis relprods o qdesign tthmprehenssti,=atestinnlreesvery tea functsqdesign naires. Tlieobslivodscos,eraopcyoablrerults atestin. Analysis ofanigposnctially llportanr,sbecauseo p insuggeuet= very lire may zlso bescos,eraopcyoinw liewayst pery preiq.8, p&lsr , wbei(F=used, and lie aeuros fy liererults ifiducod. Tlierelprive lacks fythi,eraopcyozations. Analysis ofg nythoos (F=xml:langem>whion em>g="en" lang="en">Similar to th plrtieweo0o ctis werelificatiaescs dsomiwverytroubl (F, althoughyitscouldozlso besargued= very lire was very lietliedisagreecopy with reriqcrs o whions fy liseo ctis werelseverely flawod.

22Similar to thHowever o liecurropy study doeignot addresst wo furablibvi timqdesign s – (a) How dogwieknowethery lieificatis veryare dentified proughychnique (F=actually existyinw prefieldoenviroscopy, and (b) Evenwiblthe dentified ificatisyare “rep ”, wveryanow/spti=dogwiehave thery liemodificaestis verywe makes o lise=qdesign s slivos o rectify lise=ificatisswithouryzlso ie hiduc (F=new ones? Tlieformlibissue has beenwaddressodsvery ee mally (Davis and DeMaio, 1993; Will toand Sclichter, 1997), and lielprtlibissa ellmost completely unexpldied arep. An lasncsstirabl prestudy dercussodsinw l inlrtiele was undertfkenw o addresst lisiererationsqdesign s. Spqcifically,ywescosducted an expericopy in whionsthelordeinal 15 ificatiaescs ctis usodsinw l instudy,yas welloas revirs ts abl pose ctis, wereladmineraorods n a splie-sample expericopy. Analysisrobl prefieldorerults were evpreteod, us (F=severalsinrepenropy=ourcomiequaliey measuros (e.g., behaviorlcod (F, ie typicallyng><(Fsforms) Compari(F= liererults from atonschnique (F=methodhwith liererults abl prefieldostudy airodsus n dniqree (F=how wello prevariousschnique (F=methodsy dentified pieiisos ablificatis whionssurfacodsdur (F=fieldoique (F=(seesForsyth, Rorbgeb and Will t o2004.) Furabli,lcompari(F= lieourcomiequaliey measuros from liefieldostudy forlthe ordeinal and revised qdesign ewdir (Fs reveal wherblibrevirs ts in qdesign ewdir (F,sbased n echnique (F=rerults,=actually llprovodstata qualiey.

2god #0" href="#lrtiele-348thHauryderpage secestian

h2">(Questsecestianparanumber">2s xtthBibl ographil >

div">Beatty, P. (2004). The Dynamics of Cognitive Interviewing. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Methods for Testing and Evaluating Survey Questionnaires, pp 45-66. New York: Wiley.

Beatty, P., Willis, G. B., and Schechter, S. (1997). Evaluating the generalizability of cognitive interview findings. In Office of Management and Budget Seminar on Statistical Methodology in the Public Service, Statistical Policy Working Paper 26, pp. 353-362. Washington, DC: Statistical Policy Office.

Beatty, P., and Willis, G. (2007). The Practice of Cognitive Interviewing. Public Opinion Quarterly, 71: 288-311.

Campanelli, P. (1997). Testing survey questions: New directions in cognitive interviewing. Bulletin de Methodologie Sociologique, 55, 5-17.

Conrad, F., and Blair, J. (1997). From impressions to data: Increasing the objectivity of cognitive interviews. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1-9.

Davis, W. L., and DeMaio, T. J. (1993). Comparing the think-aloud interviewing technique with standard interviewing in the redesign of a dietary recall questionnaire. Proceedings of the Section on Survey Research Methods, American Statistical Association, 565-570.

DeMaio, T., Mathiowetz, N., Rothgeb, J., Beach, M. E., and Durant, S. (1993). Protocol for pretesting demographic surveys at the Census Bureau. Unpublished manuscript, Center for Survey Methods Research, U.S. Bureau of the Census.

DeMaio, T., and Landreth, L. (2004). Do different cognitive interview techniques produce different results. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Methods for Testing and Evaluating Survey Questionnaires, pp 89-108. New York: Wiley.

Esposito, J. L., and Rothgeb, J. M. (1997). Evaluating survey data: Making the transition from Pretesting to Quality Assessment. In L. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin (Eds.), Survey Measurement and Process Quality, pp 541-571. New York: Wiley.

Foddy, W. (1996). The In-Depth Testing of Survey Questions: A Critical Appraisal of Methods. Quality and Quantity, 30, 361-370.

Forsyth, B., and Lessler, J. (1991). Cognitive Laboratory Methods. In P. Biemer et al. (eds.), Measurement Errors in Surveys, New York: Wiley.

Forsyth, B., Rothgeb, J., and Willis, G. (2004). Does Pretesting Make a Difference? An Experimental Test. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Methods for Testing and Evaluating Survey Questionnaires, pp 525-546. New York: Wiley.

Fowler, F. J. (1992). How unclear terms affect survey data. Public Opinion Quarterly, 56: 218-231.

Fowler, F. J. and Cosenza, C. (2000). Evaluating the results of cognitive interviews. Proceedings of the Workshop on Quality Issues in Question Testing, Office for National for National Statistics, London, 35-41.

Gerber, E. R., and Wellens, T. R. (1997). Perspectives on pretesting: “Cognition” in the cognitive interview? Bulletin de Methodologie Sociologique, 55, 18-39.

Lessler, J.T., Tourangeau, R. and Salter, W. (1989). Questionnaire design research in the cognitive research laboratory. Vital and Health Statistics (Series 6, No. 1; DHHS Publication No. PHS-89-1076). Washington, DC: U.S. Government Printing Office.

Lessler, J. T., and Forsyth, B. H. (1996). A coding system for appraising questionnaires. In N. Schwarz and S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 259-291). San Francisco: Jossey-Bass.

Lessler, J. T., and Rothgeb, J. (1999). Integrating cognitive research into household survey design. In A New Agenda for Interdisciplinary Survey Research Methods: Proceedings of the CASM II Seminar. National Center for Health Statistics, pp. 67-69.

Oksenberg, L., Cannell, C., and Kalton, G. (1991). New Strategies for Pretesting Survey Questions. Journal of Official Statistics, 7, 3, pp. 349-365.

Presser, J., and Blair, J. (1994). Survey Pretesting: Do Different Methods Produce Different Results?, in P.V. Marsden (ed.), Sociological Methodology, Vol. 24, Washington, DC: American Sociological Association, pp. 73-104.

Rothgeb, J., and Willis, G. (1999). Evaluating pretesting techniques for finding and fixing questionnaire problems. Proceedings of the Workshop on Quality Issues in Question Testing, Office for National Statistics, London, 100-102.

Schaeffer, N. and Dykema, J. (2004). A Multiple-Method Approach to Improving the Clarity of Closely Related Concepts: Distinguishing Legal and Physical Custody of Children. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Methods for Testing and Evaluating Survey Questionnaires, pp 475-502. New York: Wiley

Tourangeau, R. (1984). Cognitive Science and Survey Methods. In T. Jabine et al. (eds.), Cognitive Aspects of Survey Design: Building a Bridge Between Disciplines, Washington: National Academy Press, pp. 73-100.

Tucker, C. (1997). Measurement issues surrounding the use of cognitive methods in survey research. Bulletin de Methodologie Sociologique, 55, 67-92.

Willis, G. B. (2005). Cognitive Interviewing. Thousand Oaks, CA: Sage.

Willis, G.B., and Schechter, S. (1997). Evaluation of Cognitive interviewing Techniques: Do the Results Generalize to the Field? Bulletin de Methodologie Sociologique, 55, pp. 40-66.

Willis, G.B., DeMaio T.J, and Harris-Kojetin B. (1999). Is the Bandwagon Headed to the Methodological Promised Land? Evaluating the Validity of Cognitive interviewing Techniques. In M. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. Tanur, and R. Tourangeau (Eds.). Cognition and Survey Research. New York: Wiley, 133-153.

Willis, G. B., S. Schechter, and K. Whitaker (1999). A Comparison of Cognitive Interviewing, Expert Review, and Behavior Coding: What do They Tell Us? American Statistical Association, Proceedings of the Section on Survey Research Methods.

Haut de page


* This paper reports the results of research and analysis undertaken by Census Bureau staff and colleagues. It has undergone a Census Bureau review more limited in scope than that given to official Census Bureau publications. This report is released to inform interested parties of ongoing research and to encourage discussion of work in progress. The views expressed are those of the author and not necessarily those of the U.S. Census Bureau.

1 Throughout this paper we refer to the detection of “problems” in tested questions by the pretesting techniques that were evaluated. We recognize, however, that the presence or absence of actual problems is unknown, given the absence of validation data. Rather, we use this terminology for purposes of labeling; that is, to indicate that the result of pretesting has been to designate the question as potentially having a problem.

2 Note that the lowest-level CCS codes are in fact very similar to those used in the QAS. This similarity may reflect a tendency for question coding systems to converge on a key set of problems that are relatively standard across questionnaires.

3 Although the QAS system provided its own coding system, only the QAS written notes were coded, in order to maintain consistency across pretesting techniques.

Haut de page

Table des illustrations

(Questionnair33 xm
Titre Table 2. Frequency of CCR Codes assigned to 15 most problematic items
Fichier image/jpeg, 56k
Fichier image/jpeg, 36k
Titre Table 5. CCS Highest level Coding Category Distribution, by Pretesting Technique
Fichier image/jpeg, 48k
Titre Table 6. CCS Highest Level Coding Category Distribution, by Organization
Fichier image/jpeg, 44k
Haut de page

Pour citer cet article

Référence papier

Jennifer M. Rothgeb, Gordon Willis et Barbara Forsyth, « Questionnaire Pretesting Methods », Bulletin de méthodologie sociologique, 96 | 2007, 5-31.

Référence électronique

Jennifer M. Rothgeb, Gordon Willis et Barbara Forsyth, « Questionnaire Pretesting Methods », Bulletin de méthodologie sociologique [En ligne], 96 | 2007, mis en ligne le 04 décembre 2008, consulté le 11 décembre 2017. URL :

Haut de page


Jennifer M. Rothgeb

U.S. Census Bureau,

Articles du même auteur

Gordon Willis

National Cancer Institute

Barbara Forsyth

University of Maryland Center for Advanced Study of Language

Haut de page

Droits d’auteur


Haut de page
  • OpenEdition Journals