Pretesting and Pilot data
This lesson looks at pilot studies and why they are crucial to a good study design.
Published on: Mar 4, 2016
Transcripts - Pretesting and Pilot data
Pr e t est i ng & anal ysi ng
pi l ot dat aDr Ki r st en Chal l i nor
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
This material has been reproduced and communicated to you by or on behalf of the
University of New South Wales pursuant to Part VB of the Copyright Act 1968 (the Act).
The material in this communication may be subject to copyright under the Act. Any further
reproduction or communication of this material by you may be the subject of copyright protection
under the Act.
Do not remove this notice.
Pi l ot st udi es
• What is a pilot study?
Mini version of a full-scale study = 'feasibility' study
Specific pre-testing of a particular research instrument
such as a questionnaire or interview schedule.
• Pilot studies are a crucial element of a good study
• Conducting a pilot study does not guarantee success in
the main study, but it does increase the likelihood.
Image from http://www.eh.uc.edu/cares/learn/pilot_study.html
"Do not take the risk.
Pilot test first.”
De Vaus (1993: 54)
Reasons t o pi l ot
• Developing and testing adequacy of research instruments
• Assessing the feasibility of a (full-scale) study/survey
• Designing a research protocol
• Assessing whether the research protocol is realistic and workable
• Establishing whether the sampling frame and technique are effective
• Assessing the likely success of proposed recruitment approaches
• Identifying logistical problems which might occur using proposed methods
• Estimating variability in outcomes to help determining sample size
• Collecting preliminary data
• Determining what resources (finance, staff) are needed for a planned study
• Assessing the proposed data analysis techniques to uncover potential problems
• Developing a research question and research plan
• Training a researcher in as many elements of the research process as possible
• Convincing funding bodies that the research team is competent and knowledgeable
• Convincing funding bodies that the main study is feasible and worth funding
• Convincing other stakeholders that the main study is worth supporting.
Pi l ot st udi es
• Pilot studies can be based
on quantitative and/or qualitative
• Large-scale studies might employ a number of pilot
studies before the main survey is conducted. One
1. Qualitative data collection and analysis on a
relatively unexplored topic,
2. Use the results to design a subsequent
quantitative phase of the study.
• The NIEHS
• Evidence to be used to
plan and prioritise the
effective delivery of eye
care for Indigenous
• Assess the prevalence
and main causes of
• Assess the utilisation of
eye care services,
barriers to health and
the impact of vision
Pi l ot f or The NI EHS
Page 13 of National Indigenous Eye Health Survey- 1.4.1 The pilot study
A pilot study titled ‘Evaluation of Selected Vision and Eye Conditions in Aboriginal and Torres Strait
Islander Communities’, was conducted in Moree NSW.
The purpose of this study was to develop the procedures and conduct the
preliminary pilot studies for the National Indigenous Eye Health Survey.
This was to ensure that the processes and protocols were both valid and
culturally appropriate for Aboriginal Australian people.
The ethics for the study had been approved by the Aboriginal Health and Medical Research Council
of NSW. The rapid assessment methodology utilised was based on the Vision Initiative in Victoria,
and was compared to a gold standard eye examination by an eye care practitioner. The rapid
examination was designed to detect the five predominant and significant eye conditions, including
diabetic retinopathy, trachoma, glaucoma, refractive error and cataract. It was found that 90.4% of the
retinal images obtained with the non-mydriatic camera were gradable, and that the rapid examination
was able to detect proliferative diabetic retinopathy and macula oedema with very high sensitivity and
specificity. The study found that the rapid examination was highly specific in correctly identifying
normal vision in adults. The rapid assessment also showed high sensitivity in detecting cup to disc
ratios greater than 0.6. The rapid assessment method used in this study resulted in a validated, rapid
examination methodology which was able to detect diabetic retinopathy, risk of glaucoma, trachoma,
refractive error and visual impairment in Aboriginal and Torres Strait Islander populations in Australia,
when compared to a gold standard eye care practitioner examination, with minimal staff training and
Bur net t , A. Adapt at i on & val i dat i on
of a vi si on r el at ed qual i t y- of - l i f e
t ool f or use i n i ndi genous
communi t i es
Link to PhD thesis:
The st eps used t o pi l ot a
quest i onnai r e
Pilot study procedures to improve the internal validity of a questionnaire
•Administer the questionnaire to pilot subjects in exactly the same way as it will
be administered in the main study.
•Ask the subjects for feedback to identify ambiguities and difficult questions.
•Record the time taken to complete the questionnaire and decide whether it is
•Discard all unnecessary, difficult or ambiguous questions.
•Assess whether each question gives an adequate range of responses.
•Establish that replies can be interpreted in terms of the information that is
•Check that all questions are answered.
•Re-word or re-scale any questions that are not answered as expected.
•Shorten, revise and, if possible, pilot again.
(Source: Table 3.23 in Peat et al. 2002: 123)
Li mi t at i ons of pi l ot t est i ng
• Making inaccurate predictions or assumptions
on the basis of pilot data.
• Problems arising from contamination
– Pilot data included in main study
– Same participants re-tested
• Problems related to funding.
Prescott and Soeken, 1989 p60
Why ar e pi l ot st udi es not r epor t ed?
• Publication bias - only papers that have
statistically significant results and not to
report non-significant effects
• Selective publication of research results
• overestimation of the effectiveness of
interventions = exposing patients to
useless or harmful treatments.
• overestimation of adverse effects =
patients are denied effective forms of
• It is important to share lessons learned
otherwise patients may be subjected to
poorly developed tools or money may be
wasted because methods of recruitment
The i mpor t ance of r epor t i ng t he Pi l ot
• When reported, they often only justify the research methods or particular
research tool used.
• Rarely reporting offers little about exactly what was learnt.
• Therefore investigators should be encouraged to report their pilot studies,
and in particular to report in more detail the actual improvements made
to the study design and the research process.
Ref er ences
Edwin R. van Teijlingen and Vanora Hundley. The importance of pilot studies, Issue 35 Winter 2001
Social Research Update is published quarterly by the Department of Sociology, University of Surrey,
Guildford GU7 5XH, England. ht t p: / / sr u. soc. sur r ey. ac. uk/ SRU35. ht ml
Gillian A. Lancaster, Susanna Dodd, and Paula R. Williamson.
Design and analysis of pilot studies: recommendations for good practice
Journal of Evaluation in Clinical Practice, 10, 2, 307–312
Indigenous eye survey
National Indigenous Eye Health Survey Team. National Indigenous Eye Health Survey: Minum
Barreng (Tracking Eyes) Full Report. Indigenous Eye Health Unit, Melbourne 2009. ISBN
Anthea Burnett PhD thesis (2009)
Vision impairment in Aboriginal and Torres Strait Islander peoples: a toolkit to assess prevalence and
Homewor k- r ead t hi s paper