Pr e t est i ng & anal ysi ng
pi l ot dat aDr Ki r st en Chal l i nor
COMMONWEALTH OF AUSTRALIA
Copyright Regulations 1969
WARNING
This material has been reproduced and communicated to you by ...
Pi l ot st udi es
• What is a pilot study?
 Mini version of a full-scale study = 'feasibility' study
 Specific pre-testi...
"Do not take the risk.
Pilot test first.”
De Vaus (1993: 54)
Reasons t o pi l ot
• Developing and testing adequacy of research instruments
• Assessing the feasibility of a (full-scale...
Pi l ot st udi es
• Pilot studies can be based
on quantitative and/or qualitative
methods.
• Large-scale studies might emp...
• The NIEHS
• Evidence to be used to
plan and prioritise the
effective delivery of eye
care for Indigenous
Australians.
• ...
Pi l ot f or The NI EHS
Page 13 of National Indigenous Eye Health Survey- 1.4.1 The pilot study
A pilot study titled ‘Eval...
Bur net t , A. Adapt at i on & val i dat i on
of a vi si on r el at ed qual i t y- of - l i f e
t ool f or use i n i ndi g...
The st eps used t o pi l ot a
quest i onnai r e
Pilot study procedures to improve the internal validity of a questionnaire...
Li mi t at i ons of pi l ot t est i ng
• Making inaccurate predictions or assumptions
on the basis of pilot data.
• Proble...
Pilot studies
“under-discussed,
underused &
underreported”
Prescott and Soeken, 1989 p60
Why ar e pi l ot st udi es not r epor t ed?
• Publication bias - only papers that have
statistically significant results a...
The i mpor t ance of r epor t i ng t he Pi l ot
st udy
• When reported, they often only justify the research methods or pa...
Ref er ences
Pilot studies
Edwin R. van Teijlingen and Vanora Hundley. The importance of pilot studies, Issue 35 Winter 20...
Homewor k- r ead t hi s paper
of 16

Pretesting and Pilot data

This lesson looks at pilot studies and why they are crucial to a good study design.
Published on: Mar 4, 2016
Published in: Education      
Source: www.slideshare.net


Transcripts - Pretesting and Pilot data

  • 1. Pr e t est i ng & anal ysi ng pi l ot dat aDr Ki r st en Chal l i nor
  • 2. COMMONWEALTH OF AUSTRALIA Copyright Regulations 1969 WARNING This material has been reproduced and communicated to you by or on behalf of the University of New South Wales pursuant to Part VB of the Copyright Act 1968 (the Act). The material in this communication may be subject to copyright under the Act. Any further reproduction or communication of this material by you may be the subject of copyright protection under the Act. Do not remove this notice.
  • 3. Pi l ot st udi es • What is a pilot study?  Mini version of a full-scale study = 'feasibility' study  Specific pre-testing of a particular research instrument such as a questionnaire or interview schedule. • Pilot studies are a crucial element of a good study design. • Conducting a pilot study does not guarantee success in the main study, but it does increase the likelihood. Image from http://www.eh.uc.edu/cares/learn/pilot_study.html
  • 4. "Do not take the risk. Pilot test first.” De Vaus (1993: 54)
  • 5. Reasons t o pi l ot • Developing and testing adequacy of research instruments • Assessing the feasibility of a (full-scale) study/survey • Designing a research protocol • Assessing whether the research protocol is realistic and workable • Establishing whether the sampling frame and technique are effective • Assessing the likely success of proposed recruitment approaches • Identifying logistical problems which might occur using proposed methods • Estimating variability in outcomes to help determining sample size • Collecting preliminary data • Determining what resources (finance, staff) are needed for a planned study • Assessing the proposed data analysis techniques to uncover potential problems • Developing a research question and research plan • Training a researcher in as many elements of the research process as possible • Convincing funding bodies that the research team is competent and knowledgeable • Convincing funding bodies that the main study is feasible and worth funding • Convincing other stakeholders that the main study is worth supporting.
  • 6. Pi l ot st udi es • Pilot studies can be based on quantitative and/or qualitative methods. • Large-scale studies might employ a number of pilot studies before the main survey is conducted. One might: 1. Qualitative data collection and analysis on a relatively unexplored topic, 2. Use the results to design a subsequent quantitative phase of the study. Image: http://collaborationforgood.org/tag/pilot/
  • 7. • The NIEHS • Evidence to be used to plan and prioritise the effective delivery of eye care for Indigenous Australians. • Assess the prevalence and main causes of vision impairment. • Assess the utilisation of eye care services, barriers to health and the impact of vision impairment on Indigenous people.
  • 8. Pi l ot f or The NI EHS Page 13 of National Indigenous Eye Health Survey- 1.4.1 The pilot study A pilot study titled ‘Evaluation of Selected Vision and Eye Conditions in Aboriginal and Torres Strait Islander Communities’, was conducted in Moree NSW. The purpose of this study was to develop the procedures and conduct the preliminary pilot studies for the National Indigenous Eye Health Survey. This was to ensure that the processes and protocols were both valid and culturally appropriate for Aboriginal Australian people. The ethics for the study had been approved by the Aboriginal Health and Medical Research Council of NSW. The rapid assessment methodology utilised was based on the Vision Initiative in Victoria, and was compared to a gold standard eye examination by an eye care practitioner. The rapid examination was designed to detect the five predominant and significant eye conditions, including diabetic retinopathy, trachoma, glaucoma, refractive error and cataract. It was found that 90.4% of the retinal images obtained with the non-mydriatic camera were gradable, and that the rapid examination was able to detect proliferative diabetic retinopathy and macula oedema with very high sensitivity and specificity. The study found that the rapid examination was highly specific in correctly identifying normal vision in adults. The rapid assessment also showed high sensitivity in detecting cup to disc ratios greater than 0.6. The rapid assessment method used in this study resulted in a validated, rapid examination methodology which was able to detect diabetic retinopathy, risk of glaucoma, trachoma, refractive error and visual impairment in Aboriginal and Torres Strait Islander populations in Australia, when compared to a gold standard eye care practitioner examination, with minimal staff training and equipment (12).
  • 9. Bur net t , A. Adapt at i on & val i dat i on of a vi si on r el at ed qual i t y- of - l i f e t ool f or use i n i ndi genous communi t i es Link to PhD thesis: http://www.unsworks.unsw.edu.au/primo_library/libweb/action/dlDisplay. do?
  • 10. The st eps used t o pi l ot a quest i onnai r e Pilot study procedures to improve the internal validity of a questionnaire •Administer the questionnaire to pilot subjects in exactly the same way as it will be administered in the main study. •Ask the subjects for feedback to identify ambiguities and difficult questions. •Record the time taken to complete the questionnaire and decide whether it is reasonable. •Discard all unnecessary, difficult or ambiguous questions. •Assess whether each question gives an adequate range of responses. •Establish that replies can be interpreted in terms of the information that is required. •Check that all questions are answered. •Re-word or re-scale any questions that are not answered as expected. •Shorten, revise and, if possible, pilot again. (Source: Table 3.23 in Peat et al. 2002: 123)
  • 11. Li mi t at i ons of pi l ot t est i ng • Making inaccurate predictions or assumptions on the basis of pilot data. • Problems arising from contamination – Pilot data included in main study – Same participants re-tested • Problems related to funding. Image: http://www.saynotocrack.com/index.php/2007/01/
  • 12. Pilot studies “under-discussed, underused & underreported” Prescott and Soeken, 1989 p60
  • 13. Why ar e pi l ot st udi es not r epor t ed? • Publication bias - only papers that have statistically significant results and not to report non-significant effects • Selective publication of research results • overestimation of the effectiveness of interventions = exposing patients to useless or harmful treatments. • overestimation of adverse effects = patients are denied effective forms of care. • It is important to share lessons learned otherwise patients may be subjected to poorly developed tools or money may be wasted because methods of recruitment failed. Image: http://socialinfluenceconsultinggroup.com.au/the-pilot-was-an-influence-bungler/
  • 14. The i mpor t ance of r epor t i ng t he Pi l ot st udy • When reported, they often only justify the research methods or particular research tool used. • Rarely reporting offers little about exactly what was learnt. • Therefore investigators should be encouraged to report their pilot studies, and in particular to report in more detail the actual improvements made to the study design and the research process. Image: http://www.mbaskool.com/business-articles/marketing/759-pilot-project-launch-evaluating-the-market.html
  • 15. Ref er ences Pilot studies Edwin R. van Teijlingen and Vanora Hundley. The importance of pilot studies, Issue 35 Winter 2001 Social Research Update is published quarterly by the Department of Sociology, University of Surrey, Guildford GU7 5XH, England. ht t p: / / sr u. soc. sur r ey. ac. uk/ SRU35. ht ml Gillian A. Lancaster, Susanna Dodd, and Paula R. Williamson. Design and analysis of pilot studies: recommendations for good practice Journal of Evaluation in Clinical Practice, 10, 2, 307–312 Indigenous eye survey National Indigenous Eye Health Survey Team. National Indigenous Eye Health Survey: Minum Barreng (Tracking Eyes) Full Report. Indigenous Eye Health Unit, Melbourne 2009. ISBN 9780734041142 Anthea Burnett PhD thesis (2009) Vision impairment in Aboriginal and Torres Strait Islander peoples: a toolkit to assess prevalence and impact. http://www.unsworks.unsw.edu.au/primo_library/libweb/action/dlDisplay.do? vid=UNSWORKS&docId=unsworks_8655&fromSitemap=1&afterPDS=true
  • 16. Homewor k- r ead t hi s paper