Pilot study, reliability test(New)2

1、Pilot study & reliability test 2013.04 Jue Wang 1.Pilot experiment nA pilot experiment, also called a pilot study, is a small scale preliminary study conducted in order to evaluate feasibility, time, cost, adverse events, and effect size in an attempt to predict an appropriate sample size and improv

2、e upon the study design prior to performance of a full-scale research project. Implementation of pilot studies nPilot experiments are frequently carried out before large- scale quantitative research, in an attempt to avoid time and money being wasted on an inadequately designed project. A pilot stud

3、y is usually carried out on members of the relevant population, but not on those who will form part of the final sample. This is because it may influence the later behavior of research subjects if they have already been involved in the research. nA pilot study is often used to test the design of the

4、 full- scale experiment which then can be adjusted. It is a potentially valuable insight and should anything be missing in the pilot study it can be added to the full-scale experiment to improve the chances of a clear outcome. Applications nOften in engineering applications, pilot experiments are us

5、ed to sell a product and provide quantitative proof that the system has potential to succeed on a full scale basis. Pilot experiments are also used to reduce cost, as they are less expensive than full experiments. If there is not enough reason to provide full scale applications, pilot studies can ge

6、nerally provide this proof. Applications nIn sociology, pilot studies can be referred to as small-scale studies that will help identify design issues before the main research is done. 2. Reliability nIn the psychometrics, reliability is used to describe the overall consistency of a measure. A measur

7、e is said to have a high reliability if it produces similar results under consistent conditions. For example, measurements of peoples height and weight are often extremely reliable. Reliability nWill people answer the same question in the same way on different occasion? Difference from validity nRel

8、iability does not imply validity. That is, a reliable measure that is measuring something consistently, may not be measuring what you want to be measuring. Difference from validity nWhile reliability does not imply validity, a lack of reliability does place a limit on the overall validity of a test.

9、 A test that is not perfectly reliable cannot be perfectly valid. While a reliable test may provide useful valid information, a test that is not reliable cannot possibly be valid. Difference from validity nAn example often used to illustrate the difference between reliability and validity in the exp

10、erimental sciences involves a common bathroom scale. If someone who is 200 pounds steps on a scale 10 times and gets readings of 15, 250, 95, 140, etc., the scale is not reliable. If the scale consistently reads 150, then it is reliable, but not valid. If it reads 200 each time, then the measurement

11、 is both reliable and valid. General model nIn practice, testing measures are never perfectly consistent. Theories of test reliability have been developed to estimate the effects of inconsistency on the accuracy of measurement. The basic starting point for almost all theories of test reliability is

12、the idea that test scores reflect the influence of two sorts of factors: 1. Factors that contribute to consistency: stable characteristics of the individual or the attribute that one is trying to measure 2. Factors that contribute to inconsistency: features of the individual or the situation that ca

13、n affect test scores but have nothing to do with the attribute being measured General model nSome of these inconsistencies include: Temporary but general characteristics of the individual: health, fatigue, motivation, emotional strain Temporary and specific characteristics of individual: comprehensi

14、on of the specific test task, specific tricks or techniques of dealing with the particular test materials, fluctuations of memory, attention or accuracy Aspects of the testing situation: freedom from distractions, clarity of instructions, interaction of personality, sex, or race of examiner Chance f

15、actors: luck in selection of answers by sheer guessing, momentary distractions nThe goal of estimating reliability is to determine how much of the variability in test scores is due to errors in measurement and how much is due to variability in true scores. True score nA true score is the replicable

16、feature of the concept being measured. It is the part of the observed score that would recur across different measurement occasions in the absence of error. nErrors of measurement are composed of both random error and systematic error. It represents the discrepancies between scores obtained on tests

17、 and the corresponding true scores. nThis conceptual breakdown is typically represented by the simple equation: Classical test theory nReliability theory shows that the variance of obtained scores is simply the sum of the variance of true scores plus the variance of errors of measurement. nThis equa

18、tion suggests that test scores vary as the result of two factors: 1. Variability in true scores 2. Variability due to errors of measurement. 222 XTE Classical test theory nThe reliability coefficient provides an index of the relative influence of true and error scores on attained test scores. In its

19、 general form, the reliability coefficient is defined as the ratio of true score variance to the total variance of test scores. Or, equivalently, one minus the ratio of the variation of the error score and the variation of the observed score: nUnfortunately, there is no way to directly observe or ca

20、lculate the true score, so a variety of methods are used to estimate the reliability of a test. 22 22 1 TE xx XX Estimation nThe goal of estimating reliability is to determine how much of the variability in test scores is due to errors in measurement and how much is due to variability in true scores

21、. nFour practical strategies have been developed that provide workable methods of estimating test reliability. Test-retest reliability method Parallel-forms method Split-half method Internal consistency Cronbachs alpha nIn statistics, Cronbachs (alpha) is a coefficient of reliability. It is commonly used as a measure of the internal consistency or reliability of a psychometric test score for a sample of examinees. It was first named alpha by Lee Cronbach in 1951, as he had intended to continue with further coefficients. nCronbachs alpha statistic is wid