Circular inference in neuroscince – the dangers of ：在神經(jīng)科學(xué)–危險(xiǎn)的循環(huán)推理

1、circular analysis in systems neuroscience with particular attention to cross-subject correlation mappingnikolaus kriegeskortelaboratory of brain and cognition, national institute of mental health chris i baker w kyle simmons patrick sf bellgowan peter bandettinicollaboratorspart 1general introductio

2、n to circular analysis in systems neuroscience(synopsis of kriegeskorte et al. 2009)part 2specific issue: selection bias incross-subject correlation mapping(following up on vul et al. 2009)overviewdataresultsanalysisdataresultsdataresultsanalysisdataresultsanalysisassumptionsdataresultsanalysisassum

3、ptionscircular inferencedataresultsanalysisassumptionscircular inferencedataresultsanalysisassumptionshow do assumptions tinge results?elimination(binary selection)weighting(continuous selection)sorting(multiclass selection) through variants of selection!dataresultsanalysisassumptions:selection crit

4、eriaelimination(binary selection)example 1pattern-information analysisexperimental design“animate?”“pleasant?”stimulus(object category)task(property judgment)simmons et al. 2006 define roi by selecting ventral-temporal voxels for which any pairwise condition contrast is significant at p.001 (uncorr.

5、) perform nearest-neighbor classificationbased on activity-pattern correlation use odd runs for trainingand even runs for testingpattern-information analysis00.51decoding accuracytask (judged property)stimulus (object category)resultschance levelfmri datausing all datato select roi voxels using only

6、training datato select roi voxelsdata from gaussianrandom generator00.5100.5100.5100.51decoding accuracychance leveltaskstimulus.but we used cleanly independenttraining and test data!? !conclusion for pattern-information analysisthe test data must not be used in either. training a classifier or defi

7、ning the roicontinuous weightingbinary weightingdata selection is key to many conventional analyses.can it entail similar biases in other contexts?example 2regional activation analysisroi definition is affected by noisetrue regionoverfitted roiroi-averageactivationoverestimated effectindependent roi

8、data sortingdataresultsanalysisassumptions:sorting criteriaset-average tuning curvesstimulus parameter (e.g. orientation)response.for data sorted by tuningnoise dataroi-averagefmri responseabcdconditionset-average activation profiles.for data sorted by activationnoise datato avoid selection bias, we

9、 can.perform a nonselective analysisor.make sure that selection and results statistics are independent under the null hypothesis, because they are either: inherently independent or computed on independent datae.g. independent contrastse.g. whole-brain mapping(no roi analysis)does selection by an ort

10、hogonal contrast vector ensure unbiased analysis?roi-definition contrast: a+broi-average analysis contrast: a-bcselection=1 1tctest=1 -1torthogonal contrast vectors does selection by an orthogonal contrast vector ensure unbiased analysis?not sufficientcontrastvectorthe design and noise dependencies

11、matter.designnoise dependencies no, there can still be bias.still not sufficientcircular analysisproshighly sensitivewidely accepted (examples in all high-impact journals)doesnt require independent data setsgrants scientists independencefrom the dataallows smooth blending of blind faith and empirici

12、smconscircular analysisproshighly sensitivewidely accepted (examples in all high-impact journals)doesnt require independent data setsgrants scientists independencefrom the dataallows smooth blending of blind faith and empiricismconscircular analysisproshighly sensitivewidely accepted (examples in al

13、l high-impact journals)doesnt require independent data setsgrants scientists independencefrom the dataallows smooth blending of blind faith and empiricismconscant think of any right nowprosthe error that beautifies resultsconfirms even incorrect hypothesesimproves chances ofhigh-impact publicationpa

14、rt 2specific issue: selection bias incross-subject correlation mapping(following up on vul et al. 2009)motivationvul et al. (2009) posed a puzzle:why are the cross-subject correlations found in brain mapping so high?selection bias is one piece of the puzzle.but there are more pieces and we have yet

15、to put them all together.overview list and discuss six pieces of the puzzle.(they dont all point in the same direction!) suggest some guidelines for good practice.six pieces synopsis1.cross-subject correlation estimates are very noisy. 2.bin or within-subject averaging legitimately increases correla

16、tions.3.selecting among noisy estimates yields large biases.4.false-positive regions are highly likely for a whole-brain mapping thresholded at p.001, uncorrected.5.reported correlations are high, but not highly significant.6.studies have low power for finding realistic correlations in the brain if

17、multiple testing is appropriately accounted for.vul et al. 2009,populationthe geometric mean of the reliability is an upper boundon the population correlation.the reliabilities provide no boundon the sample correlation.noise-freecorrelationsample correlationsacross small numbers of subjectsare very

18、noisy estimatesof population correlations.piece 10.65correlation10 subjects95%-confidenceintervalcross-subject correlation estimatesare very noisycross-subject correlation estimatesare very noisythe more we average(reducing noise but not signal),the higher correlations become.piece 2bin-averaging in

19、flates correlations bin-averaging inflates correlations subjects are like bins.for each subject, all data is averaged to give one number. take-home messagecross-subject correlation estimates are expected to be. high (averaging all data for each subject) noisy (low number of subjects)so whats ed fuss

20、ing about?we dont need selection bias to explain the high correlations, right?selecting the maximumamong noisy estimatesyields large selection biases.piece 3expected maximum correlationselected among null regionsexpected maximum correlation16 subjectsbiasfalse-positive regions are likely to be found

21、 in whole-brain mappingusing p.001, uncorrected.piece 4mapping with p.001, uncorrectedglobal null hypothesis is true(population correlation = 0 in all brain locations)reported correlations are high,but not highly significant.piece 5reported correlations are high,but not highly significantp0.00001p0.

22、001 p0.01p0.05one-sidedtwo-sidedcorrelation thresholds as a functionof the number of subjectsreported correlations are high,but not highly significantp0.00001p0.001 p0.01p0.05one-sidedtwo-sidedcorrelation thresholds as a functionof the number of subjectsreported correlations are high,but not highly

23、significantp0.00001p0.001 p0.01p0.05one-sidedtwo-sidedcorrelation thresholds as a functionof the number of subjects(assuming each study reportsthe maximum of 500independent brain locations)what correlations would we expectunder the global null hypothesis?reported correlations are high,but not highly

24、 significantp0.00001p0.001 p0.01p0.05one-sidedtwo-sided(assuming each study reports the max.of 500 independent brain locations)what correlations would we expectunder the global null hypothesis?most of the studies have low powerfor finding realistic correlationswith whole-brain mappingif multiple tes

25、ting is appropriately accounted for. piece 6see also: yarkoni 2009numbers of subjectsin studies reviewed by vul et al. (2009)number of correlations estimatesnumber of subjects48163660100powerin order to find a single region with across-subject correlation of 0.7 in the brain.we would needabout 36 subjects16 subjectspowerin order to find a single region with across-subject correlation of 0.7 in the brain.we would needabout 36 subjects16 subjectstake-home messagewhole-brain cross-subject correlation mappingwith 16 subjectsdoes not work.need at least twic