內(nèi)容天網(wǎng)項(xiàng)目skynet courier detection via machine learning

1、TOP SECRET/COMINT/REL TO USA, FVEYGiven a handful of courier selectors, can we find othersthat “behave similarly” by analyzingmetadata?Its worth noting that:we are looking for different people using phones in similar wayswithout using any call chaining techniques from known selectorsby scanning thro

2、ugh all selectors seen in Pakistan that have not left Af/Pak (55M)TOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYThis presentation describes our search for AQSL couriers using behavioral profilingBehavioral Feature ExtractionCross Validation Experiment on AQSL CouriersPreliminar

3、y SIGINT FindingsTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYCounting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectorsTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYBy examining multiple features at once, we can see s

4、ome indicative behaviors of our courier selectorsTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYNow, well describe a cross validation experiment on the AQSL selectors that we were providedCross Validation Experiment on AQSL CouriersTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/CO

5、MINT/REL TO USA, FVEYOur initial detector uses the centroid of the AQSL couriers to “find other selectors like these”AQSL Cross-Validation Experiment7 MSISDN/IMSI pairsHold each pair out and score them when training the centroid on the restAssume that random draws of Pakistani selectors are nontarge

6、tsHow well do we do?TOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYOur initial detector uses the centroid of the AQSL couriers to “find other selectors like these”AQSL Cross-Validat ExperimentInitial experiments showed EER in 10-20% rangeHere, performance much worse again these

7、nontargets:Seen in PakistanNot seen outside Af/PakNot FVEY selectoTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYStatistical algorithms are able to find the couriers at very low false alarm rates, if were allowed to miss half of themRandom Forest Classifierbetter7 MSISDN/IMSI pa

8、irsHold each pair out and then try to find them af learning how to disting remaining couriers fro other Pakistanis(using 100k random selectors here)Assume that random draws of Pakistani selectors are nontarge0.18% False Alarm Ra 50% Miss RateTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO

9、 USA, FVEYWeve been experimenting with several error metrics on both small and large test setsRandom Forest:0.18% false alarm rate at 50% miss rate7x improvement over random performance when evaluating its tasked precision at 100TOP SECRET/COMINT/REL TO USA, FVEYTraining DataClassifierFeatures100k T

10、est Selectors55M Test SelectorsFalse Alarm Rate at 50% Miss RateMean Reciprocal RankTasked Selectors in Top 500Tasked Selectors in Top 100NoneRandomNone50%1/23k (simulated)0.64(active/Pak)0.13(active/Pak)Known CouriersCentroidAll20%1/18kOutgoing43%1/27kRandom Forest0.18%1/9.951+ Anchory SelectorsTOP

11、 SECRET/COMINT/REL TO USA, FVEYTo get more training data we scraped selectors from S2I11 Anchory reports containing keyword “courier”Anchory SelectorsSearched for reports containing “S2I11” AND “courier”Filtered out non-mobile numbers and kept selectors with “interesting” travel patterns seen in Sma

12、rtTrackerTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYAdding selectors from Anchory reports to the training data reduced the false alarm rates even furtherAnchory SelectorsSearched for reports containing “S2I11” AND “courier”Filtered out non-mob numbers and kept selectors with

13、 “interesting” travel patterns seen in SmartTrackerTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYWeve been experimenting with several error metrics on both small and large test setsRandom Forest trained on Known Couriers + Anchory Selectors:0.008% false alarm rate at 50% miss r

14、ate46x improvement over random performance when evaluating its tasked precision at 100TOP SECRET/COMINT/REL TO USA, FVEYTraining DataClassifierFeatures100k Test Selectors55M Test SelectorsFalse Alarm Rate at 50% Miss RateMean Reciprocal RankTasked Selectors in Top 500Tasked Selectors in Top 100NoneR

15、andomNone50%1/23k (simulated)0.64(active/Pak)0.13(active/Pak)Known CouriersCentroidAll20%1/18kOutgoing43%1/27kRandom Forest0.18%1/9.951+ Anchory Selectors0.008%1/14216TOP SECRET/COMINT/REL TO USA, FVEYNow, well investigate some findings after running these classifiers on +55M Pakistani selectors via

16、 MapReducePreliminary SIGINT FindingsTOP SECRET/COMINT/REL TO USA, FVEYTOP SECRET/COMINT/REL TO USA, FVEYPreliminary results indicate that were on the right track, but much remainsCross Validation Experiment:Random Forest classifier operating at 0.18% false alarm rate at 50% missEnhancing training data with Anchory selectors reduced that to 0.008%Mean Reciprocal Rank is 1/10Prelim