開(kāi)源網(wǎng)站流量統(tǒng)計(jì)系統(tǒng)Piwik源碼分析-參數(shù)統(tǒng)計(jì)（一）

1、 Piwik現(xiàn)已改名為，這是套國(guó)外著名的開(kāi)源站統(tǒng)計(jì)系統(tǒng)，類似于百度統(tǒng)計(jì)、Google Analytics等系統(tǒng)。最的區(qū)別就是可以看到其中的源碼，這正合我意。因?yàn)槲抑睂?duì)統(tǒng)計(jì)的系統(tǒng)很好奇，很想知道的運(yùn)原理是怎么樣的，碰巧了解到有這么個(gè)系統(tǒng)，因此馬上嘗試了下。國(guó)內(nèi)關(guān)于該系統(tǒng)的相關(guān)資料較匱乏，多是分享怎么安裝的，并沒(méi)有找到有關(guān)源碼分析的章。下先對(duì)其做個(gè)初步的分析，后會(huì)越來(lái)越詳細(xì)，本前的職位是前端，因此會(huì)先分析腳本代碼，后再分析后臺(tái)代碼。、整體概況Piwik的官是，使PHP編寫(xiě)的，我以前就是PHP程師，因此看代碼不會(huì)有障礙。前最新版本是3.6，Github地址是，打開(kāi)地址將會(huì)看到下圖中的內(nèi)容（只截取了關(guān)

2、鍵部分）。打開(kāi)js件夾，的piwik.js就是本次要分析的腳本代碼（如下圖紅框出部分），內(nèi)容較多，有7838代碼。先把系統(tǒng)的代碼都下載下來(lái)，然后在本地配置虛擬錄，再開(kāi)始安裝。在安裝的時(shí)候可以選擇語(yǔ)，該系統(tǒng)持簡(jiǎn)體中（注意下圖中紅框出的部分）。系統(tǒng)會(huì)執(zhí)些操作（注意看下圖左邊部分），包括檢查當(dāng)前環(huán)境能否安裝、建數(shù)據(jù)庫(kù)等，按照提步步來(lái)就，較簡(jiǎn)單，沒(méi)啥難度。安裝完后就會(huì)動(dòng)跳轉(zhuǎn)到后臺(tái)界（如下圖所），有圖表，有分析，和常的統(tǒng)計(jì)系統(tǒng)差不多。功能還沒(méi)細(xì)看，只做了初步的了解，界的友好度還是蠻不錯(cuò)的。嵌到頁(yè)中的JavaScript代碼與其它統(tǒng)計(jì)系統(tǒng)也類似，如下所，也是異步加載的式，只是發(fā)送的請(qǐng)求地址沒(méi)有偽裝成圖像地

3、址（注意看標(biāo)紅的那句代碼）。/* tracker methods like setCustomDimension should be called before trackPageView */varu=/;/定義_paq.push(setTrackerUrl, u+piwik.php);_paq.push(setSiteId, 1);)();在頁(yè)中嵌這段腳本后，頁(yè)在刷新的時(shí)候，會(huì)有下圖中的請(qǐng)求。在請(qǐng)求中帶了堆的參數(shù)，在后的內(nèi)容中會(huì)對(duì)每個(gè)參數(shù)做釋義。7000多的腳本，當(dāng)然不能的讀，需要先拆分，拆成個(gè)個(gè)的模塊，然后再逐個(gè)分析。腳本之所以這么，是因?yàn)榫帉?xiě)了量代碼來(lái)兼容各個(gè)版本的瀏覽器，這其中甚包括

4、IE4、Firefox1.0、Netscape等灰級(jí)的瀏覽器。接下來(lái)我把源碼拆分成6個(gè)部分，分別是json、private、query、content-overlay、和piwik，如下圖紅線框出的所，piwik-all中包含了全部代碼，便于對(duì)。代碼已上傳到。json.js是個(gè)開(kāi)源插件，為了兼容不持JSON對(duì)象的瀏覽器設(shè)計(jì)的，這的代碼可以單獨(dú)研究。private.js包含了些于全局的私有變量和私有函數(shù)，例如定義系統(tǒng)對(duì)象的別名、判斷類型等。query.js中包含了很多操作HTML元素的法，例如設(shè)置元素屬性、查詢某個(gè)CSS類的元素等，它類似于個(gè)微型的jQuery庫(kù)，不過(guò)有許多獨(dú)特的功能。conte

5、nt-overlay.js有兩部分組成，部分包含內(nèi)容追蹤以及URL拼接等功能，另部分是來(lái)處理嵌套的頁(yè)，這具體沒(méi)有細(xì)看。tracker.js中只有個(gè)Tracker()函數(shù)，不過(guò)內(nèi)容最多，有4700多，主要的統(tǒng)計(jì)邏輯都在這了。piwik.js中內(nèi)容不多，包含些初始化和插件的鉤等功能，鉤具體怎么運(yùn)作的還沒(méi)細(xì)看。雖然分成了6部分，但是各部分的內(nèi)容還是蠻多的，并且內(nèi)容之間是有聯(lián)系的，因此短時(shí)間的話，很難搞清楚其中所有的門道。我就挑了點(diǎn)我個(gè)感覺(jué)最重要的先做分析。1）3種傳送數(shù)據(jù)的式我原先只知道兩種傳送數(shù)據(jù)的式，種是通過(guò)Ajax的式，另種是創(chuàng)建個(gè)Image對(duì)象，然后為其定義src屬性，數(shù)據(jù)作為URL的參數(shù)傳

6、遞給后臺(tái)，這種式很通，并且還能完美解決跨域問(wèn)題。我以前編寫(xiě)的個(gè)性能參數(shù)搜集的插件，也是這么傳送數(shù)據(jù)的。在閱讀源碼的時(shí)候，發(fā)現(xiàn)了第三種傳送數(shù)據(jù)的式，使Navigator對(duì)象的。MDN上說(shuō)：此法可于通過(guò)HTTP將少量數(shù)據(jù)異步傳輸?shù)絎eb服務(wù)器。雖然這個(gè)法有兼容問(wèn)題，但我還是被震撼到了。它很適合統(tǒng)計(jì)的場(chǎng)景，MDN上講到：統(tǒng)計(jì)代碼會(huì)在頁(yè)關(guān)閉（）之前向web服務(wù)器發(fā)送數(shù)據(jù)，但過(guò)早的發(fā)送數(shù)據(jù)可能錯(cuò)過(guò)收集數(shù)據(jù)的機(jī)會(huì)。然，要保證在頁(yè)關(guān)閉期間發(fā)送數(shù)據(jù)直較困難，因?yàn)闉g覽器通常會(huì)忽略在卸載事件中產(chǎn)的異步請(qǐng)求。在使sendBeacon()法后，能使瀏覽器在有機(jī)會(huì)時(shí)異步地向服務(wù)器發(fā)送數(shù)據(jù)，同時(shí)不會(huì)延遲頁(yè)的卸載或影響下頁(yè)

7、的載。這就解決了提交分析數(shù)據(jù)時(shí)的所有的問(wèn)題：使它可靠，異步并且不會(huì)影響下頁(yè)的加載，并且代碼更簡(jiǎn)單。下是代碼段（注意看標(biāo)紅的那句代碼），存在于tracker.js中。function sendPostRequestViaSendBeacon(request) varsupportsSendBeacon =object = typeof navigatorAlias &function = typeof navigatorAlias.sendBeacon &function = typeof Blob;if (!supportsSendBeacon) returnfalse;varheaders

8、= type: application/x-www-form-urlencoded; charset=UTF-8;varsuccess = false;try varblob = newBlob(request, headers);success = navigatorAlias.sendBeacon(configTrackerUrl, blob);/ returns true if the user agent is able to successfully queue the data for transfer,/ Otherwise it returns false and we nee

9、d to try the regular way catch (e) returnfalse;returnsuccess;2）參數(shù)釋義下的法（存在于tracker.js中）專門于搜集頁(yè)中的統(tǒng)計(jì)數(shù)據(jù)，將它們拼接成指定鏈接的參數(shù)，這條鏈接中的參數(shù)最終將會(huì)發(fā)送給服務(wù)器。/* Returns the URL to call piwik.php,* with the standard parameters (plugins, resolution, url, referrer, etc.).* Sends the pageview and browser settings with every requ

10、est in case of race conditions.*/function getRequest(request, customData, pluginMethod, currentEcommerceOrderTs) vari,now = newDate(),nowTs = Math.round(now.getTime() / 1000),referralTs,referralUrl,referralUrlMaxLength = 1024,currentReferrerHostName,originalReferrerHostName,customVariablesCopy = cus

11、tomVariables,cookieSessionName = getCookieName(ses),cookieReferrerName = getCookieName(ref),cookieCustomVariablesName = getCookieName(cvar),cookieSessionValue = getCookie(cookieSessionName),attributionCookie = loadReferrerAttributionCookie(),currentUrl = configCustomUrl | locationHrefAlias,campaignN

12、ameDetected,campaignKeywordDetected;if (configCookiesDisabled) deleteCookies();if (configDoNotTrack) return;varcookieVisitorIdValues = getValuesFromVisitorIdCookie();if (!isDefined(currentEcommerceOrderTs) currentEcommerceOrderTs = ;/ send charset if document charset is not utf-8. sometimes encoding

13、/ of urls will be the same as this and not utf-8, which will cause problems/ do not send charset if it is utf8 since its assumed by default in PiwikvarcharSet = documentAlias.characterSet | documentAlias.charset;if (!charSet | charSet.toLowerCase() = utf-8) charSet = null;campaignNameDetected = attr

14、ibutionCookie0;campaignKeywordDetected = attributionCookie1;referralTs = attributionCookie2;referralUrl = attributionCookie3;if (!cookieSessionValue) / cookie ses was not found: we consider this the start of a session/ here we make sure that if ses cookie is deleted few times within the visit/ and s

15、o this code path is triggered many times for one visit,/ we only increase visitCount once per Visit window (default 30min)varvisitDuration = configSessionCookieTimeout / 1000;if (!cookieVisitorIdValues.lastVisitTs |nowTs - cookieVisitorIdValues.lastVisitTs visitDuration) cookieVisitorIdValues.visitC

16、ount+;cookieVisitorIdValues.lastVisitTs = cookieVisitorIdValues.currentVisitTs;/ Detect the campaign information from the current URL/ Only if campaign wasnt previously set/ Orif it was set but we mustattribute to the mostrecent one/ Note: we are working on the currentUrl before purify() since we ca

17、n parse the campaign parameters in the hash tagif (!configConversionAttributionFirstReferrer |!campaignNameDetected.length) for(i in configCampaignNameParameters) if (Ototype.hasOwnProperty.call(configCampaignNameParameters, i) campaignNameDetected = getUrlParameter(currentUrl,configCampaignNamePara

18、metersi);if (campaignNameDetected.length) break;for(i in configCampaignKeywordParameters) if (Ototype.hasOwnProperty.call(configCampaignKeywordParameters,i) campaignKeywordDetected = getUrlParameter(currentUrl,configCampaignKeywordParametersi);if (campaignKeywordDetected.length) break;/ Store the re

19、ferrer URL and time in the cookie;/ referral URL depends on the first or last referrer attributioncurrentReferrerHostName = getHostName(configReferrerUrl);originalReferrerHostName = referralUrl.length? getHostName(referralUrl): ;if (currentReferrerHostName.length & / there is a referrer!isSiteHostNa

20、me(currentReferrerHostName) & / domain is not the current domain(!configConversionAttributionFirstReferrer | / attribute to last known referrer!originalReferrerHostName.length | / previously emptyisSiteHostName(originalReferrerHostName) / previously set but in current domainreferralUrl = configRefer

21、rerUrl;/ Set the referral cookie if we have either a Referrer URL,or detected a Campaign (or both)if (referralUrl.length | campaignNameDetected.length) referralTs = nowTs;attributionCookie = campaignNameDetected,campaignKeywordDetected,referralTs,purify(referralUrl.slice(0, referralUrlMaxLength);set

22、Cookie(cookieReferrerName,JSON_PIWIK.stringify(attributionCookie),configReferralCookieTimeout,configCookiePath,configCookieDomain);/ build out the rest of the requestrequest +=&idsite= +configTrackerSiteId +&rec=1 +&r= +String(Math.random().slice(2, 8) + / keep the string to a minimum&h= +now.getHou

23、rs() +&m= +now.getMinutes() +&s= +now.getSeconds() +&url= +encodeWrapper(purify(currentUrl) +(configReferrerUrl.length? &urlref= + encodeWrapper(purify(configReferrerUrl): ) +(configUserId & configUserId.length? &uid= + encodeWrapper(configUserId): ) +&_id= +cookieVisitorIdValues.uuid +&_idts= +cook

24、ieVisitorIdValues.createTs +&_idvc= +cookieVisitorIdValues.visitCount +&_idn= +cookieVisitorIdValues.newVisitor + / currently unused(campaignNameDetected.length? &_rcn= + encodeWrapper(campaignNameDetected): ) +(campaignKeywordDetected.length? &_rck= + encodeWrapper(campaignKeywordDetected): ) +&_re

25、fts= +referralTs +&_viewts= +cookieVisitorIdValues.lastVisitTs +(String(cookieVisitorIdValues.lastEcommerceOrderTs).length? &_ects= + cookieVisitorIdValues.lastEcommerceOrderTs: ) +(String(referralUrl).length? &_ref= +encodeWrapper(purify(referralUrl.slice(0, referralUrlMaxLength): ) +(charSet ? &cs

26、= + encodeWrapper(charSet) : ) +&send_image=0;/ browser featuresfor(i in browserFeatures) if (Ototype.hasOwnProperty.call(browserFeatures, i) request += & + i + = + browserFeaturesi;varcustomDimensionIdsAlreadyHandled = ;if (customData) for(i in customData) if (Ototype.hasOwnProperty.call(customData

27、, i) &/dimensiond+$/.test(i) varindex = i.replace(dimension, );customDimensionIdsAlreadyHandled.push(parseInt(index, 10);customDimensionIdsAlreadyHandled.push(String(index);request += & + i + = + customDatai;deletecustomDatai;if (customData & isObjectEmpty(customData) customData = null;/ we deleted

28、all keys from custom data/ custom dimensionsfor(i in customDimensions) if (Ototype.hasOwnProperty.call(customDimensions, i) varisNotSetYet =-1 = indexOfArray(customDimensionIdsAlreadyHandled, i);if (isNotSetYet) request += &dimension + i + = + customDimensionsi;/ custom dataif (customData) request +

29、= &data= + encodeWrapper(JSON_PIWIK.stringify(customData); else if (configCustomData) request += &data= + encodeWrapper(JSON_PIWIK.stringify(configCustomData);/ Custom Variables, scope pagefunction appendCustomVariablesToRequest(customVariables, parameterName) varcustomVariablesStringified = JSON_PI

30、WIK.stringify(customVariables);if (customVariablesStringified.length 2) return(& + parameterName + = + encodeWrapper(customVariablesStringified);return;varsortedCustomVarPage = sortObjectByKeys(customVariablesPage);varsortedCustomVarEvent = sortObjectByKeys(customVariablesEvent);request += appendCus

31、tomVariablesToRequest(sortedCustomVarPage, cvar);request += appendCustomVariablesToRequest(sortedCustomVarEvent, e_cvar);/ Custom Variables, scope visitif (customVariables) request += appendCustomVariablesToRequest(customVariables, _cvar);/ Dont save deleted custom variables in the cookiefor(i in cu

32、stomVariablesCopy) if (Ototype.hasOwnProperty.call(customVariablesCopy, i) if (customVariablesi0 = | customVariablesi1 = ) deletecustomVariablesi;if (configStoreCustomVariablesInCookie) setCookie(cookieCustomVariablesName,JSON_PIWIK.stringify(customVariables),configSessionCookieTimeout,configCookieP

33、ath,configCookieDomain);/ performance trackingif (configPerformanceTrackingEnabled) if (configPerformanceGenerationTime) request += >_ms= + configPerformanceGenerationTime; else if (performanceAlias &performanceAlias.timing &performanceAlias.timing.requestStart &performanceAlias.timing.responseEnd

34、) request +=>_ms= +(performanceAlias.timing.responseEnd -performanceAlias.timing.requestStart);if (configIdPageView) request += &pv_id= + configIdPageView;/ update cookiescookieVisitorIdValues.lastEcommerceOrderTs =isDefined(currentEcommerceOrderTs) & String(currentEcommerceOrderTs).length? curren

35、tEcommerceOrderTs: cookieVisitorIdValues.lastEcommerceOrderTs;setVisitorIdCookie(cookieVisitorIdValues);setSessionCookie();/ tracker plugin hookrequest += executePluginMethod(pluginMethod, tracker: trackerInstance,request: request);if (configAppendToTrackingUrl.length) request += & + configAppendToT

36、rackingUrl;if (isFunction(configCustomRequestContentProcessing) request = configCustomRequestContentProcessing(request);returnrequest;統(tǒng)計(jì)代碼每次都會(huì)傳送數(shù)據(jù)，每次請(qǐng)求都會(huì)帶上串的參數(shù)，這些參數(shù)都是簡(jiǎn)寫(xiě)，下做個(gè)簡(jiǎn)單說(shuō)明（如有不正確的地，歡迎指正），部分參數(shù)還沒(méi)作出合適的解釋，例如UUID的成規(guī)則等。先將這些參數(shù)分為兩部分，第部分如下所列：1、idsite：站ID2、rec：1（寫(xiě)死）3、：隨機(jī)碼4、h：當(dāng)前時(shí)5、：當(dāng)前分鐘6、s：當(dāng)前秒數(shù)7、url：當(dāng)前純凈地址

37、，只留域名和協(xié)議8、_id：UUID9、_idts：訪問(wèn)的時(shí)間戳10、_idvc：訪問(wèn)數(shù)11、_idn：新訪客（前尚未使）12、：訪問(wèn)來(lái)源的時(shí)間戳13、_viewts：上次訪問(wèn)的時(shí)間戳14、cs：當(dāng)前頁(yè)的字符編碼15、send_image：是否圖像請(qǐng)求式傳輸數(shù)據(jù)16、gt_ms：內(nèi)容加載消耗的時(shí)間（響應(yīng)結(jié)束時(shí)間減去請(qǐng)求開(kāi)始時(shí)間）17、pv_id：唯性標(biāo)識(shí)再列出第部分，于統(tǒng)計(jì)瀏覽器的功能，通過(guò)Navigator對(duì)象的屬性（mimeTypes、javaEnabled等）和Screen對(duì)象的屬性（width與height）獲得。1、pdf：是否持pdf件類型2、：是否持QuickTime Playe

38、r播放器3、realp：是否持RealPlayer播放器4、wma：是否持MPlayer播放器5、dir：是否持Macromedia Director6、fla：是否持Adobe FlashPlayer7、java：是否激活了Java8、gears：是否安裝了Google Gears9、ag：是否安裝了Microsoft Silverlight10、cookie：是否啟了Cookie11、res：屏幕的寬和（未正確計(jì)算清顯器）上這11個(gè)參數(shù)的獲取代碼，可以參考下這個(gè)法（同樣存在于tracker.js中），注意看代碼中的pluginMap變量（已標(biāo)紅），它保存了多個(gè)類型，來(lái)檢測(cè)是否安裝或啟了指定的插件或功能。/* Browser features (plugins, resolution, cookies)*/function detectBrowserFeatures() vari,mimeType,pluginMap= / document typespdf: application/pdf,/ media playersqt: video/quicktime,realp: audio/x-pn-realaudio-plugin,wma: