Solaris系統(tǒng)性能管理

Solaris系統(tǒng)性能管理1精選2021版課件我們首先要做的問客戶正確的問題打沒打補丁？在哪個環(huán)節(jié)出現(xiàn)問題？吞吐量,響應時間,網(wǎng)絡,I/O,應用,…....明白客戶怎么和怎樣提出要求測試和想法是否合理?‘它應該跑得更快...’,‘很多的CPU空閑...'開始的支持和建議收集基本的性能參數(shù)使用工具進行分析,將圖形化結果展現(xiàn)給客戶,觀察所謂的‘峰值'識別資源的限制用自己的發(fā)現(xiàn)提出自己的建議2精選2021版課件Solaris性能分析的目的識別Solaris操作系統(tǒng)的瓶頸做出增加性能的適當建議并不作為應用的調試和Kernel調試從基本的統(tǒng)計數(shù)據(jù)中發(fā)現(xiàn)有用的東西3精選2021版課件性能管理4精選2021版課件性能管理的定義性能管理就是：測量，分析和優(yōu)化計算資源，以提供給最終用戶可認可的服務5精選2021版課件一般術語ThroughputCountofnumberoftransactionsLatencyTimeittakestodosomethingUtilizationAmountofresourcesconsumedduringaction6精選2021版課件性能的層面應用操作系統(tǒng)硬件網(wǎng)絡商務7精選2021版課件Solaris性能分析8精選2021版課件基本的原則系統(tǒng)是一系列的資源CPU(s)memorybussesdisks,diskcontrollersnetworksoperatingsystemsDBMSsystems(especiallylocksandinternallatches)性能的降低是以上一個或幾個資源枯竭所造成的。9精選2021版課件從哪做起一旦瓶頸被發(fā)現(xiàn)，按下列順序調試：應用數(shù)據(jù)庫硬件Solaris內核參數(shù)10精選2021版課件潛在的瓶頸問題DiskNetworkMemoryCPU11精選2021版課件磁盤性能分析12精選2021版課件硬盤瓶頸無足夠空間響應時間長不良的規(guī)劃RAID設定文件系統(tǒng)問題數(shù)據(jù)庫問題13精選2021版課件確認硬盤瓶頸使用sar、iostat查看響應時間,磁盤使用率,隊列長度,請求分配了解磁盤緩存是否工作14精選2021版課件相關的訪問時間Device RealTime Second Rel.timeCPURegister 2nsec 2x10-9 2sec.CPUcache 20nsec 2x10-9 20sec.MainMemory 2000nsec. 2x10-8 2-3min.Disk 20msec 2x10-3 7month15精選2021版課件iostat-x30

extendeddevicestatisticsdevicer/sw/skr/skw/swaitactvsvc_t%w%bfd00.00.00.00.00.00.0279.000sd00.00.00.00.00.00.00.000sd10.10.90.76.30.00.172.101sd40.00.00.00.00.00.00.000sd60.00.00.00.00.00.00.000nfs10.00.00.00.00.00.00.700nfs20.00.00.10.40.00.0298.100nfs30.00.00.20.00.00.035.400

Wait–隊列長度,等待磁盤處理的請求個數(shù)svc_t–平均服務時間（毫秒）%w–等待處理請求次數(shù)的百分比%b–忙時百分比<30%良好>60%有問題16精選2021版課件iostatI/Osize=(kr/s)/(r/s)srv_t接近磁盤尋址時間Wait:#OS對磁盤請求等待,>0warningActv:#在磁盤處理中而未完成的請求個數(shù)17精選2021版課件iostatr/s,w/s:Averagereads/writespersecond.Kr/s,Kw/s:AverageKbread/writtenpersecond.wait:Timespentbyaprocesswhilewaitingforblock(egdisk)I/Otocomplete.actv:Numberofactiverequestsinthehardwarequeue.%w:Occupancyofthewaitqueue.%b:Occupancyoftheactivequeuewiththedevicebusy.svc_t:Servicetime(ms).Includeseverything:waittime,activequeuetime,seekrotation,transfertime.us/sy:User/systemCPUtime(%).wt:WaitforI/O(%).id:Idletime(%).18精選2021版課件使用iostat查問題The%busycolumnshowsdiskutilization.Disksutilizedover65%areaproblem;disksover30%-35%canbeimprovedbyRAIDmehtods.Anydiskutilizedmorethan95%isaSERIOUSproblem.Theservicetimeshowshowlonganindividualdiskrequestwaits.Itshouldbeclosetotheaverageseektimeofthedisk.Spikesofhighsvc_tarenotaproblem.Thisdiskisatypicalproblemdisk.Theactvcolumnshowsthatthereare20outstandingI/Orequestsagainstthisdrive,inadditiontobeingfullybusyandveryslow.Thisdiskisnotaproblem,despiteitshighservicetime,becauseitisnotverybusy.ItisoftenusefultocomputeaverageI/Osizefromr/sandKr/s.#iostat-xc2

extendeddevicestatistics cpudevicer/sw/skr/skw/swaitactvsvc_t%w%bussywtidmd041.00.0382.70.00.00.614.9041415900c0t3d043.92.8405.44.10.00.714.7145c1t0d021.92.8203.84.10.00.313.9031c2t0d021.92.8201.64.10.00.313.7031c7t2d00.00.00.00.00.00.00.000c7t3d00.00.10.02.00.00.0159.001c7t4d0115.92.8228.44.10.019.7311.7199TypicalI/OsizeonthisdiskisKr/r;inthiscasetypicalI/Osizeis224.8/115=2K.svc_tincludestimewaitinginthequeue.Whensvc_tishigh,theservicetimeisapproxsvc_t/actv,about15msinthiscase.actvandwaitarerespectivelythenumberofrequestspendinginthediskitselfandintheOSwaitingtogettothedisk.wait>0isawarningsign,especiallyifthediskisinadiskarray.19精選2021版課件iostat磁盤名稱(-n)I/O每個分區(qū)(-p)可顯示磁帶的I/OshownCPU占用資源（-c)顯示每個磁盤的信息(-x)#iostat-xcpn60extendeddevicestatisticscpudevicer/sw/skr/skw/swaitactvsvc_t%w%bussywtidrmt048.30.014382.70.00.00.60.0091415900c0t6d00.00.00.00.00.00.00.000c11t14d00.20.11.41.20.00.018.600c11t14d0s00.00.10.41.20.00.024.900c11t14d0s10.00.00.00.00.00.00.000c11t14d0s20.00.00.00.00.00.00.000c11t14d0s30.00.00.00.00.00.00.000c11t14d0s40.00.00.00.00.00.00.000c11t14d0s60.10.01.00.00.00.010.200c11t14d0s70.00.00.00.00.00.00.000statsforwholedisk20精選2021版課件解決在iostat找到的問題采用RAID模式，對一個磁盤的訪問轉化為對多個磁盤的訪問當磁盤過忙時使用RAID-0當應用為multi-stream讀，使用RAID-1*當應用為讀方式時用RAID-5*多文件的建立/刪除:使用PrestoServe

PrestoServe:在數(shù)據(jù)真正寫入磁盤之前，就返回給用戶寫完信息21精選2021版課件怎么處理磁盤瓶頸負載平衡(striping,partitioning)增加磁盤的數(shù)量分配SWAP區(qū)將關鍵數(shù)據(jù)出存在低柱面相關數(shù)據(jù)放在同一分區(qū)不要將磁盤存滿增加memory(RAIDControllerMEM/UFS/DBcache)

22精選2021版課件數(shù)據(jù)庫和文件系統(tǒng)默認的newfs參數(shù)不適合數(shù)據(jù)庫UFS

singlewriterlockvs.multiplewrites使用多個數(shù)據(jù)庫文件或其它...on使用裸設備

使用forcedirection方式mount磁盤系統(tǒng)UFS現(xiàn)在的logging,directI/O,cocurrentI/O：Perfasvxfs+quickI/O23精選2021版課件Rawvs.UFS

FastEliminatessinglewriterissueDifficulttomanageDifficulttobackupNocacheorbufferingSlowerEasiertomanageEasiertobackupListedinvfstab,mount,df,lsCachedinRAM

24精選2021版課件大型數(shù)據(jù)庫的優(yōu)化建議setmaxphys=8388608#LargeSCSItransferssetufs_LW=4194304#increasewritethrottleforlargesystemssetufs_HW=67108864setmaxpgio=65536#speeduppagescannersetfastscan=65536#speeduppagescanner25精選2021版課件網(wǎng)絡性能分析26精選2021版課件netstatIerrsmeansthatthesystemwasunabletoprocessinputpacketsbeforethebufferoverflowed.UsuallythisisbecausetheCPUsorthereceivingprocessesweretoobusytohandlemoredata.(seenetstat-sformoreinfo)Oerrsusuallymeanthatthereisaproblemofthephysicalnetwork,forexamplenoiseinthewiresorbrokenrouters.Thisformwatchesasinglenetworkinterfaceovertime#netstat-iNameMtuNet/DestAddressIpktsIerrsOpktsOerrsCollisQueuelo08232loopbacklocalhost496445404964454000be01500mpk10-013labrouter4327793027665071328580nf04352mpk10-pmlab5lab-188200348602657753000sa09120mpk10-pmlab1lab-184195324515302226066530998400qe01500mpk10-pmlab2lab-1852570806903000qe11500mpk10-pmlab3lab-186180826353000hme01500mpk10-pmlab4lab-187286900838405000tr02052mpk10-pmlab8lab-1918084890808512000#netstat-Ieri02inputeri0outputinput(Total)outputpacketserrspacketserrscollspacketserrspacketserrscolls43305860277039113286314095394153015905096531132703210220012403070026015001180120003402500138015500200280014001420024011000237025100#27精選2021版課件測量網(wǎng)絡吞吐netstat-i5

inputhme0outputinput(Total)outputpacketserrspacketserrscollspacketserrspacketserrscolls7111436069473100385458711143606947310038545813000013000021010021010032018003201800Collisionrate=(Collis/Opkts)>10%網(wǎng)卡過載。需要減少網(wǎng)卡的負載，或將負載分配到其他網(wǎng)卡或服務器

如果input和output包的總和:10Mbs：高于600100Mbs高于6000,網(wǎng)絡片過忙如果

collisions/second>120,網(wǎng)卡過載28精選2021版課件內存性能分析29精選2021版課件內存瓶頸沒有足夠的物理內存沒有足夠的系統(tǒng)虛擬內存沒有足夠的進程虛擬空間不必要的/無用的進程用戶進程內存分配30精選2021版課件我們應該關心的是否不足?內存用在什么地方？Kernel,application,filesystems

文件系統(tǒng)cache使用率哪些文件使用cache

應用使用內存的方式ApplicationssharememoryLibrariessharememory31精選2021版課件不同Memory的種類32精選2021版課件r:Runqueuelength.b:ProcessesblockedwhilewaitingforI/O.w:Idleprocesseswhichhavebeenswapped.swap,free-availablespaceinKBsr-scanrate,totalnumberofpagesscannedprocsmemorypagerbwswapfreeremfpipofrdesr0001158410824013610100000564963192029704400000564643160024304400000563043144025201100000564163112025204400內存使用狀態(tài)監(jiān)控vmstat533精選2021版課件procs/r:Runqueuelength.procs/b:ProcessesblockedwhilewaitingforI/O.procs/w:Idleprocesseswhichhavebeenswapped.memory/swap:Free,unreservedswapspace(Kb).memory/free:Freememory(Kb).page/re:Pagesreclaimedfromthefreelist.(Ifapageonthefreeliststillcontainsdataneededforanewrequest,itcanberemapped.)page/mf:Minorfaults(pageinmemory,butnotmapped).(Ifthepageisstillinmemory,aminorfaultremapsthepage.Itiscomparabletothevfltsvaluereportedbysar-p.)page/pi:Pagedinfromswap(Kb/s).(Whenapageisbroughtbackfromtheswapdevice,theprocesswillstopexecutionandwait.Thismayaffectperformance.)page/po:Pagedouttoswap(Kb/s).(Thepagehasbeenwrittenandfreed.Thiscanbetheresultofactivitybythepageoutscanner,afileclose,orfsflush.)page/fr:Freedordestroyed(Kb/s).(Thiscolumnreportstheactivityofthepagescanner.)page/de:Freedafterwrites(Kb/s).(Thesepageshavebeenfreedduetoapageout.)page/sr:Scanrate(pages).Notethatthisnumberisnotreportedasa"rate,"butasatotalnumberofpagesscanned.disk/s#:Diskactivityfordisk#(I/O'spersecond).faults/in:Interrupts(persecond).faults/sy:Systemcalls(persecond).faults/cs:Contextswitches(persecond).cpu/us:UserCPUtime(%).cpu/sy:KernelCPUtime(%).cpu/id:Idle+I/OwaitCPUtime(%).內存使用狀態(tài)監(jiān)控參數(shù)34精選2021版課件vmstatThefirstlinefromvmstatisasummaryofallactivitysinceboottime!Mostsystemshavealotofidletime,sothisisn’tasusefulasonemightexpect.Thecpucolumnstellifthecpucomplexisoverloaded.Ifthereisnoidletime(asinthissample),thecpuispartoftheproblem.Thisparticularsliceisheavilysystem-bound,whichmightbeaproblem.Thescanrateisthemostimportantindicatorofashortageofrealmemory.Theprocscolumnshowshowmanyprocessesarerunning,waitingforI/O,orwaitingforotheractivity(likealockorasleep).The“r”columnincludesprocessesthatarerunningonthecpu.Thesystemfaults(sy)columnshowsthenumberofsyscallsexecutedpersecond.Payspecialattentiontothis,especiallywhentimesharingsystemsareinvolved.ThepageinandpageoutcolumnsshowbothfilesystemI/Oactivityandvirtualmemoryactivity.IfthesystemisanNFSserver,highpiandpoarenormal.ifit’satimesharingsystemrunningcompilers,thesamebehaviorisquiteabnormal.

#vmstat10procsmemorypagediskfaultscpurbwswapfreeremfpipofrdesrm0m1m2m3insycsussyid

000237602024803160610021112302834230198

183050747260488838542952254860439107107535358850119905248018205074404152563690302025604025773895049637479697755450

18205074964264513609480215438012890985252545473291054460

183050744845843236974542012004801011015151450478586452480

1930507376604827389331160120222012967675344941102851490

35精選2021版課件解決在vmstat找到的問題無空閑時間–增加cpu(s)空閑,但有進程在等待隊列里–查看磁盤使用率高掃描頻率sr>200-增加內存pageins和pageouts值異常–查看scanrate36精選2021版課件解決在vmstat找到的問題‘w’–進程已經(jīng)swapped，但沒有在被執(zhí)行查找每一個進程的狀態(tài)/usr/ucb/ps–runing?sleeping?runable?‘b’–進程被本地I/O阻止磁帶/磁盤查找哪一個進程的被阻止非官方的方法:echo'$<threadlist'|adb-k><dir>/<file>LookforbiowaitThosearethethreadwereblockedonI/O37精選2021版課件找到問題pmap–xPID進程使用內存的情況ps–ly或topipcs-mb:sharedmemory使用情況38精選2021版課件CPU性能分析39精選2021版課件CPU瓶頸過多的系統(tǒng)占用時間進程優(yōu)先級互斥鎖40精選2021版課件查看CPU使用率mpstat5cswicswmigrsmtxsrwsysclusrsyswtidl44114100017778508718523000615210971932400063421097190220006132109718320000612210981922600064821097smtx–互斥鎖的個數(shù)usr–用戶使用的資源sys–系統(tǒng)使用的資源(>35%isbad)wt–等待I/O操作完成的空閑時間idl–CPU空閑時間41精選2021版課件mpstatCPU互斥鎖的多少I/O設備的等待時間smtxshowsfailedmutexenters,ameasureofSolarisinternallockcontention.The“wt”columnindicatestimespentinI/Owait.Ifthiscolumnshowsmuchtime,thesystemisdefinitelyI/Obound.#mpstat5CPUminfmjfxcalintrithrcswicswmigrsmtxsrwsysclusrsyswtidl0217162723713813574792333456427571112061448250546587948853735254414594197161629920113974493313431327581252161654343236109154873236015335110CPUminfmjfxcalintrithrcswicswmigrsmtxsrwsysclusrsyswtidl01772196238106135071148621174462435321178156825014431331741466110562524948144861893264152122664142528265234352251572150572417140674149619644224255242精選2021版課件解決在mpstat找到的問題smtx值較高代表Solaris中鎖較多FasterCPU-smtxhigher鎖的產(chǎn)生43精選2021版課件I/O等待時間系統(tǒng)空閑有時是因為所有的任務都在等待I/O參數(shù)值包括磁盤的I/O不包括:對NFS等待DBMSNetworkI/O其實I/O等待的時間應大于wt所顯示的時間44精選2021版課件I/O等待如果I/O等待有問題,檢查iostat-x查看vmstat結果中“b”列中的結果來確定被堵塞的進程個數(shù)45精選2021版課件減輕CPU的瓶頸問題增加CPU的個數(shù)，或換更快的CPU增加CPUcacheSolarisResourceManager如果scanrate值較高，增加內存*調整進程的優(yōu)先級(priocntl,nice)控制處理器使用率(psrset)調整時間片(kerneltuning)46精選2021版課件常用命令psrinfo-v-numberandspeedofCPUsprtconf-memoryanddevicetreeinformationprtdiag-CPU,Cache,Board,Memory,SBUSslotinfo/etc/release-SolarisHWreleaseinfoshowrev-p-Installedpatchinfosysdef-kernelparameterinfoswap-l-swappartitioninfopkginfo-installedpackages47精選2021版課件Solaris內核優(yōu)化

48精選2021版課件內核優(yōu)化的結果和期望如果整個系統(tǒng)的使用率已到最大，優(yōu)化沒有幫助對內核的優(yōu)化一般可以增加3%-10%的吞吐及反應時間極端情況下可增加200%的吞吐能力/db/doc/806-4015/6jd4gh8g4?l=zh_TW&a=view49精選2021版課件內核參數(shù)調整(bufhwm)在內存中，為I/O所開的最大的緩存數(shù)用于寫入metadata（superblocks,inodes,indirectblocks,directories）默認值為物理內存的2%512Megphysicalmemory==10.2Megbufhwm4Gigphysicalmemory==81.9Megbufhwm64Gigphysicalmemory==1.3Gigbufhwm50精選2021版課件內核參數(shù)調整(bufhwm)沒有固定的值，只有合理的值依靠內存大小和使用情況定值In/etc/system: setbufhwm=10240

51精選2021版課件內核參數(shù)調整(bufhwm)#sar-b510

SunOShostname5.2Genericsun4c08/06/9323:43:39bread/slread/s%rcachebwrit/slwrit/s%wcachepread/spwrit/sAverage0251003228800增加bufhwm的值當：

bread/s,lread/s>50%rcache<90%wcache<65bread/s:transferspersecondofdatabetweensystembufferanddiskorotherblockdevices%rcache,%wache:chachehitratios52精選2021版課件內核參數(shù)調整(maxusers)maxusers參數(shù)設定不同kernel表大小的值值的大小參照物理內存的大小，動態(tài)調整等于物理內存M數(shù)的值動態(tài)調整最高值為1024手工調整最高值為2048(4096inSolaris8)影響其它設置Numberofptys,proctableentries,openfiles,...In/etc/system: setmaxusers=204853精選2021版課件內核參數(shù)調整(maxphys)物理I/O的最大請求值（inbytes)經(jīng)常與VeritasandSVM配合使用，用于大規(guī)模及快速存儲但在大型決策支持系統(tǒng),大型數(shù)據(jù)通用信息系統(tǒng)及大型高性能計算系統(tǒng)中可不必使用默認值為131072byte-128K對裸設備可增加到1M,最大32MIn/etc/system: setmaxphys=104857654精選2021版課件內核參數(shù)調整(pt_cnt)默認生成偽終端（pty)的最大個數(shù)1.偽終端在Solaris里的作用：支持telnet,rlogin,rsh遠程登陸方式

2.提供XWindow的訪問界面需要“boot-r”或drvconfig命令才能生效從Solaris8開始，此參數(shù)已經(jīng)可以根據(jù)內存的大小而‘動態(tài)’調整如此值過大則會浪費CPU的資源In/etc/system: setpt_cnt=102455精選2021版課件內核參數(shù)調整(max_nprocs)在系統(tǒng)中可以生成的最大進程數(shù)，包括系統(tǒng)進程及用戶進程默認值為(10+16*maxusers)范圍266-pidmax從Solaris8開始，最大值可到30,000In/etc/system: setmax_nprocs=3276856精選2021版課件內核參數(shù)調整(maxuprc)每一個用戶可以生成的進程數(shù)默認值為(max_nproc–reserved_proces)范圍266-maxpid

In/etc/system: setmaxuprc=6553557精選2021版課件內核參數(shù)調整(reserved_procs)系統(tǒng)為root保留的UID進程數(shù)默認值為5范圍5-MAXINTIn/etc/system: setreserved_procs=1558精選2021版課件內核參數(shù)調整(pidmax)最大的進程ID(PID)數(shù)默認值為30,000范圍266-999,999In/etc/system: setpidmax=99999959精選2021版課件文件系統(tǒng)緩存以下是Solaris文件系統(tǒng)里幾種cache：DNLC(DirectoryNameLookupCache):這個緩存中存儲著目錄查找信息inodecache:這個緩存中存儲著內存中文件的信息

(size,accesstime,etc).它是一個連接表存儲著在內存中文件的索引點和指針rnodecache:它管理著在NFSclients上的NFS-mounted信息.buffercache:這個緩存中存儲著inode,indirectblock，cylindergroup-relateddiskI/O.

60精選2021版課件元數(shù)據(jù)（Metadata）概念數(shù)據(jù)的數(shù)據(jù)(dataaboutdata)結構化數(shù)據(jù)(Structureddataaboutdata)用于描述數(shù)據(jù)的內容（what）、覆蓋范