某客戶系統(tǒng)健康檢查報告

1、XXXX客戶系統(tǒng)健康檢查報告一、綜述3二、系統(tǒng)環(huán)境3核心業(yè)務(wù)系統(tǒng)3三、工作日志4四、系統(tǒng)健康檢查詳細列表41.核心業(yè)務(wù)系統(tǒng)4五、日常維護建議10六、故障出現(xiàn)時，系統(tǒng)處理的步驟11七、命令介紹11一、 綜述應(yīng)XXXX客戶要求，倚博公司中間件技術(shù)專家針對客戶的核心業(yè)務(wù)系統(tǒng)做了一次完整的系統(tǒng)健康檢查。整體上講，公司的核心系統(tǒng)運行比較穩(wěn)定，故障率和客戶業(yè)務(wù)系統(tǒng)的MTBF (Mean Time Between Failure)都比較小。但從系統(tǒng)維護的管理工作的規(guī)范和完善角度講，還未建立完善中間件的維護，監(jiān)控的方案?，F(xiàn)階段，主要是利用維護人員的現(xiàn)場應(yīng)急處理經(jīng)驗保證系統(tǒng)的運行正常，未合理的規(guī)劃維護流程，未確

2、認中間件的監(jiān)控指標(biāo)，未建立完善的監(jiān)控，預(yù)警和應(yīng)急處理方案。本文檔總結(jié)了這次現(xiàn)場工作的各項情況，并根據(jù)現(xiàn)場維護人員的要求，增加了系統(tǒng)維護方法的扼要說明。二、 系統(tǒng)環(huán)境核心業(yè)務(wù)系統(tǒng)客戶系統(tǒng)名稱：核心業(yè)務(wù)系統(tǒng)OS CPU：8OS Memory: 16 GBOS: Red Hat AS 4 KENERL: 2.6.9-42.ELsmpJBOSS Application Server 4.2.3JDK：Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)Java HotSpot(TM) Server VM (build

3、 1.5.0_17-b04, mixed mode)三、 工作日志2008年6月5日 上午 9：30 系統(tǒng)健康檢查四、 系統(tǒng)健康檢查詳細列表1. 核心業(yè)務(wù)系統(tǒng) 操作系統(tǒng)配置檢查系統(tǒng)配置：名稱建議值實際值是否符合要求操作系統(tǒng)版本Linux/HpRed Hat AS4Y主機名N/AApp1YCPU> 1GHZ8*1.9G,Y內(nèi)存> 2G16GY磁盤空間>20G360GY系統(tǒng)內(nèi)核參數(shù)：名稱建議值實際值是否符合要求說明文件句柄數(shù)4096或更大8192Y進程的最大文件句柄數(shù)JAVA虛擬機配置檢查虛擬機安裝路徑：/opt/jdk1.5.0_17/名稱建議值實際值是否符合要求說明JVM 版

4、本1.5JDK 1.5.0_17YJVM HEAP初始值 -Xms514M1024MYJVM HEAP最大值 -Xmx512M-1536M1536MYPermsize64M256MJBOSS配置檢查安裝路徑：/opt/jboss/jboss-4.2.3.GA參數(shù)值建議值說明線程數(shù)400一個CPU對應(yīng)50個線程左右線程的個數(shù)應(yīng)與CPU的處理能力匹配。JSP編譯的調(diào)試信息標(biāo)志falsefalse生產(chǎn)環(huán)境中建議將JSP編譯調(diào)試功能關(guān)閉，減少因編譯占用系統(tǒng)資源對系統(tǒng)的影響。開發(fā)模式的設(shè)置falsefalse應(yīng)用使用的連接池數(shù)量400400生產(chǎn)環(huán)境中建議將連接的數(shù)量與線程總數(shù)保持統(tǒng)一。應(yīng)用使用的連接池初

5、始值100400最小值和最大值保持一致，避免運行狀態(tài)在創(chuàng)建連接時消耗系統(tǒng)資源。連接池Prepared Statement Cache無10缺省JBOSS服務(wù)器是不會Cache Prepared Statement對象的，為了提高性能我們可以設(shè)定每個連接Cache的 Prepared Statement對象的個數(shù)運行期系統(tǒng)狀態(tài)檢查名稱平均值峰值是否異常CPU35%以下60%未出現(xiàn)長時間CPU使用率超過100%內(nèi)存使用3.5G以下無頁交換的情況磁盤空間20%滿足應(yīng)用日志或者JBOSS日志所用磁盤空間要求網(wǎng)絡(luò)流量200K/S800K/S無網(wǎng)絡(luò)阻塞和網(wǎng)絡(luò)風(fēng)暴運行期虛擬機狀態(tài)檢查名稱實際情況是否符合要求

6、說明JVM HEAP最小值280MYJVM HEAP最大值1100MY為今天的觀察結(jié)果運行期JBOSS組件狀態(tài)檢查名稱正常值實際值是否符合要求說明線程使用情況使用率<50%<70%Y啟動目錄core文件的檢查在JBOSS啟動目錄中發(fā)現(xiàn)“core”文件和“hs_err_pid*.log”的錯誤日志文件，說明系統(tǒng)出現(xiàn)過JVM core dump和異常退出故障，故障出現(xiàn)在一個虛擬機的解壓縮操作，希望能通過進一步與開發(fā)商，維護人員溝通找出具體的故障原因，并提出解決方案。日志檢查名稱建議值實際值是否符合要求說明日志中Error信息檢查Review All LogsReview All Log

7、sN對日志中Error消息進行檢查，參考下面例子。日志文件中發(fā)現(xiàn)錯誤信息如下：1) 啟動JBoss時報如下錯誤：出現(xiàn)過一次22:53:52,907 ERROR MainDeployer Could not create deployment: 新建文件夾/jboss-4.2.2.GA/server/default/conf/jboss-service.xml org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException) at org.j

8、boss.system.ServiceConfigurator.install(ServiceConfigurator.java:196) at org.jboss.system.ServiceController.install(ServiceController.java:226) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMetho

9、dAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at erceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at

10、org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy4.install(Unknown Source) at org.jboss.deployment.SARDeployer.create(SARDe

11、ployer.java:249) at org.jboss.deployment.MainDeployer.create(MainDeployer.java:969) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:818) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766) at sun.reflect.

12、NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at erceptor.ReflectedDispatcher.invoke(ReflectedDispatc

13、her.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at erceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at erceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanO

14、perationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt

15、.java:210) at $Proxy5.deploy(Unknown Source) at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482) at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362) at org.jboss.Main.boot(Main.java:200) at org.jboss.Main$1.run(Main.java:508) at java.lang.Thread.run(Unknown Source) Caused

16、 by: java.lang.reflect.UndeclaredThrowableException at org.jboss.system.ServiceCreator.install(ServiceCreator.java:236) at org.jboss.system.ServiceCernalInstall(ServiceConfigurator.java:451) at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:171) . 36 more Caused

17、 by: java.lang.Error: ERROR: it turns out the root cause is not always a runtime exception! at org.jboss.mx.server.MBeanServerImpl.handleInstantiateExceptions(MBeanServerImpl.java:1295) at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:1237) at org.jboss.mx.server.MBeanServerIm

18、pl.instantiate(MBeanServerImpl.java:286) at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:344) at org.jboss.system.ServiceCreator.install(ServiceCreator.java:157) . 38 morejboss-service.x配置錯誤，通過修改具體錯誤配置，并解決了這個問題。2) NoClassDefFoundError錯誤，當(dāng)前的類裝載器不能找到這個類文件，頻率：出現(xiàn)一次java.lang.NoCla

19、ssDefFoundError: Could not initialize class org.jboss.aspects.security.SecurityActions$ContextInfoActions at org.jboss.aspects.security.SecurityActions.getContextException(SecurityActions.java:467) at org.jboss.aspects.security.AuthenticationInterceptor.authenticate(AuthenticationInterceptor.java:13

20、1) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:66) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.

21、ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invok

22、eNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Pro

23、xy147.getInitConfigAsInteger(Unknown Source) at com.police.alarm.AlarmAction.execute(AlarmAction.java:28) at org.apache.struts.action.RequestPcessActionPerform(RequestProcessor.java:431) at org.apache.struts.action.RequestPcess(RequestProcessor.java:236) at org.apache.struts.

24、action.ActionScess(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterC

25、ernalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.police.filter.SystemUserFilter.doFilter(SystemUserFilter.java:61) at org.apache.catalina.core.ApplicationFilterCernalDoFilter(Application

26、FilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.police.filter.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:37) at org.apache.catalina.core.ApplicationFilterCernalDoFilter(ApplicationFilterChain.j

27、ava:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterCernalDoFilter(ApplicationFilterChain.java:235) at org.apac

28、he.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAss

29、ociationValve.invoke(SecurityAssociationValve.java:182) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.

30、java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.conn

31、ector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Pcess(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHcess(Http11Protocol.java:583) at .JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619)NoClassDefFoundError錯誤，通過找到類所在的文件，并通過正確配置或使其在正確的類路徑中來解決此問題。 故障匯總1)