經(jīng)濟(jì)資本模型驗(yàn)證方法PPT學(xué)習(xí)教案

1、會(huì)計(jì)學(xué)1經(jīng)濟(jì)資本模型驗(yàn)證方法經(jīng)濟(jì)資本模型驗(yàn)證方法第1頁/共36頁第2頁/共36頁第3頁/共36頁第4頁/共36頁第5頁/共36頁第6頁/共36頁While we will describe the types of validation processes that are in use or could be used, note that the list is not comprehensiveWe do not suggest that all techniques should or could be used by banks We wish to demonstrate that

2、 there is a wide range of techniques potentially covered by our broad definition of validationThis is creating a layered approach, the more (fewer) of which that can be provided, the more (less) comfort that validation is able to provide evidence for or against the performance of the model Each vali

3、dation process provides evidence for (or against) only some of the desirable properties of a model The list presented below moves from the more qualitative to the more quantitative validation processes, and the extent of use is briefly discussed第7頁/共36頁The philosophy of the use test as incorporated

4、into the Basel II framework: if a bank is actually using its risk measurement systems for internal purposes, then we can place more reliance on itApplying the use test successfully will entail gaining a careful understanding of which model properties are being used and which are notBanks tend to sub

5、ject their models to some form of qualitative review process, which could entail: Review of documentation or development workDialogue with model developers or model managersReview and derivation of any formulae or algorithmsComparison to other firms or with publicly available information Qualitative

6、 review is best able to answer questions such as: Does the model work in theory? Does the model incorporate the right risk drivers? Is any theory underpinning it conceptually well-founded? Is the mathematics of the model right?第8頁/共36頁Extensive systems implementation testing is standard for producti

7、on-level risk measurement systems prior to implementationE.g., user acceptance testing, checking of model code, etc.These processes could be viewed as part of the overall validation effort, since they would assist in evaluating whether the model is implemented with integrityManagement oversight is t

8、he involvement of senior management in the validation processE.g., reviewing output from the model & using the results in business decisions Senior management knowing how the model is used & outputs are interpreted This should take account of the specific implementation framework adopted and the ass

9、umptions underlying the model and its parameterizationData quality checks refer to the processes designed to provide assurance of the completeness, accuracy and appropriateness of data used to develop, validate and operate the model E.g., Review of: data collection and storage, data cleaning of erro

10、rs, extent of proxy data, processes that need to be followed to convert raw data into suitable model inputs, and verification of transaction data such as exposure levels While not traditionally viewed by the industry as a form of validation, increasingly forming a major part of supervisory thinking第

11、9頁/共36頁As all models rest on premises of various kinds, varying in the degree to which obvious, we have examination of assumptionsCertain aspects of an EC model are built-in and cannot be altered without fundamentally changing the model To illustrate, these assumptions could be about: Fixed model pa

12、rameters (PDs, correlations or recovery rates)Distributional assumptions (margins, copulae & shape of tail distributions)Behavior of senior management or of customers Some banks go through a deliberate process of detailing the assumptions underpinning their models, including examination of:Impact on

13、 model outputsLimitations that the assumptions place on model usage and applicability第10頁/共36頁A complete validation of an EC model would involve the inputs and parameters, both statistically estimated and notExamples of estimated (assumed) parameters are the main IRB parameters (PD or LGD) (correlat

14、ions, PD in a low default portfolio) Techniques could include assessing parameters against: Historical data through replication of estimatorsOutcomes over time through backtestingMarket-implied parameters (e.g., implied vol or correlation, CDS spreads for PD)Materiality through sensitivity testingTh

15、is testing of input parameters could complement examination of assumptions previously & sensitivity testing to described laterHowever, that checking of model inputs is unlikely to be fully satisfactory since, every model is based on underlying assumptionsThe more sophisticated the model, the more su

16、sceptible to model error, so checking input parameters will not help here However, model accuracy and appropriateness can be assessed, at least to some degree, using the processes described in this section第11頁/共36頁Model replication is useful technique that attempts to replicate EC model results obta

17、ined by the bank This could use independently developed algorithms or data sources, but in practice replication might leverage a banks existing processes E.g., run a model of the same type or class on a the banks data-set However, but once the either the original or test model has been validatedThis

18、 technique and the questions that often arise in implementing replication can help identify if: Definitions & algorithms the bank claims to use correctly are understood by staff who develop, maintain, operate and validate the model The bank is using in practice the modeling framework that it purport

19、s to Computer code is correct, efficient and well-documented Data claimed to be used by the bank to obtain its results is in fact being usedHowever, this technique is rarely sufficient to validate models, and in practice there is little evidence of it being used by banks for either validation or to

20、explore the degree of accuracy of their models Note that replication simply by re-running a set of algorithms to produce an identical set of results would not be sufficient model validation due diligence第12頁/共36頁Benchmarking the comparison of a banks EC model to alternative models on the banks portf

21、olio E.g., to a vendor model after standardization of parametersAmong the most commonly used forms of quantitative validation used internallyA limitation of benchmarking is it only provides relative assessments and provides little assurance that any model accurately reflects reality or about the abs

22、olute levels of capital Therefore, as a validation technique, benchmarking is limited to providing comparison of one model against another or one calibration to others, but not testing against realityIt is therefore difficult to assess the degree of comfort provided by such benchmarking methods, as

23、they may only be capable of providing broad comparisons confirming that input parameters or model outputs are broadly comparable第13頁/共36頁There may be good reasons why models produce outliers in benchmarking, all of which complicate interpretation of the results:May be designed to perform well under

24、differing circumstancesMay be more or less conservatively parameterizedMay differ in their economic foundations Comparisons of internal EC are made with varied alternatives:Industry survey resultsRating agency or industry-wide modelsConsultancy marketed models Academic papersRegulatory capital model

25、s 第14頁/共36頁Hypothetical portfolio testing is an examination of either different banks EC models on a reference portfolio, or different banks EC output from a given reference modelThis is typically a either a reference model or portfolio external to any one bankFrom a supervisory perspective: permits

26、 identification of models that produce outliers amongst a set of banks A “model risk management” toolAlternatively, this helps supervisors identify banks that are outliers in risk with respect to a reference model A “bank portfolio risk management” toolIn either case this means comparison across ban

27、ks models against the same reference portfolio (external to the bank) or of banks themselves (their EC for a given reference model) Capable of addressing similar questions to benchmarking, but by different means The technique is a powerful one and can be adapted to analyze many of the preferred mode

28、l properties such as rank-ordering and relative risk quantification 第15頁/共36頁Backtesting addresses the question of how well the model forecasts the distribution of outcomes There are many forms of this that entail some degree of comparison of outcomes to forecasts, and there is a wide literature on

29、the subjectHowever, weak power of backtesting tests for models of risk that quantify high quantiles has been noted E.g., for portfolio credit models see BCBS (1999)Variations to the basic backtesting approach which can increase the power of the tests have been suggested in the literature: Backtestin

30、g more frequently over shorter holding periods (e.g., in market risk using a one-day standard versus the 10-day regulatory capital standardUsing cross-sectional data on a range of reference portfoliosUsing information in forecasts of the full distributionTesting expected values of distributions as o

31、pposed to high quantiles第16頁/共36頁Backtesting is useful principally for models whose outputs are a quantifiable metric with which to compare an outcomeHowever, some risk measurement systems in use have outputs cannot be interpreted in this way and cannot be backtested Such risk measurement approaches

32、 not amenable to outcomes-based validation might nevertheless be valuable tools for banks E.g., rating systems, sensitivity tests and aggregated stress losses. The role of backtesting for such models, if used, would need elaborationIn practice, backtesting is not yet a key component of banks validat

33、ion practices for economic capital purposes第17頁/共36頁Stress testing covers both stressing of the model and comparison of model outputs to stress lossesThe outputs of the model might be examined under conditions where model inputs and model assumptions might be stressed This process can reveal model l

34、imitations or highlight capital constraints that might only become apparent under stress While stress testing of regulatory capital models, particularly IRB models, is commonly undertaken by banks, there is more limited evidence of stress testing of economic capital modelsThrough a complementary pro

35、gramme of stress testing, the bank may be able to quantify the likely losses that the firm would confront under a range of stress events Comparison of stress losses against model-based capital estimates may provide a modest degree of comfort of the absolute level of capital Banks report some use of

36、this stress testing technique to validate the approximate level of model output第18頁/共36頁We have not mentioned internal audit, but validation of the overall implementation framework and process should also be subject to independent and periodic reviewThis work should be made by parties within the ban

37、king organization that are independent of those accountable for the design and implementation of the validation process A possibility is that internal audit would be in charge of undertaking this review process, and as such it could be viewed as comprising a part of the management oversight process

38、discussed previouslyThe list of validation tools also does not address the issue of adequate internal standards relevant for validation Examples of such standards include:A description of the issues that need to be addressed as part of validation The standards that capital models are expected to ach

39、ieve A series of quantitative thresholds that models need to meetWarning indicators for particular monitoring metrics Assessment against model development standards第19頁/共36頁Given the inherent limitations of EC model validation, there is an emphasis on effective reporting to avoid misuse or misunders

40、tandingA recognition of intrinsic difficulty in evaluation of high quantiles of loss over long periods, data scarcity & technical difficultiesClear reporting of such difficulties & limitations to users & senior management is necessary for them to understand that there may be greater uncertainty arou

41、nd the output from EC models Models not fully validated implies output should generally be treated with extra conservatism Understand & explore the potential costs of using models in this situation (i.e., if key assumptions prove to be inaccurate)While validation practices depend on model type & use

42、, weaknesses targeted at evaluation of overall performance might result in banks operating with inappropriately calibrated models This could be of concern if assessment of overall capital adequacy is an important application of the model Improvements in these areas could include further benchmarking

43、 & industry-wide exercises, backtesting, P/L analysis and stress testing第20頁/共36頁Main areas of improvement seen to be in benchmarking of model parameters and the conduct of cross-firm comparisons of models E.g.,IACPM and ISDA study (2006) on portfolio credit risk models Evidence that banks ensure EC

44、 models are sensitive to the expected drivers of risk & outputs permit evaluation of the relative riskAlthough there is scope improvement, some signs of progressWeaknesses of validation particularly when the total capital adequacy & overall calibration of the model is an important consideration 第21頁

45、/共36頁A fundamental difficulty faced in EC modeling is the lack of data to estimate high quantiles in the tails of the loss distribution leads to a very high degree of uncertaintyOne approach to dealing with this problem is Bayesian techniques, which combine expert assessments with available data See

46、 Kiefer & Jacobs (2010, forthcoming)However, this is computationally demanding, and also requires the elicitation of a prior distribution from an expect, which is very involvedBut if the prior is diffuse, then much uncertainty still remainsAs we dont see this used in practice currently, we will not

47、further pursue this approach here As noted previously, traditional backtesting procedures as applied in market risk VaR models are impractical in an EC model settingAn alternative approach is to try to assess the accuracy of the EC output by approximating a statistical measure of uncertainty E.g., t

48、hrough resampling or bootstrap methods (Efron et al, 1986) But thin data in the tails implies confidence bounds are likely to be wide . 第22頁/共36頁“Models for Risk Aggregation and Sensitivity Analysis: An Application to Bank Economic Capital”, by Hulusi Inanoglu and Michael Jacobs, OCC & Federal Reser

49、ve BOG, Working PaperDevelops proxies for 5 risk types (credit, market, operational, trading and interest income) from historical quarterly call report data for 5 largest banks as of 4Q08Compares EC output of different copula models for combining these according to absolute levels and variabilityUse

50、 a non-parametric bootstrap to assess accuracy of output to estimation error in inputs (parameters of margins and correlations)While not a study of EC model validation per se, this illustrates several quantitative techniques discussed herein:Benchmarking / hypothetical portfolio analysis of alternat

51、ive modelsSensitivity analysis to inputsTesting accuracy of EC (model output) quantile estimatesConclusion: a non-parametric model (empirical copula) is more conservative than common copulas (e.g., Gaussian) and also less variable in the resampling experiment (more stable or accurate)第23頁/共36頁第24頁/共

52、36頁-2024x 107012x 10-7Credit Risk: GCO-20246x 10705x 10-8Op Risk: ONIE-1012x 10700.51x 10-6Market Risk:NTR-4QD-505x 10800.51x 10-8Liquidity Risk: LG-4QD-4-2024x 10805x 10-9Interest Rate Risk: IRG-4QD-50510 x 108024x 10-9Total Risk: Sum of Cr.,Ops.,Mkt.,Liqu.&Int.Historical Loss Distributions (200 La

53、rgest Banks 1984-2008)Validation Example: Alternative Risk Aggregation Models Distributions of Risk Proxies (Largest Banks As Of 4Q08)第25頁/共36頁Validation Example: Alternative Risk Aggregation Models Correlations of Risk Proxies (Largest Banks As Of 4Q08) )-202x 108-505x 108-202x 107024x 107024x 107-

54、202x 108-505x 108-202x 107024x 107Pairwise Scattergraph & Pearson Correlations of 5 Risk TypesTop 200 Banks (Call Report Data 1984-2008)024x 107CreditLiqu.Operat.MarketInt.Rt.corr(cr,ops)= 0.6517corr(mkt,liqu)= 0.1127corr(int,liqu)= 0.1897corr(cr,mkt)= 0.2241corr(ops,liqu)= 0.1533corr(mkt,int)= 0.24

55、78corr(cr,liqu)= 0.5343corr(ops,int)= -0.1174corr(ops,mkt)= 0.1989corr(cr,int)= -0.1328第26頁/共36頁Validation Example: Alternative Risk Aggregation Models Absolute EC Comparison (Largest Banks As of 4Q08)第27頁/共36頁2e+084e+086e+088e+081e+090.0e+005.0e-101.0e-091.5e-09Gaussian Copula Simulated vs. Normal

56、Approximation and Empicial Copula Annual Loss DistributionGausCopVaR-99.7%=7.64e+8K=$764B,NormApproxVaR-99.7%=6.95e+8K=$695B,EmpCopVaR-99.7%=8.59e+8K=$859B5 Risk Types (Credit,Market,Operational,Liquidity&InterestRate): 200 Large Banks (1984-2008)DensityNormal ApproximationGaussian CopulaEmpirical CopulaValidation Example: Alternative Risk Aggregation Models Absolute EC Comparison (Largest Banks As of 4Q08)第28頁/共36頁Validation Example: Alternative Risk