SoftwareX Update (1.1) to ANDURIL - A MATLAB toolbox for ANDURIL 的 SoftwareX 更新 (1.1) - 用于的 MATLAB 工具箱

Contentslistsavailableat

ScienceDirect

SoftwareX

journalhomepage:

/locate/softx

SoftwareX10(2019)100295

Softwareupdate

Update(1.1)toANDURIL—AMATLABtoolboxforANalysisandDecisionswithUnceRtaInty:Learningfromexpertjudgments

ANDURYL

CornelisMarcelPieter’tHart

a

,

b

,

?

,GeorgiosLeontaris

a

,OswaldoMorales-Nápoles

a

aCivilEngineeringandGeosciences,DelftUniversityofTechnology,TheNetherlands

bTunnelEngineeringConsultants(TEC),Amersfoort,TheNetherlands

article info

Articlehistory:

Received9July2019

Receivedinrevisedform19July2019Accepted23July2019

Keywords:

StructuredexpertjudgmentCooke’sclassicalmodelExpertopinion

PythontoolboxEXCALIBURsoftwareANDURIL

abstract

ThisisanupdatetoPII:

S2352711018300608

Inthispaper,wediscussANDURYL,whichisaPython-basedopensourcesuccessoroftheMATLABtoolboxANDURIL.TheoutputofANDURYLisingoodagreementwiththeresultsobtainedfromANDURILandEXCALIBUR.AdditionalfeaturesavailableinANDURYL,andnotavailableinitspredecessors,arediscussed.

?2018TheAuthors.PublishedbyElsevierB.V.Allrightsreserved.

Codemetadata

Currentcodeversion Code:ANDURYLv1.0,Paperv1.1

Permanentlinktocode/repositoryusedforthiscodeversion

/ElsevierSoftwareX/SOFTX_2019_237

CodeOceancomputecapsule

/10.24433/CO.7459237.v1

LegalCodeLicense GNUGeneralPublicLicense

Codeversioningsystemused None

Softwarecodelanguages,tools,andservicesused Python,SCIPY,NUMPY,MATPLOTLIB

Compilationrequirements,operatingenvironments&dependencies PythonVERSION3.6

IfavailableLinktodeveloperdocumentation/manual

/10.24433/CO.7459237.v1

Supportemailforquestions

C.M.P.tHart@tudelft.nl

Softwaremetadata

Currentcodeversion ANDURYLv1.0

Permanentlinktocode/repositoryusedforthiscodeversion

CodeOcean

LegalCodeLicense GNUGeneralPublicLicense

Codeversioningsystemused CodeOcean

Softwarecodelanguages,tools,andservicesused Python,SCIPY,NUMPY,MATPLOTLIB

Compilationrequirements,operatingenvironments&dependencies PythonVERSION3.6

IfavailableLinktodeveloperdocumentation/manual

/10.24433/CO.7459237.v1

Supportemailforquestions

C.M.P.tHart@tudelft.nl

DOIoforiginalarticle:

/10.1016/j.softx.2018.07.001

.

?

Correspondingauthorat:CivilEngineeringandGeosciences,DelftUniversityofTechnology,TheNetherlands.

E-mailaddress:

c.m.p.thart@tudelft.nl

(C.M.P.’tHart).

/10.1016/j.softx.2019.100295

2352-7110/?2018TheAuthors.PublishedbyElsevierB.V.Allrightsreserved.

PAGE

2

C.M.P.’tHart,G.LeontarisandO.Morales-Nápoles/SoftwareX10(2019)100295

C.M.P.’tHart,G.LeontarisandO.Morales-Nápoles/SoftwareX10(2019)100295

PAGE

3

Table1

OverviewofresultcomparisonAIandAYagainstCC.

Software

Numberofstudiescompared

Numberofdifferentscoresin

Table

2

Numberofscoreswithapproximationdifferences

NumberofscoreswhereAI

=AYbutdifferenttoCC

RelativeagreementaftercorrectionforapproximationandAI=AY

AI

18(55%)

13(96%)

4

9

100%

AY

33(100%)

23(96%)

8

9

99%

Motivationandsignificance

AMATLABtoolbox,namedANDURIL,

1

(AI),implementingCooke’sclassicalmodel[

1

]forstructuredexpertjudgmentispresentedin[

2

].UntilrecentlyEXCALIBUR

2

(CC)wastheonlyavailablesoftwareimplementingCooke’sclassicalmethod.ThoughEggstaff’sstudieswerebasedonaMATLABimplemen-tation

3

[

3

,

4

],thedevelopedsourcecodeforthesestudiesisnotavailablefordistribution.

InthispaperwepresentANDURYL(AY),whichisaPython[

5

]implementationofCooke’sclassicalmodel[

1

].TheprogramnamereplacingtheIwithYindicatesthattheAYsourceisbasedonPythoninsteadofMATLAB.TheprogramstructureofAIhasbeenretainedinthisimplementation.ThemainobviousadvantageofAYisthattheMATLABlicenserequiredforAIisnotrequiredforAY.OtheraddedfeatureswithrespecttoAIwillbediscussedalongthispaper.

Softwaredescription

AYisrunfromthecommandlinewiththePythonfunctionmain.py,asitdoesnothaveagraphicaluserinterface.Userscanadaptthecodetoruntheirownstudiesinsequencesaspresentedinanduryl_example.py.TheprogramstructureissetupinsuchawaythatthereisonemainPythonfunctionandurylwhichisusedtorunthefullscopeofAY.Inthismainscript,thedataobtainedfromexpertjudgmentsmaybeenteredinordertoconductthedesiredanalysis.Theinputvariablesaresetasglobalvariablesandbackedup.With‘restore’statementsthevariablescanberesettotheoriginalinputvalues,whichcanbeusedinlatercalculations,butmightalsobeusefulinfurtherdevelopmentsofAY.Inthecurrentimplementation,thisisusedintheprocessforinvestigatingtherobustnessoftheobtainedDecisionMakers(DM).ThesupportedfunctionalitiesofCooke’sclassicalmodelinAYare:

CalculationofDMusingglobalweights;

CalculationofDMusingitemweights;

CalculationofDMusingequaloruserdefinedweights;

OptimizationofDM;

Robustnesscheckitemwise;

Robustnesscheckexpertwise;

Plottingassessmentsitemwise;

Plottingrobustnessresults.

ThefunctionsofAYaresimilartothefunctionspresentedforAI.AYkeepsitsarchitectureassimilaraspossibletothatofAI.Themaindifferencehoweverisinthefunctioncalcu-late_weights,whichmergesAI’sfunctionsglobal_weightsanditem_weights.AmoredetailedexplanationoftheprogramispresentedintheSupplement.TheremainingdifferenceswillbefurtherdiscussedinSection

4

.Nextwepresentresultsofcom-paringAY’soutputtobothCCandtheMATLABimplementationAI.

Freelyavailableat

/ElsevierSoftwareX/SOFTX_2018_39

.

Freelyavailableat

/wp/excalibur

.

ThisMATLABimplementationisnotEXCALIBUR.

ComparingoutputofANDURYLwithpreviousexpertjudg-mentstudies

In[

4

],33post-2006studiesusingCooke’sclassicalmethodarepresentedusingCC.WeusethesedatatocompareoutputfromAYtobothCCandtheMATLABimplementationAIofthepreviouspaper[

2

].

Table

2

presentstheresultsreportedinTable1of[

4

](thestudynamefollowedbyCC)extendedwithcalculationsfromAI(AI)andAY(AY).

Table

2

includesthestatisticalaccuracy(SA),information(In)andthecombinedscores(Co).

Equalweight,Globalweightswithoutoptimization(GlobalNoOp.),Globalweightsoptimized(PWGlobal),Itemweightsoptimized(PWItem)andtheexpertwithhighestcombinedscore(BestExpert)arepresented.Inthesupplement,anextendedtableincludingItemweightswithoutoptimization(ItemNoOp.)andtheexpertwiththelowestcombinedscoreispresented.

Fromthe33studiesreported[

4

],14wereperformedusing5quantiles,3withquantilesotherthanthe5th,50thand95thorcontainedmissingitemsforsomeexperts.TheseresultscannotbecomparedwithAIandaremarkedby(*).OntheEBPPstudy,asoftwareerrorappearedintheMATLABcode.ThiserrorwillberesolvedinafutureupdateofAI.Hence,atotal18studieswerecomparedwithAI.Eachstudyin

Table

2

presents17numbers.Differencesbetweenthecalculationsreportedin[

4

]andAIarehighlightedinblue.Thereareatotalof153bluenumbersin

306

Table

2

andhenceanagreementof(1?13)×100≈96%between

AIandthecalculationsreportedin[

4

]forthestudiesthatcanbecompared.Fromthe13numbers4areclearlyapproximationdifferences.NoticethatthoughthenumbersinCCareMATLAB-basedwecompareourresultstothepublishedresultsin[

4

]andnowaytoinvestigatefurthertheapproximationusedin[

4

]isavailabletotheauthors.Additionally,9numbersareequaltotheresultsobtainedwithAY.Thesetwoobservationswouldbringtheagreementto100%.

Differencesbetweenthecalculationsreportedin[

4

]andAYarehighlightedinredinthesametable.Thereareatotalof23

561

rednumbersin

Table

2

andhenceanagreementof(1?23)×

100≈96%betweenAYandthecalculationsreportedin[

4

].Fromthe23rednumbers8areclearlyapproximationdifferences.Additionally,9AYresultsareequaltothoseobtainedwithAIwhichwouldbringtheagreementto≈99%.ThisresultindicatethatbothAIandAYmaybeusedwithenoughconfidencebyinterestedusers.

Theresultsofthecomparisonaresummarizedin

1

.

In

Table

2

,9valuesareequalforAIandAYbutdifferentcom-paredtoCC.Theauthorscheckedtheinputfilesofthe‘‘Icesheets"study.Itwasfoundthattherealizationfile(*.rls)andthefilewithassessments(*.dtt)presentedinconsistenciesinthelabelingofassessmentquestions.WespeculatethatthiscouldbethesourceofthismisalignmentofbothAIandAYwithCC.

Thedifferencesfoundinthe‘‘Gerstenberger",‘‘Goodheart"and‘‘Hemopilia"studyarerelatedtotheoptimizationprocess.Forexample,theoptimizationprocessfor‘‘Goodheart"datashowsinCC1expertastheoptimalcombination.ForbothAIandAYtheoptimalcombinationconsistsof3experts.WithoutthesourcecodeofCCtheauthorscannotinvestigatefurtherthissourceofmisalignment.

Table2

ComparisonofresultspresentedinTable1of[

4

](CC)andcalculationswithAI(AI)andAY(AY).

aTheauthorsfoundasoftwareerrorinAI,thisparticularstudyhasnotbeenvalidatedtoAI.InafutureupdateofAIthesoftwareerrorwillbesolved.

Fig.1.Hypotheticalexampleof4expertsassessing10seedvariables.

Table3

StatisticalaccuracyandInformativenesscomputedwithAYandCCforthehypotheticalexamplepresentedin

Fig.

1

assumingexpertselicited10th,50thand90thpercentilesoftheiruncertaintydistribution.

ExpertID

Calibration

Calibration

Information

Information

(CC)

(AY)

(CC)

(AY)

ExpertA

5.529E?10

5.530E?10

1.371

1.371

ExpertB

5.529E?10

5.530E?10

0.571

0.571

ExpertC

0.371

0.371

0.039

0.039

ExpertD

0.526

0.526

0.629

0.629

Global

0.526

0.526

0.431

0.431

(non-opt.)

Impact

TheadvantagesofAI,discussedin[

2

],withrespecttoCCareinheritedbyAY.AnumberoflimitationsofAIwerediscussedinthesupplementof[

2

].BesidesthefullopensourcecharacterusingPythonasaprogramminglanguage,twootheradvantageswereimplementedincomparisonwithCCand/orAI.Theseareelaboratedfurthernext.

Userdefinedquantiles

From

Table

2

itmaybeobservedthatAYpresentsgoodagree-mentwiththe11studiesreportedin[

4

]where5quantiles(5th,25th50th,75thand95th)wereusedtoelicitexpertjudgments,hencewedonotelaboratefurtheronthisissue.

Asstatedearlier,AYprovidestheoptionofuserdefinedquan-tiles.CCallowsfortheuseof3,4or5userdefinedquantiles.

Fig.

1

presentsahypotheticalexampleof4experts:A,BCandD,assessing10calibrationorseedvariables.Therealization(R)isalsoshown.

Intuitively,thereadermayalreadyappreciatethatexpertAwillbeinformativebutwithlowSA.ExpertBwillbelessinfor-mativeandalsopresentlowSA.TheSAforCandDwillbeequal,however,DwillbemoreinformativethanC.

Table

3

presentsacomparisonofthecalculationsofSAandinformativenessbe-tweenAYandCCassumingexpertselicited10th,50thand90thpercentilesoftheiruncertaintydistribution.Thereadermayap-preciatethattheagreementbetweenthecalculationsperformedbyCCandAYisalmostexact.

BecausethesourcecodeofAYisavailableandextendedwithrespecttoCC,practitionersmayusemorethat3,4or5userdefinedquantilestoelicitexpertjudgments.Thesamehypothet-icalexamplewithfourexpertsasin

Table

3

isusedbutwithexpertsassessing7quantiles(10th,25th,35th,50th,65th,75th

Table4

StatisticalaccuracyandInformativenesscomputedwithAYwith7quantilesforthehypotheticalexamplepresentedinSection

4.1

assumingexpertselicited10th,25th,35th50th,65th,75thand90thpercentilesoftheiruncertaintydistribution.

ExpertID

Calibrationscore

Informationscore

Un-normalizedweights

Normalizedweights

ExpertA

8.542E?08

1.3738

1.173E?07

9.403E?07

ExpertB

8.542E?08

0.5710

4.877E?08

3.908E?07

ExpertC

0.0041

0.0393

0.0002

0.0013

ExpertD

0.1004

0.6302

0.0633

0.5069

Global

0.1004

0.6114

0.0614

0.4918

(non-opt.)

and90th)ispresentedin

Table

4

(intermediateassessmentshavebeenobtainedbyinterpolatinglinearlytheestimatessummarizedin

Fig.

1

).

ThoughthisoptionisavailableinAY,itisuncleartotheauthorsitsapplicabilityinpracticesincethecomplexityofelic-itingexpertjudgmentsgrowssignificantlywiththenumberofquantilestobeelicitedfromexperts.Itisalsouncleartotheauthorsifnostudyconsideredtheelicitationofmorethan5quantilesbecausethisfeaturewasnotavailableinanysoftwareimplementation.

Missingitemsforsomeexperts

In[

6

]twopanelsof9expertsweregatheredinordertoassessuncertaintyovereconomicgrowthandoilpricesforMexicoin2020and2030.Inthepanelcorrespondingtointernationalgasandoilprices,expertAdidnotanswer10of26calibrationvariables.NoanswerforexpertDwasrecordedfor5calibra-tionvariables.Similarly,noanswerto1calibrationvariablewasobservedforexpertG.TheresultsofcalculationsobtainedwithmissingitemsforbothAYandCCarepresentedin

Table

5

.Similarlyasin

Table

3

,theagreementbetweenthecalculationsobtainedwithCCandAYisalmostexact.

Conclusions

TheMATLABtoolboxnamedAIforcombiningexpertjudg-mentsapplyingCooke’sclassicalmodelforstructuredexpertjudgmenthasbeenextended.ThenewsoftwareiscalledAN-DURYL.ThemainpurposefordevelopingthesetoolboxesistocreateopensourcesolutionsthatcanbeusedbypractitionersandresearcherswhoareinterestedinapplyingordevelopingfurtherCooke’smethod.IncomparisonwithAIand/orCC,AYpresentsthefollowingnewfeatures:

AYhasinheritedalladvantagesofAIdiscussedin[

2

].Ad-ditionally,AYisfullyopensourceandallowsforuserdefinedquantiles(see

4.1

)andmissingitems(see

4.2

).

ThesoftwaretoolpresentedinthispapervalidatesCooke’sclassicalmodelsuccessfullywitharangeofstudiespresentedin[

4

].DespitethelimitationsofthecurrentversionofAY,itistotheauthorsbeliefthatsimilarlyasAIthedevelopedtoolboxwillbevaluabletothosewhoareinterestedindevelopingandfurtherapplyingthemethod.ItistheambitionoftheauthorstoextendAIandAYwithmorefeaturesthanthosecurrentlyavailableinCCandwiththemorerecenttechniquesofelicitationofmultivariatedependence[

7

].

Declarationofcompetinginterest

Wewishtoconfirmthattherearenoknownconflictsofinter-estassociatedwiththispublicationandtherehasbeennosignif-icantfinancialsupportforthisworkthatcouldhaveinfluenceditsoutcome.

Table5

ComparisonofcalculationsfromAYandCCfortheexpertpanelpresentedin[

6

].

ExpertID

Calibration(CC)

Calibration(AY)

Information(CC)

Information(AY)

Information(CC)

Information(AY)

ExpertA

1.634E?7

1.635E?7

1.347

1.347

1.235

1.235

ExpertD

0.07205

0.07209

1.045

1.045

1.004

1.004

ExpertG

0.0004775

0.0004774

1.075

1.0745

1.262

1.262

Global

0.1512

0.1512

0.8549

0.8549

0.8