跨語(yǔ)言信息檢索技術(shù)

11三月2024跨語(yǔ)言信息檢索技術(shù)RoadMapCrossLingualIRMotivationDefinitionGeneralIssuesWithCLIRBasicApproachestoCLIRCLIRevaluationCLIRapplicationsInformationRetrievalSinglelanguage：boththeuser’squeryanddocumentstobesearchedareinsamelanguage.Crosslanguage：documentswritteninalanguagedifferentfromthelanguageoftheuser'squerydocumentsquery3/11/202432000-2010年世界各大洲網(wǎng)絡(luò)語(yǔ)言使用增長(zhǎng)率(數(shù)據(jù)更新時(shí)間：2010年6月30日)TheInternetBigPictureWorldRegionsPopulationInternetUsersPenetration(%population)Users%ofTableGrowth2000-2015Africa1,158,355,663313,257,07427.0%9.6%6,839%Asia4,032,466,8821,563,208,14338.8%47.8%1,268%Europe821,555,904604,122,38073.5%18.5%475%MiddleEast236,137,235115,823,88249.0%3.5%3,426%NorthAmerica357,172,209313,862,86387.9%9.6%191%LatinAmerica617,776,105333,115,90853.9%10.2%1,743%Oceania/Australia37,157,12027,100,33472.9%0.8%256%WorldTotal7260,621,1183,270,490,58445%100%806%WorldInternetUsersand2015PopulationStats3/11/202443/11/202453/11/20246Usageofcontentlanguagesforwebsites20022015English72%English54.5%German7%Russian5.9%Japanese6%German5.7%Spanish3%Japanese5.0%French3%Spanish4.7%Italian2%French4.1%Dutch2%Portuguese2.6%Chinese2%Chinese2.2%Korean1%Italian2.1%Russian1%Polish1.9%Portuguese1%Turkish1.6%Source:/technologies/overview/content_language/all/research/activities/wcp/stats/intnl.html3/11/20247CrossLanguageIRMotivationInformationunavailabilityinsomelanguagesLanguagebarrierDefinition:Cross-languageinformationretrieval(CLIR)

isasubfieldof

informationretrieval

dealingwithretrievinginformationwritteninalanguagedifferentfromthelanguageoftheuser'squery(wikipedia)Example:AusermayaskqueryinChinesebutretrieverelevantdocumentswritteninEnglish.WhydoweneedCLIRsystems?Needstechnologiesthatenableaccesstoinforegardlessofgeographic/languagebarriers.Tofind,retrieveandunderstandrelevantinformationinwhateverlanguage/form.CLIRhasbecomeoneofthekeyfactorsaffectingknowledgesharingallovertheworld.

GeneralIssuesWithCLIRMultilingualtextaccess(charactersets,etc.)Differencesbetweenlanguages

-stemming,compoundwords,breaksbetweenwords,etc.TermambiguitybetweenlanguagesWhattotranslate(queryvs.document)andhowMatchingstrategiesNotranslation(1)CognatematchingTranslation(2)Querytranslation(3)Documenttranslation(4)Interlingualtechniques3/11/202411Cognatematching(同源匹配）Inthecaseofthemostnaivecognatematching,untranslatabletermssuchaspropernounsortechnicalterminologyareleftunchangedthroughthestageoftranslation.Theunchangedtermcanbeexpectedtomatchsuccessfullywithacorrespondingterminanotherlanguageifthetwolanguageshaveacloselinguisticrelationship.(forexample,generationinEnglishandFrench)Whentwolanguagesareverydifferent,byexploringamethodformeasuringsimilaritybetweentransliterationanditsoriginalword,wemaymakecognatematchingfeasible(音譯）..3/11/202412Querytranslation搜索引擎翻譯系統(tǒng)法語(yǔ)查詢法語(yǔ)文檔結(jié)果中文查詢選擇瀏覽法語(yǔ)文檔集合過(guò)程:將中文查詢翻譯成法語(yǔ)檢索法語(yǔ)文檔集合將檢索結(jié)果翻譯成中文3/11/202413querytranslationQuerytranslationisthemostwidelyusedmatchingstrategyforCLIRduetoitstractability.theretrievalsystemdoesnothavetochangeitsinvertedfilesofindextermsinanywayagainstqueriesinanylanguage.ItislesscomputationallycostlytoprocessthetranslationofaquerythanthatofalargesetofdocumentsChallenge:termambiguity‘queriesareoftenshortandshortqueriesprovidelittlecontextfordisambiguation’Termdisambiguationwillbediscussedlater.3/11/202414查詢翻譯優(yōu)缺點(diǎn)優(yōu)點(diǎn)簡(jiǎn)單容易操作靈活節(jié)約時(shí)間、空間，效率高缺點(diǎn)缺乏上下文對(duì)于短查詢式，翻譯歧義性大3/11/202415Documenttranslation中文查詢法語(yǔ)文檔集合搜索引擎翻譯系統(tǒng)中文文檔集合結(jié)果選擇瀏覽過(guò)程:將整個(gè)法語(yǔ)文檔翻譯成中文文檔直接用中文文檔檢索3/11/202416DocumenttranslationDocumenttranslationhasoppositeadvantagesanddisadvantagesfromquerytranslation.InCLIRexperiments,thisapproachisnotusuallyutilized,andquerytranslationisdominant.However,someresearchershaveusedittotranslatelargesetsofdocumentssincemorevariedcontextwithineachdocumentisavailablefortranslation,whichcanimprovetranslationquality.OardandHackett(1998)reportedthatautomaticmachinetranslationofasetofdocumentsusingacommercialMTsystemoutperformsquerytranslationinanexperimentofCLIRfromGermantoEnglish3/11/202417文檔翻譯優(yōu)缺點(diǎn)優(yōu)點(diǎn)只翻譯一次文檔提供的上下文比較豐富文檔可以線下事先翻譯好缺點(diǎn)翻譯速度慢占用大量空間、時(shí)間，效率低依賴機(jī)器翻譯系統(tǒng)的質(zhì)量3/11/202418查詢翻譯vs.文檔翻譯取決于特定語(yǔ)言資源通常查詢翻譯使用更廣兩種方法都提出了“交互性”挑戰(zhàn)3/11/202419Interlingualapproachanintermediatespaceofsubjectrepresentationintowhichboththequeryandthedocumentsareconvertedisusedtocomparethem.Onetypeofinterlingualapproachistousethe‘‘synsets’’providedinWordNet,whichisawellknownmachine-readablethesaurus.Forexample,Diekema,Oroumchian,Sheridan,andLiddy(1999)employedtheWordNetsynsetnumbersaslanguage-independentrepresentationsforCLIR.Sinceasynsetnumber(label)representingaconceptiscorrespondedtoasetofconcretewordsineachoflanguagessupported(e.g.,EnglishandFrench),itispossiblethataqueryterminthesourcelanguagesislinkedtowordsinthetargetlanguageviathesynsetnumber.3/11/202420TranslationtechniquesDictionary-basedmethodsParallelcorpora-basedmethodUseofWWWresources3/11/202421Dictionary-basedmethodsUsingabilingualMachineReadableDictionary(MRD).mostretrievalsystemsarestillbasedonso-called‘‘bag-of-words’’architectures,inwhichbothquerystatementsanddocumenttextsaredecomposedintoasetofwords(orphrases)throughaprocessofindexing.Thuswecantranslateaqueryeasilybyreplacingeachquerytermwithitstranslationequivalentsappearinginabilingualdictionaryorabilingualtermlist.3/11/202422bilingualdictionary人工構(gòu)建的雙語(yǔ)詞典printedMerriam-Webster'sDictionariesLongmanDictionarieselectronicFreedictat/

Travlangat/

問(wèn)題HastobeprocessedtobereadablebymachineLimitedvocabularyDictionarytranslationsareinherentlyambiguousandaddextraneousinformation機(jī)器自動(dòng)構(gòu)建的詞典稱為機(jī)讀詞典MachineReadableDictionaries(MRD)3/11/202423Termtranslationoilpetroleumprobesurveytakesamples選哪個(gè)翻譯?沒(méi)有翻譯!restraincymbidiumgoeringii分詞錯(cuò)誤oilpetroleumprobesurveytakesamples3/11/202424SomeissuesintermtranslationCompoundwords，forexampleGermandecompositionNoboundarybetweenwords，e.g.ChinesesegmentationSpecializedvocabularynotcontainedinthedictionary，e.g.namedentity3/11/202425ExamplesCompounddecomposition(復(fù)合詞分解)chinesewordsegmentation新西蘭花新西蘭花 NewZealandflowers新西蘭花 freshbroccolis3/11/202426Corpora-basedmethodParallel(雙語(yǔ)平行語(yǔ)料庫(kù))orcomparablecorpora(雙語(yǔ)可比語(yǔ)料庫(kù))areusefulresourcesenablingustoextractbeneficialinformationforCLIR.Forexample,inordertotranslateEnglishqueriesintoSpanish,DavisandDunning(1995)extractedmoderatelyfrequentSpanishtermsfromSpanishdocumentsalignedwithEnglishdocumentswhichhadbeensearchedusinganEnglishquery(sourcequery).3/11/202427ParallelcorporaAparallelcorpus(pl.corpora)isadocumentcollectioncomposedoftwoormoredisjointsubsets,eachwritteninadifferentlanguage,suchthatdocumentsineachsubsetaretranslationsofdocumentsineachothersubset.Veryhighaccuracy3/11/202428象形文字古埃及文字希臘文3/11/202429羅塞塔石碑羅塞塔石碑（RosettaStone，也譯作羅塞達(dá)碑），高1.14米，寬0.73米，是一塊制作于公元前196年的大理石石碑，原本是一塊刻有埃及國(guó)王托勒密五世（PtolemyV）詔書(shū)的石碑。石碑上用希臘文字、古埃及文字和當(dāng)時(shí)的通俗體文字刻了同樣的內(nèi)容。由于這塊石碑刻有三種不同語(yǔ)言版本，使得近代的考古學(xué)家得以有機(jī)會(huì)對(duì)照各語(yǔ)言版本的內(nèi)容后，解讀出已經(jīng)失傳千余年的埃及象形文之意義與結(jié)構(gòu)，而成為今日研究古埃及歷史的重要里程碑。3/11/202430Moreparallelcorporanews:DE-News(German-English)Hong-KongNews,XinhuaNews(Chinese-English)Governmentdocuemtns:Canadian-Hansards(French-English)Europarl(Danish,Dutch,English,Finnish,French,German,Greek,Italian,Portugese,Spanish,Swedish)UNTreaties(Russian,English,Arabic,…)Bible(many,manylanguages)3/11/202431ExamplesEnglishGermanDivergingopinionsaboutplannedtaxreformUnterschiedlicheMeinungenzurgeplantenSteuerreformThediscussionaroundtheenvisagedmajortaxreformcontinues.DieDiskussionumdievorgesehenegrosseSteuerreformdauertan.TheFDPeconomicsexpert,GrafLambsdorff,todaycameoutinfavorofadvancingtheenactmentofsignificantpartsoftheoverhaul,currentlyplannedfor1999.DerFDP-WirtschaftsexperteGrafLambsdorffsprachsichheutedafueraus,wesentlicheTeilederfuer1999geplantenReformvorzuziehen.3/11/202432ComparablecorporaAcomparablecorpusisapairofcorporaintwodifferentlanguages,whichcomefromthesamedomain.TalkingthesametopicParallelsentencesmayalsobeminedfromcomparablecorporasuchasnewsstorieswrittenonthesametopicindifferentlanguages.Someresearchersextractphrasepairsfromcomparablecorporausingaclassifierapproach.3/11/202433Example3/11/202434UseofWWWresourcesTheWWWcanproviderichandubiquitousmachine-readableresources,fromwhichwemaybeabletoautomaticallyextractinformationusefulforCLIR.Forexample,Chen(2002)andChenandGey(2003)madeuseofageneralsearchengineontheInternetandtriedtofindEnglishtranslationequivalentsofChineseorJapaneseterms(mainlypropernouns)byanalyzingcontextsofthesetermsinChineseandJapaneseWebdocumentsreturnedbytheengine.3/11/202435Termdisambiguationtechniques(翻譯歧義性)Disambiguationfromamongmultiplealternativetermtranslations,多個(gè)翻譯如何選擇？e.g.,Apple,BankUseofpart-of-speech(POS)tags.Useofparallelcorpus.Useofco-occurrencestatisticsinthetargetcorpus.Useofthequeryexpansiontechnique.3/11/202436Useofpart-of-speechtagsThebasicideaofusingpart-of-speech(POS)tagsfortranslationdisambiguationistoselectonlytranslationshavingthesamePOSwiththatofthesourcequeryterm.ThismethodrequiresthatPOStaggingsoftwareisavailableforbothlanguages.3/11/202437Parallelcorpus-baseddisambiguationAparallelcorpuswasusedfordeterminingthe‘‘best’’translationorsetoftranslationsbyDavis(1997,1998),whereasingletranslationforeachsourcetermwasselectedfromasetoftranslationslistedinanMRDaccordingtotheresultofsearchingaparallelcorpus.3/11/202438Translationprobability探測(cè)survey試探樣品測(cè)量(p=0.4)(p=0.3)(p=0.25)(p=0.05)多個(gè)翻譯翻譯概率3/11/202439Disambiguationbasedonco-occurrencestatisticsthecorrecttranslationsofquerytermsshouldco-occurintargetlanguagedocumentsandincorrecttranslationsshouldtendnottoco-occur.First,thetwomostrelatedtermsinthequeryweredeterminedbasedoncooccurrencestatisticsinthesourcelanguagecorpus,andthenthe‘‘best’’translationswereselectedfromallpairsoftranslationsofthesetwotermsaccordingtoco-occurrencestatisticsinthetargetlanguagecorpus.Itshouldbenotedthatthesetwocorporadonothavetobeparallelorcomparable.3/11/202440QueryexpansionfordisambiguationPseudorelevancefeedback(PRF),alsoknownasblindfeedback,iswidelyrecognizedasaneffectivetechniqueforenhancingperformanceofinformationretrieval.PRFalsoworkseffectivelyforCLIRtasks.InthecaseofCLIR,twokindsofPRFarefeasible:Pre-translationfeedbackandPost-translationfeedback3/11/202441Pre-translationfeedbackDocumentsfromacorpusinthesourcelanguagecanberetrievedpriortotranslationinordertoaddasetofnewtermstothesourcequery(pre-translationfeedback)ifsuchacorpusisavailable.Pre-translationfeedbackmaycontributetoimprovementofprecision.ThisisduetothefactthatthePRFisbasicallydoneusingtheentirequery––noteachsourcetermrespectively.Thatis,synonymsorrelatedtermscorrespondingtothe‘‘correct’’meaningofeachsourcetermwithinacontextofthequeryareexpectedtobeautomaticallyaddedthroughthePRFprocess.3/11/202442Post-translationfeedbackAftertranslation,standardPRFcanbeappliedusingthetargetdocumentcollection(post-translationfeedback).post-translationfeedbackcanbeconsideredadeviceforimprovingrecallratio,asshowninstandardexperimentsofmonolingualretrieval.InCLIR,twowell-knownmethodsforweightingtermsinthetop-rankeddocumentsareoftenutilizedforselecting‘‘good’’terms,i.e.,theRocchiomethodandtheprobabilisticmethod.3/11/202443bi-directionaltranslationBoughanemetal.(2002),exploreda‘‘bi-directionaltranslation’’techniqueinwhichaformofbackwardtranslationisusedforrankingtranslationcandidates.SupposethatweneedtotranslateEnglishquerytermsintoFrenchones.In‘‘bi-directionaltranslation,’’firstasetofFrenchequivalentsforanEnglishtermisfoundinanEnglish–Frenchdictionary.Next,usingaFrench–Englishdictionary,eachFrenchequivalentisreverselytranslatedintoasetofEnglishterms.Basically,ifthesetincludestheoriginalsourceterm,theFrenchtranslationequivalentischosenasapreferredtranslation.3/11/202444跨語(yǔ)言檢索評(píng)價(jià)信息檢索評(píng)價(jià)給定一個(gè)檢索主題，一個(gè)文檔集合，一些人工判斷好的相關(guān)文獻(xiàn)對(duì)系統(tǒng)返回的檢索結(jié)果進(jìn)行判斷TRECCLIR(96-02):英語(yǔ)到其他語(yǔ)言CLEF(00-):歐洲語(yǔ)言之間NTCIR(99-):亞洲語(yǔ)言與英語(yǔ)3/11/202445跨語(yǔ)言檢索評(píng)價(jià)模型3/11/202446ApplicationsofCLIR472.1CrosslanguageSearchEngineApril25,2006:Europeansearchengine“Quaero”

FrenchPresidentannounced90million-eurosupport.May16,2007:GoogleTranslateProvideCLIRfor12languagesGoal:take"alltheWeb&translateintomultiplelangs."May5,2008:YahooBabelFishProvideCLIRbetween12languagesItwasAltaVista'sproject,laterboughtbyYahoo3/11/202448GoogleTranslate

3/11/2024493/11/202450YahooBabelFish

3/11/2024513/11/2024523/11/202453提問(wèn)請(qǐng)比較Google和Yahoo！的跨語(yǔ)言搜索引擎的區(qū)別，分析各自的優(yōu)缺點(diǎn)Google：一步完成（translate&search），檢索結(jié)果翻譯回源語(yǔ)言。優(yōu)點(diǎn)：快速，便于用戶理解檢索結(jié)果。缺點(diǎn)：用戶無(wú)法修改翻譯。Yahoo！：兩步完成（translate+search），檢索結(jié)果未翻譯。優(yōu)點(diǎn)：有中間步驟，用戶可以修改翻譯。缺點(diǎn)：復(fù)雜，檢索結(jié)果無(wú)法識(shí)別。3/11/2024542.2數(shù)字圖書(shū)館的跨語(yǔ)言檢索2010年6月11日在芬蘭首都赫爾辛基舉行的ICSTI（國(guó)際科技信息理事會(huì)）夏季會(huì)議上發(fā)布的世界科學(xué)跨語(yǔ)言檢索平臺(tái)WorldWideScience3/11/202455WorldWideScience

/multilingual聯(lián)盟的成員單位都是專業(yè)圖書(shū)情報(bào)機(jī)構(gòu)或科技信息事業(yè)的領(lǐng)導(dǎo)機(jī)構(gòu)，如美國(guó)能源部科技信息局（OSTI）、美國(guó)國(guó)會(huì)圖書(shū)館、大英圖書(shū)館、加拿大科技信息研究所、韓國(guó)科技信息研究所、中國(guó)科技信息研究所等。該平臺(tái)還可以自動(dòng)進(jìn)行跨語(yǔ)言跨庫(kù)檢索3/11/202456WorldWideScience

/multilingual3/11/2024572.3跨語(yǔ)言專利檢索根據(jù)世界知識(shí)產(chǎn)權(quán)組織（WorldIntellectualPropertyOrganization,WIPO）報(bào)導(dǎo)，專利文件包含全世界90%～95%的科研成果，而其他技術(shù)文件（論文或期刊等）中只含5%～10%的研發(fā)成果。在研究工作中若能善于利用專利檢索可以縮短60%的研發(fā)時(shí)間，同時(shí)減少40%的研發(fā)經(jīng)費(fèi)。3/11/202458PATENTSCOPE

/patentscope/search/en/clir/clir.jsp2010年5月，世界知識(shí)產(chǎn)權(quán)組織WIPO發(fā)布了跨語(yǔ)言專利檢索系統(tǒng)PATENTSCOPE的測(cè)試版，標(biāo)志著跨語(yǔ)言信息檢索在專利檢索中的應(yīng)用從實(shí)驗(yàn)室走向?qū)嵱没?。該系統(tǒng)只能提供英語(yǔ)、法語(yǔ)、德語(yǔ)、日語(yǔ)、西班牙語(yǔ)5種語(yǔ)言之間的跨語(yǔ)言專利檢索。3/11/202459PATENTSCOPE

/patentscope/search/en/clir/clir.jsp3/11/202460PATENTSCOPE

/patentscope/search/en/clir/clir.jsp3/11/2024612.4跨語(yǔ)言圖像檢索目前，已走向?qū)嵱没目缯Z(yǔ)言圖像檢索的代表是由華盛頓大學(xué)開(kāi)發(fā)的一個(gè)跨語(yǔ)言圖像搜索引擎PanImages（/）PanImages提供100多種語(yǔ)言的翻譯用戶輸入關(guān)鍵字并選擇其隸屬于哪種語(yǔ)言，通過(guò)機(jī)器翻譯將關(guān)鍵詞轉(zhuǎn)換成各個(gè)國(guó)家的語(yǔ)言，將翻譯的關(guān)鍵詞在Google圖片搜索和Flickr圖片搜索中進(jìn)行搜索3/11/202462PanImages

/3/11/202463PanImages

/3/11/2024642.5電子商務(wù)中的應(yīng)用CINDOR是目前比較成功的一個(gè)商業(yè)跨語(yǔ)言信息檢索系統(tǒng)CINDOR系統(tǒng)擁有概念中間語(yǔ)言（ConceptualInterlingua）、語(yǔ)言分析（LanguageAnalysis）、搜索管理（SearchManagement）三大核心技術(shù)。CINDOR目前支持英語(yǔ)、法語(yǔ)、西班牙語(yǔ)，正在研制簡(jiǎn)體中文、俄語(yǔ)、阿拉伯語(yǔ)。3/11/202465CINDOR

/home.html3/11/202466CINDOR

/home.html3/11/202467ReferenceKazuakiKishida.Technicalissuesofcross-languageinformationretrieval:areview.InformationProcessingandManagement.2005(41),pp433-455.葛運(yùn)東;跨語(yǔ)言信息檢索查詢翻譯技術(shù)研究[D];蘇州大學(xué);2010王序文.基于主題偽相關(guān)反饋的跨語(yǔ)言信息檢索技術(shù)研究[D];北京郵電大學(xué),2014彭琳.漢語(yǔ)詞語(yǔ)語(yǔ)義相似度度量及其在跨語(yǔ)言信息檢索中的應(yīng)用研究[D];復(fù)旦大學(xué),20103/11/202468對(duì)“交互”的挑戰(zhàn)CLIRposessomeuniquechallengesforinteractionHowdoyouhelpusersselecttranslatedqueryterms?Howdoyouhelpusersse