Link Search Menu Expand Document


An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation.

Paper Data Code Pretrained Models

Table of contents
  1. Cite
  2. Cite Single Resources
  3. License


Transformer-based architectures brought a breeze of change to Word Sense Disambiguation (WSD) improving models’ performances by a large margin. The fast development of new approaches has been further encouraged by a well-framed evaluation suite for English, which allowed to keep track and fairly compare their performances. However, other languages remained mostly unexplored as testing data are available for a few languages only and the evaluation setting is rather matted. In this paper, we untangle this situation by proposing XL-WSD, a cross-lingual evaluation benchmark for the WSD task featuring sense-annotated development and test sets in 18 languages from six different linguistic families, together with language-specific silver training data. We leverage XL-WSD datasets to conduct an extensive evaluation of neural and knowledge-based approaches, including the most recent multilingual language models. Results show that the zero-shot knowledge transfer across languages is a promising research direction within the WSD field, especially when considering low-resourced languages where large pretrained multilingual models still perform poorly.


  title={ {XL-WSD}: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation.},
  author={Pasini, Tommaso and Raganato, Alessandro and Navigli, Roberto},
  booktitle={Proc. of AAAI},

Cite Single Resources

  • English: WordNet \cite{miller-etal-1990-wordnet}, SemCor \cite{miller-etal-93-semcor}, English WSD Framework \cite{raganato-etal-2017-word}, Senseval-2 \cite{edmonds-cotton-2001-senseval2}, Senseval-3 \cite{snyder-palmer-2004-senseval3}, SemEval-2007 Task 17 \cite{pradhanetal-etal-2007-semeval2007}, SemEval-2007 Coarse Task 7 \cite{navigli-etal-2007-coarse}, SemEval-2010 Task 17 \cite{agirre-etal-2010-domain}, SemEval-2013 Task 12 \cite{navigli-etal-2013-semeval2013} and SemEval-2015 Task 13 \cite{moro-navigli-2015-semeval2015}.
  • Basque: WordNet \cite{pociello-etal-2008-wnterm},
  • Bulgarian: WordNet \cite{simovandpetya-2010-wordnet-bg},
  • Catalan: WordNet \cite{benitez-et-al-1998-wordnet-ca},
  • Chinese: WordNet \cite{huangetal-2014-wordnet-zh},
  • Croatian: WordNet \cite{raffaelli-etal-2008-wordnet-ch},
  • Danish: WordNet \cite{pedersen-et-al-2009-wordnet-da},
  • Dutch: WordNet \cite{postma-et-al-2016-wordnet-nl},
  • Estonian: WordNet \cite{vider-and-orav-2002-wordnet-et},
  • French: SemEval-2013 Task 12 \cite{navigli-etal-2013-semeval2013},
  • Galician: WordNet \cite{guinovart-2011-wordnet-eu},
  • German: SemEval-2013 Task 12 \cite{navigli-etal-2013-semeval2013},
  • Hungarian: WordNet \cite{mihaltz-et-al-2008-wordnet-hu},
  • Italian: SemEval-2013 Task 12 \cite{navigli-etal-2013-semeval2013} and SemEval-2015 Task 13 \cite{moro-navigli-2015-semeval2015}
  • Japanese: WordNet \cite{isahara-et-al-2008-wordnet-ja},
  • Korean: WordNet \cite{yoon-et-al-2009-wordnet-ko},
  • Slovenian: WordNet \cite{fiser-etal-2012-wordnet-sl},
  • Spanish: SemEval-2013 Task 12 \cite{navigli-etal-2013-semeval2013} and SemEval-2015 Task 13 \cite{moro-navigli-2015-semeval2015}.
  title = "{WNTERM}: {E}nriching the {MCR} with a {T}erminological {D}ictionary",
  author = "Pociello, Eli and Gurrutxaga, Antton and Agirre, Eneko and Aldezabal, Izaskun and Rigau, German",
  booktitle = "Proc. of the Sixth International Conference on Language Resources and Evaluation",
  month = may,
  year = 2008
  author = "Kiril Simov and Petya Osenova",
  title ="{C}onstructing of an {O}ntology-based {L}exicon for {B}ulgarian",
  booktitle = "Proc. of LREC",
  year = "2010",
  isbn = 2-9517408-6-7
  title="{M}ethods and tools for building the {C}atalan {W}ord{N}et",
  author="Ben{\'\i}tez, Laura and Cervell, Sergi and Escudero, Gerard and L{\'o}pez, M{\`o}nica and Rigau, German and Taul{\'e}, Mariona",
  journal="Proc. of ELRA Workshop on Language Resources for European Minority Languages",
  author = "Huang, Chu-Ren and Hsieh, Shu-Kai and Hong, Jia-Fei and Chen, Yun-Zhu and Su, I-Li and Chen, Yong-Xiang and Huang, Sheng-Wei",
  title = "{C}hinese {W}ord{N}et: {D}esign, {I}mplementation and {A}pplication of an {I}nfrastructure for {C}ross-{L}ingual
 {K}nowledge {P}rocessing",
  year = 2010,
  journal = "Journal of Chinese Information Processing",
  volume = 24,
  number = 2,
  eid = 14
  title="{B}uilding {C}roatian {W}ord{N}et",
  author="Raffaelli, Ida and Tadi{\'c}, Marko and Bekavac, Bo{\v{z}}o and Agi{\'c}, {\v{Z}}eljko",
  booktitle="Fourth global wordnet conference (gwc 2008)",
  title="{D}an{N}et: the challenge of compiling a wordnet for {D}anish by reusing a monolingual dictionary",
  author="Bolette S. Pedersen and Sanni Nimb and J{\o}rg Asmussen and Nicolai Hartvig S{\o}rensen and Lars Trap-Jensen and Henrik Lorentzen",
  journal="Language Resources and Evaluation",
  title="{O}pen {D}utch {W}ord{N}et",
  author="Postma, Marten and van Miltenburg, Emiel and Segers, Roxane and Schoen, Anneleen and Vossen, Piek",
  booktitle="Proc. of the Eight Global Wordnet Conference",
  author="Vider, Kadri and Orav, Heili",
  title="{E}stonian {W}ord{N}et and {L}exicography",
  booktitle="Proc. of the Eleventh International Symposium on Lexicography",
  title="{G}alnet: {W}ord{N}et 3.0 do galego",
  author="Guinovart, Xavier G{\'o}mez",
  title = "{M}ethods and {R}esults of the {H}ungarian {W}ord{N}et {P}roject",
  author = "M{\'{a}}rton Mih{\'{a}}ltz and Csaba Hatvani and Judit Kuti and Gy{\"{o}}rgy Szarvas and J{\'{a}}nos Csirik and G{\'{a}}bor Pr{\'{o}}sz{\'{e}}ky and Tam{\'{a}}s V{\'{a}}radi",
  booktitle = "Proc. of The Fourth Global WordNet Conference",
  year = 2008
  author = "Hitoshi Isahara and Francis Bond and Kiyotaka Uchimoto and Masao Utiyama and Kyoko Kanzaki",
  title = "{D}evelopment of the {J}apanese {W}ord{N}et",
  booktitle = "Sixth International conference on Language Resources and Evaluation",
  year = 2008
  title="{C}onstruction of {K}orean {W}ord{N}et",
  author="Yoon, Ae-Sun and Hwang, Soon-Hee and Lee, Eun-Ryoung and Kwon, Hyuk-Chul",
  journal="Journal of KIISE: Software and Applications",
  author = "Fi{\v{s}}er, Darja and Novak, Jernej and Erjavec, Toma{\v{z}}",
  title = "{S}lo{WN}et 3.0: development, extension and cleaning",
  booktitle = "Proc. of 6th International Global Wordnet Conference",
  year = 2012
    title = "{S}em{E}val-2010 Task 17: All-Words Word Sense Disambiguation on a Specific Domain",
    author = "Agirre, Eneko and Lopez de Lacalle, Oier and Fellbaum, Christiane and Hsieh, Shu-Kai and Tesconi, Maurizio and Monachini, Monica and Vossen, Piek and Segers, Roxanne",
    booktitle = "Proc. of SemEval",
    month = jul,
    year = "2010"
    title = "{S}em{E}val-2007 Task 07: Coarse-Grained {E}nglish All-Words Task",
    author = "Navigli, Roberto and Litkowski, Kenneth C. and Hargraves, Orin",
    booktitle = "Proc. of SemEval",
    month = jun,
    year = "2007"
    title = "{S}em{E}val-2015 Task 13: Multilingual All-Words Sense Disambiguation and Entity Linking",
    author = "Moro, Andrea and Navigli, Roberto",
    booktitle = "Proc. of SemEval",
    month = jun,
    year = "2015"
    title = "{SENSEVAL}-2: Overview",
    author = "Edmonds, Philip and Cotton, Scott",
    booktitle = "Proc. of {SENSEVAL}-2",
    month = jul,
    year = "2001"
    title = "{S}em{E}val-2013 Task 12: Multilingual Word Sense Disambiguation",
    author = "Navigli, Roberto  and Jurgens, David and Vannella, Daniele",
    booktitle = "Proc. of SemEval",
    month = jun,
    year = "2013"
    title = "The {E}nglish all-words task",
    author = "Snyder, Benjamin and Palmer, Martha",
    booktitle = "Proc. of Senseval",
    month = jul,
    year = "2004",
    title = "{S}em{E}val-2007 Task-17: {E}nglish Lexical Sample, {SRL} and All Words",
    author = "Pradhan, Sameer  and
      Loper, Edward  and
      Dligach, Dmitriy  and
      Palmer, Martha",
    booktitle = "Proc. of SemEval",
    month = jun,
    year = "2007",
    title = "Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison",
    author = "Raganato, Alessandro and Camacho-Collados, Jose and Navigli, Roberto",
    booktitle = "Proc. of EACL",
    month = apr,
    year = "2017",
  title="A semantic concordance",
  author="Miller, George A and Leacock, Claudia and Tengi, Randee and Bunker, Ross T",
  booktitle="Proc. of the workshop on Human Language Technology",
  author = "George A. Miller and R.T. Beckwith and Christiane D. Fellbaum and D. Gross and K. Miller",
  title = "Introduction to {W}ord{N}et: an Online Lexical Database",
  journal = "International Journal of Lexicography",
  year = "1990",
  volume = "3",
  number = "4", 


XL-WSD is distributed under a non-commercial license. This is a human-readable summary of (and not a substitute for) the license (you should read the full terms and conditions of the XL-WSD license before using the material).

If You belong to a research institution, You are free to

  • Share — copy and redistribute the material in any medium or format by making it available only to research institutions (alternatively to sharing the offline indices under the above conditions, and much simpler, a link to the BabelNet website can be provided for download of the official data).
  • Adapt — remix, transform, and build upon the material, provided that you make explicit that what you release contains or “is a processed version of XL-WSD [APPROPRIATE_VERSION_HERE, e.g. v1] downloaded from, made available with the XL-WSD license (see”. Alternatively, a link to the XL-WSD website can be provided for download of the official data and code for the creation of the Adaptation can be provided to any user. XL-WSD is distributed by a CC BY-NC Licence. Please refer to the full license for read all terms.