Title: Meaning in a cross-lingual language model

Large Language Models or LLMs have two major components: the vocabulary and the attention model. The vocabulary is derived from the training data by tokenising the textual input and inferring what tokens are most frequent. Vocabularies are limited in size, which necessarily means that words that represent concepts are broken down into smaller parts. If that happens a word looses meaning or gets misrepresented in the model. Wen the model is initialised, the meaning needs to be repaired using the context. For example, my name is not in the vocabulary of an English model and will be broken down into “pie” and “k” and similarly the two where I live “weesp” is broken down in “we” and “esp". Obviously, these tokens obtained very different meanings so the model wil be off after initialisation with my name and my town. Still models can recover that “pie”, “k”, “we” and “esp" actually represent the name of a person and a town in e.g. “My name is Piek and I live in Weesp” but only because the context is clear. This raises the question what happens in a cross-lingual language model in which up to a hundred languages need to share the vocabulary? These models can still transfer very well information from one language to another even though they share tokens with different meanings, such as the tokens “star” and “ster" in English and Dutch, or do not share any token at al, in case of languages with unique scripts. How can these models share associations and attention, how do they transfer conceptual relations, or impose linguistics structures. In this talk, I will present some of our research to answer these questions, which sheds light on the nature of these models as well.

 

In 2013 he won the prestigious Spinoza Award of the Netherlands Organisation for Scientific Research (NWO) and in 2015 he has been honoured by the Dutch Royal House as a “Knight in the Order of the Dutch Lion”. He is also a member of the Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) in three domains and a member of the Koninklijke Hollandsche Maatschappij der Wetenschappen (KHMW).

 

De Amsterdam AI Thesis Awards hebben als doel het bevorderen van excellentie in AI en Data Science van studenten op bachelor- en masterniveau van de in Amsterdam gevestigde Amsterdam-AI universiteitspartners (HvA, UvA en VU). De doelen van de Amsterdam AI Thesis Award zijn:
 
1. Belonen en bepleiten van kwalitatief hoogstaand scriptie werk;
2. Vrouwen en ondervertegenwoordigde minderheden stimuleren en aanmoedigen om hun opleiding voort te zetten;
3. Diversiteit in AI en Data Science aanmoedigen;
4. Amsterdam en het Amsterdamse AI-ecosysteem promoten als innovatie hub door excellente scripties onder de aandacht te brengen.

 

De winnaars zullen hun scriptie werk presenteren tijdens het Highlights Event en hun prijs ontvangen.