Implantation d’un tri lexical respectant la particularité des noms propres

Abstract

The computational treatment of multilingual language resources (as in the Prolex projet, cf (Grass et al. 2002)) should respect lexical conventions admitted by each language’s native speakers. These conventions may vary from one language to another, as in the case of alphabetical sorting algorithm. This algorithm must take a number of universal as well as language-dependent particularities into account, such as the distinction of upper- and lowercase letters, the sorting bi-directionality (from the left to the right or conversely), the role of diacritics (resulting either in variants of a letter, as é, è and ê in French, or in independent letters, as å in Danish or ą in Polish), the role of punctuation characters, the multi-character letters (as ch or ll in Spanish, or dzs in Hungarian) and the ligatures (as œ in French, or ß in German). We describe a Unicode-based sorting algorithm inspired by (LaBonté 1998) for proper names. In the particular case of the sorting of the proper names, three additional points are to be taken into account : the presence of numerical values (Arab numerals or Roman numerals), the variation of spelling of the ligatures and the permutation in the sorting of the multi-word units. Apart from the word list to be sorted, its input is a language-dependent code table which defines the language’s alphabet, the number of algorithm’s passes, the direction of each pass, and the order of letters or groups of letters in each pass. The implementation of the algorithm is done by a finite-state transducer which allows a fast assignment of sort keys to words. The algorithm proved correct for European languages such as English, French, and Polish, as well as for Thai. It outperforms other sorting algorithms, such as those implemented in Intex (Silberztein 1993) and Unitex (Paumier 2003) systems.