[ skip to content ]

A rule based Tibetan part-of-speech (POS) tagger for the creation of gold standard training data

Garrett, Edward and Hill, Nathan W. (2017) A rule based Tibetan part-of-speech (POS) tagger for the creation of gold standard training data. [Datasets]

[img] Archive - Other
Available under License Creative Commons Attribution 4.0 (CC-BY 4.0).

Download (42kB)
Alternative Location: http://doi.org/10.5281/zenodo.574882

Abstract

This rule based Tibetan part-of-speech (POS) tagger was prepared in the course of the research project 'Tibetan in Digital Communication' (2012-2015) hosted at SOAS, University of London and funded by the UK's Arts and Humanities Research Council (grant code: AH/J00152X/1). For a description of the tag set see Garrett et al. 2014. and Garrett et al. 2015. For a description of the tagger itself see Garrett et al. 2014. Note that the tagger must be used together with a lexicon (for example Hill & Garrett 2017a). One must use one's own script to tag all words with all tags in the lexicon and then apply the tagger to remove incorrect tags. On the associated corpus of 318,230 words (Hill & Garrett 2017b) the lexical tagger (i.e. simply applying all available tags to all words) tags 141,911 words with the correct unique tag, achieves as accuracy of 1.000 (by definition getting the right tag among others for each word) with an ambiguity of 2.73111. In contrast, the Rule Tagger tags 241,256 words with the correct unique tag, achieves an accuracy of 0.99893 and an ambiguity of 1.38577. Because this tagger does not achieve ambiguity 1.000 it is not suitable for tagging large scale corpora, but instead is useful for the creation of gold standard training data. N.B. In some rare cases the tagger removes all POS-tags.

Item Type: Datasets
SOAS Departments & Centres: Departments and Subunits > School of History, Religions & Philosophies > Department of Religions & Philosophies
Departments and Subunits > Department of East Asian Languages & Cultures
Legacy Departments > Faculty of Languages and Cultures > Department of the Languages and Cultures of China and Inner Asia
Legacy Departments > Faculty of Languages and Cultures > Department of Linguistics
Subjects: P Language and Literature > PI Oriental languages and literatures
P Language and Literature > PL Languages and literatures of Eastern Asia, Africa, Oceania
Depositing User: Nathan Hill
Date Deposited: 16 Jun 2017 08:21
URI: http://eprints.soas.ac.uk/id/eprint/24155

Statistics

Download activity - last 12 months
Downloads since deposit
12Downloads
97Hits
Accesses by country - last 12 months
Accesses by referrer - last 12 months
Additional statistics for this record are available via IRStats2

Repository staff only

Edit Item Edit Item