Code mixing: a challenge for language identification in the language of
social media
Barman, Utsab, Das, AmitavaORCID: 0000-0003-3418-463X, Wagner, JoachimORCID: 0000-0002-8290-3849 and Foster, JenniferORCID: 0000-0002-7789-4853
(2014)
Code mixing: a challenge for language identification in the language of
social media.
In: First Workshop on Computational Approaches to Code Switching, 25 Oct 2014, Doha, Qatar.
In social media communication, multilingual speakers often switch between languages, and, in such an environment, automatic language identification becomes
both a necessary and challenging task.
In this paper, we describe our work in
progress on the problem of automatic
language identification for the language
of social media. We describe a new
dataset that we are in the process of creating, which contains Facebook posts and
comments that exhibit code mixing between Bengali, English and Hindi. We
also present some preliminary word-level
language identification experiments using
this dataset. Different techniques are
employed, including a simple unsupervised dictionary-based approach, supervised word-level classification with and
without contextual clues, and sequence labelling using Conditional Random Fields.
We find that the dictionary-based approach
is surpassed by supervised classification
and sequence labelling, and that it is important to take contextual clues into consideration.
Metadata
Item Type:
Conference or Workshop Item (Paper)
Event Type:
Workshop
Refereed:
Yes
Uncontrolled Keywords:
code switching; language identification; natural language processing; social media