The HMM is implemented in scripts/hmm.py. It takes about 60 seconds to train the model and decode the development split. Sentence boundaries are marked by introducing an artificial 'start-of-sentence' state ( '-s-') occuring with 'newline' tokens ( '-n-'). A pseudo count alpha > 0 is used as the smoothing parameter, with alpha = 0.001 giving best results on the development split (see results below).įor training and decoding, the input sequences are treated as a continuous sequence of tokens. The transition and emission counts are then converted to proper probability distributions, using additive smoothing to estimate probabilities for transitions/emissions that have not been observed in the training data. To decode the development and test splits, the input sequence is first transformed according to the unknown word rules mentioned above. The second pass uses the transformed training data to collect the bigram transition and emission counts and saves them to a model file. most tokens with the suffix '-ism' are nouns). Any token occurring less than twice in the training data is assigned a special unknown word token based on a few selected morphological idiosyncrasies of common English word classes (e.g. The first pass over the training data generates a fixed list of vocabulary tokens. The HMM is trained on bigram distributions (distributions of pairs of adjacent tokens). finding the most likely sequence of hidden states (POS tags) for previously unseen observations (sentences). The Viterbi algorithm is used for decoding, i.e. The goal of this project was to implement and train a part-of-speech (POS) tagger, as described in 'Speech and Language Processing' (Jurafsky and Martin).Ī hidden Markov model is implemented to estimate the transition and emission probabilities from the training data.

A GitHub repository for this project is available online.

Pos tagging python code