NLTK learning: classifying and annotating vocabulary

[TOC]

Part-of-speech tagger

Many subsequent tasks require tagged words. nltk comes with its own English taggerpos_tag

import nltk
text = nltk.word_tokenize("And now for something compleyely difference")print(text)print(nltk.pos_tag(text))

annotation corpus

represents the tagged identifier: nltk.tag.str2tuple('word/type')

text = "The/AT grand/JJ is/VBD ."print([nltk.tag.str2tuple(t) for t in text.split()])

Read the annotated corpus

nltk corpus ue navel provides unification interface, you can ignore different file formats. Format: Corpus.tagged_word()/tagged_sents(). Parameters can specify categories and fields

print(nltk.corpus.brown.tagged_words())

Nouns, verbs, adjectives, etc.

Here we take nouns as an example

from nltk.corpus import brown
word_tag = nltk.FreqDist(brown.tagged_words(categories="news"))print([word+'/'+tag for (word,tag)in word_tag if tag.startswith('V')])################下面是查找money的不同标注#################################wsj = brown.tagged_words(categories="news")
cfd = nltk.ConditionalFreqDist(wsj)print(cfd['money'].keys())

Try to find the most frequent noun of each noun type

def findtag(tag_prefix,tagged_text):
    cfd = nltk.ConditionalFreqDist((tag,word) for (word,tag) in tagged_text if tag.startswith(tag_prefix))return dict((tag,list(cfd[tag].keys())[:5]) for tag in cfd.conditions())#数据类型必须转换为list才能进行切片操作tagdict = findtag('NN',nltk.corpus.brown.tagged_words(categories="news"))for tag in sorted(tagdict):print(tag,tagdict[tag])

Explore the annotated corpus

Requirednltk.bigrams() and nltk.trigrams() correspond to the 2-gram model and the 3-gram model respectively.

brown_tagged = brown.tagged_words(categories="learned")
tags = [b[1] for (a,b) in nltk.bigrams(brown_tagged) if a[0]=="often"]
fd = nltk.FreqDist(tags)
fd.tabulate()

Automatic tagging

Default tagger

The simplest tagger is for each identifier Assign uniform tags. Below is a tagger that turns all words into NN. And use evaluate() to test. It facilitates first analysis and improves stability when many words are nouns.

brown_tagged_sents = brown.tagged_sents(categories="news")

raw = 'I do not like eggs and ham, I do not like them Sam I am'tokens = nltk.word_tokenize(raw)
default_tagger = nltk.DefaultTagger('NN')#创建标注器print(default_tagger.tag(tokens)) # 调用tag()方法进行标注print(default_tagger.evaluate(brown_tagged_sents))

Regular expression tagger

Note that the rules here are fixed (determined by yourself). As the rules become more and more complete, the accuracy becomes higher.

patterns = [
    (r'.*ing$','VBG'),
    (r'.*ed$','VBD'),
    (r'.*es$','VBZ'),
    (r'.*','NN')#为了方便，只有少量规则]
regexp_tagger = nltk.RegexpTagger(patterns)
regexp_tagger.evaluate(brown_tagged_sents)

Query annotator

There is a difference between this and the book. It is different from python2. Pay attention to debugging. The query tagger stores the most likely tags, and the backoff parameter can be set. If the tag cannot be marked, use this tagger (this process is backoff)

fd = nltk.FreqDist(brown.words(categories="news"))
cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories="news"))##############################################python2和3的区别#########most_freq_words = fd.most_common(100)
likely_tags = dict((word,cfd[word].max()) for (word,times) in most_freq_words)#######################################################################baseline_tagger = nltk.UnigramTagger(model=likely_tags,backoff=nltk.DefaultTagger('NN'))
baseline_tagger.evaluate(brown_tagged_sents)

N-gram annotation

Basic unary annotator

The behavior of the unary annotator and the search annotator Very similar to the techniques used to build unary annotators, train for .

Here our annotator only memorizes the training set instead of building a general model. The agreement is very good, but it cannot be generalized to new texts.

size = int(len(brown_tagged_sents)*0.9)
train_sents = brown_tagged_sents[:size]
test_sents = brown_tagged_sents[size+1:]
unigram_tagger = nltk.UnigramTagger(train_sents)
unigram_tagger.evaluate(test_sents)

General N-gram tagger

N-gram tagger is to retrieve the word with index= n, and retrieve n-N< ;=index<=n-1 tag. That is, the tag of the current word is further determined through the tag tag of the previous word. Similar to nltk.UnigramTagger(), the built-in binary tagger is: nltk.BigramTagger() The usage is consistent.

Combined Tagger

Many times, an algorithm with wider coverage is more useful than an algorithm with higher accuracy. Use backoff to specify the backoff annotator to implement the combination of annotators. If the parameter cutoff is explicitly declared as int type, contexts that only appear 1-n times will be automatically discarded.

t0 = nltk.DefaultTagger('NN')
t1 = nltk.UnigramTagger(train_sents,backoff=t0)
t2 = nltk.BigramTagger(train_sents,backoff=t1)
t2.evaluate(test_sents)

It can be found that after comparing with the original, the accuracy is significantly improved

Cross-sentence boundary marking

For the beginning of the sentence of words, there are no first n words. Solution: Train the tagger with tagged tagged_sents.

Transformation-based annotation: Brill annotator

is superior to the above. The idea of ​​​​implementation: start with a big stroke, then fix the details, and make detailed changes little by little.
Not only does it take up a small amount of memory, but it is also associated with the context, and corrects errors in real time as the problem becomes smaller, rather than static. Of course, the calls are different in python3 and python2.

from nltk.tag import brill
brill.nltkdemo18plus()
brill.nltkdemo18()

The above is the detailed content of NLTK learning: classifying and annotating vocabulary. For more information, please follow other related articles on the PHP Chinese website!