Tokenization & Part-of-Speech Tagging Evaluation Methods

Regardless of the employed tokenization and POS tagging methods, the evaluation of those methods can be expressed statistically using predefined evaluation metrics. The notion of evaluation in the context of POS tagging means evaluating the performance of a POS tagger by comparing its output to a ground truth or a gold standard. Both the output and gold standard are lists of tokens paired with POS tags. While the tags in the output are predicted by the tagger, in the gold standard, those tags are either assigned by a POS tagger or a human annotator.
In the following two subsections, we explain the evaluation of tokenization and POS tagging and what evaluation metrics we use in this study.

Tokenization evaluation metrics

In NLP pipelines, errors in the tokenization stage have a great impact on the task of POS tagging as it relies on accurate token boundaries to classify each token into its corresponding grammatical category (POS tag). Thus, the integrity of tokenization is critical to ensure optimal performance throughout our NLP pipeline. Therefore, evaluating the output of the tokenization methods helps us understand the mistakes the tokenizers make and enables us to reduce error propagation by detecting errors in the very first stages of the pipeline.

The success of any automated tokenization method is measured by the number of tokens it produces in comparison with the expected tokens number in the ground truth. Thus, we distinguish three three performance states of any tokenization method. Consider a tokenization method $T$ that takes in an input text $S$ , $O$ will be a list of output tokens upon applying $T$ to $S$ , and $N$ is a predefined number of expected tokens. The tokenization task will be considered perfect if $|O| = N$ , under-tokenized (tokens omission) if $|O| < N$ , or over-tokenized (tokens addition) if $|O| > N$ . More formally:

Moreover, for evaluation purposes, we distinguish between two types of tokenization evaluation:

intrinsic evaluation and
extrinsic evaluation.

Within the intrinsic evaluation, we want to evaluate the quality of the tokenization system in isolation from the later stages, POS tagging in our case. The intrinsic evaluation directly measures the tokenization system’s capabilities by comparing it to similar systems. However, there are no specific evaluation metrics made for tokenization evaluation. Nonetheless, treating the tokenization task as a machine translation or a natural language generation task gives us the space to evaluate the tokenization performance using the overlap of n-grams between the generated and the gold standard tokens. We follow the same approach of (Ahmadi 2020) by performing tokenization evaluation using the Bilingual Evaluation Understudy Score (BLEU).

BLEU (Papineni et al. 2002) is an n-gram precision-based measure for measuring the quality of generated texts. It is computed by multiplying the test corpus’s precision score’s geometric mean by an exponential brevity penalty factor, $\rho$ . In the context of machine translation, the brevity penalty factor is used to penalize the system for generating short yet accurate translations. Given a gold standard tokenized text with length $r$ and a tokenization system output with length $c$ , the brevity penalty and the BLEU score are calculated as follows:

(2) $\begin{equation*}\centering\rho =\begin{cases}1 & \text{if } c > r \\exp\left(\frac{1-r}{c}\right) & \text{if } c <= r \\end{cases}\end{equation*}$

(3) $\begin{equation*}\centering\text{BLEU} = \rho \times \exp\left( \sum_{i=1}^{N} w_i \log_{p_i} \right)\end{equation*}$

We use $N=4$ because we want to calculate the metric for 4-grams. The $w_n$ represents the weight for each n-gram; since we are using $N=4$ , the weights will be $w={0.25, 0.25, 0.25, 0.25}$ . The $p_n$ represents the precision for each n-gram. BLEU is an objective measure of the system performance; it quantifies the overlap between the n-grams tokens generated by the tokenizer and those present in the gold standard. However, while being fast and simple to use, BLEU does not consider sentence meaning or structure. In addition, it must not be the only utilized evaluation metric as reported by (Reiter 2018). Therefore, we verify our obtained BLEU score by performing an extrinsic evaluation.

Within the extrinsic evaluation, we evaluate the tokenization system by measuring its impact on our whole NLP pipeline. In our case, the tokenization system’s quality greatly affects the POS tagger’s performance. Therefore, the tokenization correctness can also be determined by examining the F1 and accuracy scores of the POS tagger.

POS tagging evaluation metrics

Several evaluation metrics exist for the POS tagging task, such as accuracy, precision, recall, and F1 score.
Accuracy is the percentage of correctly tagged words in the sample. Precision is the proportion of correctly tagged words out of all words that the tagger labeled with a specific POS tag. The recall is the ratio of correctly tagged words out of all words that actually have that POS tag.
F1 score is an algebraic mix or the harmonic mean of precision and recall. It reflects the system’s robustness by highlighting the high-precision, low-recall, or low-precision high-recall trade-off.
The precision, recall, accuracy, and F1 score can be computed at a document, sentence, or tag level.
A text’s true, contextual interpretation and understanding can be more effectively measured at a higher level instead of a fine-grained level (Jatav et al. 2017; Chiche et al. 2022).

Formally, the metrics can be defined as follows:

(4) $\begin{equation*}\text{Precision } = \frac{TP}{ TP + FP}\end{equation*}$

(5) $\begin{equation*}\text{Recall} = \frac{TP}{ TP + FN}\end{equation*}$

(6) $\begin{equation*}\text{Accuracy} = \frac{TP + TN}{ TP + TN + FP +FN}\end{equation*}$

(7) $\begin{equation*}\text{F1 score} = 2*\frac{Precision * Recall}{ Precision + Recall}\end{equation*}$

TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative samples, respectively.

In multi-class classification scenarios, when we are dealing with training data with class imbalance, it is highly recommended to use macro-averaged metrics over micro. Thus, we mitigate the effect of some classes being over-represented while others are under-represented. The macro-averaged metrics entail calculating the metrics for every class independently and then averaging the results across all classes. For example. in any given textual dataset, the percentage of word tokens with noun tag will be very high, while some other tags such as ADJ might be very low. Therefore, we report the macro-averaged metrics because we want to make sure that each POS tag (class) is treated with equal importance.
In this work, we report only the macro-averaged F1 score and accuracy. Taking the 6 and 7, the averaged metrics for $n$ classes are defined as follows:

(8) $\begin{equation*}\text{Accuracy}_{macro} = \frac{1}{n} \sum{i=1}^{n} \text{Accuracy}_i\end{equation*}$

(9) $\begin{equation*}\text{F1 score}_{macro} = \frac{1}{n} \sum{i=1}^{n} \text{F1 score}_i\end{equation*}$

In addition to the aforementioned metrics, the confusion matrix can offer a comprehensive view of a model’s performance across all classes. It helps reveal correct and incorrect classifications and where exactly they occur. The confusion matrix provides a deep insight into the tagger performance by revealing how well the tagger handles under-represented classes in the dataset.

Tokenization & Part-of-Speech Tagging Evaluation Methods6 min read

Tokenization evaluation metrics

POS tagging evaluation metrics

Bibliography

Leave a Reply Cancel reply