← Back to docs

Getting started

Evaluating Qwen2.5-14B-Instruct for Historical French Text Normalization

A human-AI editing comparison

Abstract

This pilot study investigates the ability of a large language model (LLM), Qwen2.5-14B-Instruct, to perform historical French text normalization and linguistic editing.

The experiment compares AI-generated corrections with human expert-style normalization using a small parallel corpus consisting of original historical French texts, human-edited versions, and AI-generated outputs.

This work represents the baseline stage of a larger research direction. The next phase aims to develop a specialized language editing model by collecting larger expert-annotated datasets, incorporating corpus linguistic resources, and applying parameter-efficient fine-tuning techniques such as LoRA.

The long-term objective is to create an AI-assisted linguistic editing system capable of handling historical, academic, and domain-specific language variation.

1. Introduction

Historical documents contain complex spelling variations, grammatical structures, and linguistic forms that change over time. Automatic processing of these documents requires models that can distinguish between:

  • acceptable historical variation
  • spelling inconsistencies
  • genuine grammatical errors
  • unnecessary modernization

Recent advances in generative AI provide new possibilities for language editing and normalization.

This project investigates whether a general-purpose instruction-following model can approximate expert linguistic editing decisions.

2. Related Work

This study is inspired by:

Rubino, R., Gerlach, J., Mutal, J., & Bouillon, P. (2024). Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants. Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3394–3402.

The authors studied automatic normalization of Middle French historical documents from the 16th century.

Their approach focused on reducing spelling variation while preserving historical language characteristics instead of fully modernizing the text.

They created a manually curated parallel corpus (Historical Text → Human Normalization) and trained neural models based on pre-trained architectures.

Their experiments evaluated models using:

  • BLEU
  • chrF
  • TER
  • WER

3. Our Experimental Approach

Unlike the original study, this experiment evaluates:

Model: Qwen2.5-14B-Instruct

The model receives historical French text and generates:

  1. Edited text
  2. Explanation of modifications
  3. Classification of changes

The comparison pipeline:

Original Historical Text → Human Linguistic Editing → AI Generated Editing → Statistical Comparison

4. Dataset Construction

A small evaluation dataset was created from historical French examples.

Each sample contains:

FieldDescription
OriginalHistorical manuscript form
HumanHuman-style normalized version
AIQwen generated correction

Example

Original: Ledictz jour, vendredy 28 octobrix 1547, en l'Evesché

Human: Ledit jour vendredi 28 octobris 1547 en l'Évêché

AI: Le dict jour, vendredi 28 octobre 1547, en l'Évêché

5. Evaluation Metrics

BLEU

BLEU measures similarity between generated text and reference human editing.

Formula:

BLEU = BP × exp(Σ w_n log(p_n))

Higher BLEU indicates closer similarity.

chrF

chrF evaluates character-level similarity. It is useful for historical languages because many differences occur at spelling level.

Formula:

chrF = (1 + β²)PR / (β²P + R)

Word Error Rate (WER)

WER measures word-level editing distance.

Formula:

WER = (Substitutions + Deletions + Insertions) / Reference Words

Lower WER indicates better performance.

6. Statistical Analysis with R

The evaluation data was processed using R.

Example workflow:

library(stringdist)

distance <- stringdist(
  human_text,
  ai_text,
  method = "lv"
)

Metrics are visualized using R-generated charts.

7. Results Visualization

Comparing human normalization against the AI-generated output across the three metrics:

8. Error Analysis

The preliminary evaluation shows several types of AI editing behavior.

Modernization

The model sometimes replaces historical forms with modern equivalents.

Example: H: octobris → AI: octobre

The AI correction is linguistically understandable but changes the historical form.

Lexical Replacement

Example: H: ferratier → AI: ferrailleur

The model selects a modern lexical equivalent.

Structural Modification

Some corrections modify syntax rather than only spelling. This indicates the importance of domain-specific training.

9. Future Development

This experiment represents the first baseline version of the system.

Future research will focus on:

  • Building a larger human-annotated editing corpus
  • Collecting expert linguistic feedback
  • Training Qwen using LoRA adapters
  • Comparing multiple open-source LLMs
  • Developing evaluation datasets for academic and historical language editing

The final objective is to create a specialized AI language editor combining:

  • Generative AI
  • Corpus linguistics
  • Human linguistic expertise
  • Statistical evaluation

Conclusion

This preliminary experiment demonstrates that large language models can perform meaningful historical language editing tasks.

However, comparison with human normalization shows that general-purpose models may over-modernize historical forms.

Future domain adaptation and expert-guided fine-tuning can improve reliability and move AI editing systems closer to expert linguistic performance.