Normalization vs. Standardization in Amharic NLP

In modern spoken Amharic, several historical consonants have merged. The voiceless glottal and pharyngeal fricatives have collapsed into [h], written with ሀ, ሐ, ኀ, and ኸ. [s] can be spelled ሰ or ሠ. The glottal stop and voiced pharyngeal fricative ([ʔ] / [ʕ]) have merged, written አ and ዐ. The ejective affricate and lateral ([tsʼ] / [ɬʼ]) have collapsed into ጸ and ፀ.

Preprocessing pipelines must decide whether to normalize these homophones into one glyph or preserve the spelling. The choice changes vocabulary size, tokenization and semantic accuracy.

The Four Homophone Families

Homophone Set	Ethiopic Characters	Historical Proto-Semitic / Ge’ez Phone	Modern Amharic Realization	Common Canonical Target in IR
The H-Family	ሀ (Haläha), ሐ (Hawt), ኀ (Harm), ኸ (Kaf)	Voiceless glottal [h], voiceless pharyngeal [ħ], voiceless velar [x]	Glottal fricative `[h]` / Velar fricative `[x]`	ሀ
The S-Family	ሰ (Esat), ሠ (Nəguś)	Alveolar fricative [s], lateral fricative [ɬ] / [ś]	Voiceless alveolar fricative `[s]`	ሰ
The A-Family	አ (ʔAlf), ዐ (ʕAyn)	Glottal stop [ʔ], voiced pharyngeal fricative [ʕ]	Glottal stop / null onset `[ʔ]`	አ
The Ts-Family	ጸ (Ts’ädäy), ፀ (Ts’äp’p’a)	Ejective alveolar affricate [tsʼ], ejective lateral [ɬʼ]	Ejective alveolar affricate `[tsʼ]`	ጸ

Note on ኸ (Kaf): ኸ developed as a spirantized form of ከ (/k/ → /x/) and is preserved in northern dialects and loanwords. Information Retrieval (IR) pipelines often normalize it to ሀ. Other NLP pipelines treat it as a distinct velar fricative.

1. The Case for Normalization

Without normalization, a subword tokenizer (BPE or WordPiece) allocates separate vocabulary slots to all seven orders of every homophonic variant.

Informal text also varies in spelling:

ሰላም vs. ሠላም (peace)
ፀሐይ vs. ጸሀይ vs. ፀሀይ vs. ጸሐይ (sun)

A strict string match for ሰላም misses documents indexed under ሠላም.

2. The Case for Standardization

Consonants are root radicals in Semitic languages, so flattening homophones can turn words from different roots into homographs:

Word A (Standard Orthography)	Meaning A	Word B (Homophone Variant)	Meaning B	Result Under Naive Normalization
ድህነት (dəhənät, root d-h-n)	Poverty / destitution	ድኅነት (dəḫənät, root d-ḫ-n)	Salvation / redemption	Both collapse to `ድህነት` (Salvation = Poverty)
ሰረቀ (särräq’ä, root s-r-q’)	He stole / robbed	ሠረቀ (śärräq’ä, root ś-r-q’)	It dawned / rose / emerged	Both collapse to `ሰረቀ` (Sunrise = Theft)
አረፈ (ʔäräfä, root ʔ-r-f)	He rested / took a break	ዐረፈ (ʕärräfä, root ʕ-r-f)	He passed away / died	Both collapse to `አረፈ` (Death = Resting)

Translating የሰው ልጅ ድኅነት (“the salvation of mankind”) after naive normalization gives: "The poverty of mankind."

3. Preprocessing by Task

NLP Task	Recommended Strategy	Rationale
Information Retrieval & RAG	Full Normalization	Maximizes recall across diverse user queries and uncurated web indices.
Neural Machine Translation (NMT)	Strict Standardization	Preserves root semantics to prevent mistranslations.
Automatic Speech Recognition (ASR)	Training Normalization	Spoken phones are identical; penalizing an acoustic model for predicting `ሰ` instead of `ሠ` adds artificial loss noise.
Text-to-Speech (TTS)	Orthographically Agnostic	TTS maps both `ሰ` and `ሠ` to phoneme `/s/`; text normalization focuses instead on number expansion and punctuation.
Instruction-Tuned LLMs	Contextual Standardization	Legal, literary and professional output should keep standard orthography.

4. Configurable Normalization in Python

import re
import unicodedata
from enum import Enum

class NormalizationMode(str, Enum):
    STRICT = "strict"      # Punctuation & whitespace only; preserves etymological homophones
    STANDARD = "standard"  # Normalizes S, A, and Ts families; preserves velar ኸ
    AGGRESSIVE = "aggressive"  # Full collapse for high-recall IR & search indexing

class AmharicTextProcessor:
    def __init__(self, mode: NormalizationMode = NormalizationMode.STANDARD):
        self.mode = mode
        
        # Canonical homophone mapping
        self.s_family = {'ሠ': 'ሰ', 'ሡ': 'ሱ', 'ሢ': 'ሲ', 'ሣ': 'ሳ', 'ሤ': 'ሴ', 'ሥ': 'ስ', 'ሦ': 'ሶ', 'ሧ': 'ሷ'}
        self.a_family = {'ዐ': 'አ', 'ዑ': 'ኡ', 'ዒ': 'ኢ', 'ዓ': 'ኣ', 'ዔ': 'ኤ', 'ዕ': 'እ', 'ዖ': 'ኦ'}
        self.ts_family = {'ፀ': 'ጸ', 'ፁ': 'ጹ', 'ፂ': 'ጺ', 'ፃ': 'ጻ', 'ፄ': 'ጼ', 'ፅ': 'ጽ', 'ፆ': 'ጾ'}
        
        # H-family sub-mappings
        self.h_primary = {
            'ሐ': 'ሀ', 'ሑ': 'ሁ', 'ሒ': 'ሂ', 'ሓ': 'ሃ', 'ሔ': 'ሄ', 'ሕ': 'ህ', 'ሖ': 'ሆ',
            'ኀ': 'ሀ', 'ኁ': 'ሁ', 'ኂ': 'ሂ', 'ኃ': 'ሃ', 'ኄ': 'ሄ', 'ኅ': 'ህ', 'ኆ': 'ሆ',
        }
        self.h_velar = {
            'ኸ': 'ሀ', 'ኹ': 'ሁ', 'ኺ': 'ሂ', 'ኻ': 'ሃ', 'ኼ': 'ሄ', 'ኽ': 'ህ', 'ኾ': 'ሆ'
        }
        
        self.punct_map = {
            '፡': ' ',   # Word divider -> space
            '።': '.',   # Full stop
            '፣': ',',   # Comma
            '፤': ';',   # Semicolon
            '፥': ':',   # Colon
            '፦': ':-',  # Preface colon
            '፧': '?',   # Question mark
            '፨': '\n\n' # Section break
        }

    def process(self, text: str) -> str:
        if not text:
            return ""

        # Step 1: Unicode NFC normalization
        text = unicodedata.normalize("NFC", text)

        # Step 2: Punctuation normalization
        for src, target in self.punct_map.items():
            text = text.replace(src, target)

        # Step 3: Conditional homophone resolution
        if self.mode != NormalizationMode.STRICT:
            # Apply S, A, and Ts unifications
            for src, target in {**self.s_family, **self.a_family, **self.ts_family}.items():
                text = text.replace(src, target)
            
            # Apply primary H unification (ሐ, ኀ -> ሀ)
            for src, target in self.h_primary.items():
                text = text.replace(src, target)
                
            # If aggressive IR mode, also collapse velar ኸ
            if self.mode == NormalizationMode.AGGRESSIVE:
                for src, target in self.h_velar.items():
                    text = text.replace(src, target)

        # Step 4: Whitespace cleanup
        text = re.sub(r'[ ]+', ' ', text)
        return text.strip()

# Verification demonstration:
processor_ir = AmharicTextProcessor(mode=NormalizationMode.AGGRESSIVE)
processor_mt = AmharicTextProcessor(mode=NormalizationMode.STRICT)

sample = "ፀሐይ፡በምሥራቅ፡ሠረቀች።ድኅነት፡የነፍስ፡ዋስትና፡ነው።"

print("Original:   ", sample)
print("Strict (MT):", processor_mt.process(sample))
print("Aggressive: ", processor_ir.process(sample))