Sindhi Tokenizer Documentation

Overview & Key Findings

Sindhi (سنڌي) is an Indo-Aryan language spoken by over 35 million people, written in an extended Perso-Arabic alphabet with 52 characters. It features distinctive linguistic properties that pose severe challenges to off-the-shelf NLP tokenizers:

• Phonemic Richness: Four implosive consonants (ٻ, ڄ, ڏ, ڳ), distinctive retroflexes, and aspirated digraphs (ڀ, ٿ, ڇ, ڍ, ڦ, ڳ, ڱ).
• Complex Morphology: Extensive pronominal clitic suffixation where prepositions and pronouns attach directly to verbs and nouns (e.g., ڪتابم "my book", ڏنم "I gave").
• Etymological Diversity: Heavy borrowing from Sanskrit, Arabic, and Persian, requiring subword vocabularies to share roots without fragmenting loan stems.

When evaluated on standard Sindhi corpora, standard frontier models suffer from subword bloat: LLaMA 3.2 fragments Sindhi at 2.896 tokens per word, with 28.5% of characters degenerating into byte-level fallbacks. Meanwhile, existing community Sindhi tokenizers suffer from silent character deletion (e.g., TinyLlama dropping ہ in 24.9% of sentences).

Installation & Quick Start

The tokenizers can be used either through the official Google SentencePiece Python library or via Hugging Face's transformers library.

1. Installation

# Install required runtime dependencies
pip install sentencepiece transformers tokenizers

2. Native SentencePiece Usage (Recommended)

Loading via SentencePiece directly is the most robust and performant interface:

import sentencepiece as spm

# Initialize processor with trained model file
sp = spm.SentencePieceProcessor()
sp.load("tokenizers/custom_unigram_64k/sindhi_unigram_64000.model")

text = "سنڌ جي تاريخ تمام قديم ۽ وسيع ثقافتي ورثي تي مشتمل آهي"

# Encode to subword tokens and IDs
pieces = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)

print("Subword Pieces:", pieces)
# [' سنڌ', ' جي', ' تاريخ', ' تمام', ' قديم', ' ۽', ' وسيع', ' ثقافتي', ' ورثي', ' تي', ' مشتمل', ' آهي']
print("Token IDs:", ids)

# Lossless decode
decoded = sp.decode_pieces(pieces)
assert decoded == text, "Lossless reconstruction guaranteed"

3. Hugging Face Transformers Integration

You can load the published tokenizer directly from the Hugging Face Hub using PreTrainedTokenizerFast:

from transformers import PreTrainedTokenizerFast

# Load from Hugging Face Model Hub
tokenizer = PreTrainedTokenizerFast.from_pretrained("alinawazmahar/sindhi-tokenizer")

text = "خداوند ڪريم کيس اعليٰ ذهانت سان نوازيو هو"
inputs = tokenizer(text, return_tensors="pt")
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
print(tokens)

Interactive Playground

Test Sindhi text segmentations below. Select sample sentences representing different linguistic registers or type arbitrary Sindhi text to compare token counts and subword fertility.

Linguistic Motivation

Tokenization is the foundational layer of any Large Language Model. When a tokenizer is sub-optimally trained for a script, downstream models suffer from two primary penalties:

1. Context Window Degradation

If a language requires 2.90 tokens per word (like Sindhi under LLaMA 3.2) compared to 1.15 in our custom tokenizer, an 8,192-token context window can only fit 2,820 Sindhi words instead of 7,120 words. This 60% loss in effective context window length severely restricts document summarization, retrieval-augmented generation (RAG), and few-shot reasoning.

2. Quadractic Attention Cost

Transformer self-attention scales quadratically with sequence length ($O(N^2)$). A 2.5× increase in token count per word leads to a ~6.25× increase in attention computation during pretraining and inference.

3. Character Deletion and Semantic Drift

Tokenizers with incomplete character coverage or faulty normalization silently drop characters. In Sindhi, stripping the single letter ہ (HEH GOAL, U+06C1) alters word meanings entirely: هي (this) becomes ي, and ڪريو is corrupted.

Tokenizer Inventory

We trained and evaluated four vocabulary capacities of the SentencePiece Unigram architecture to identify the optimal Pareto frontier between vocabulary size and inference memory overhead:

SentencePiece Methodology

Unlike Byte-Pair Encoding (BPE), which iteratively merges frequent character pairs deterministically from the bottom up, the Unigram language model (Kudo, 2018) starts with an oversized vocabulary of candidate substrings and iteratively prunes candidates based on their marginal contribution to the corpus log-likelihood.

During tokenization, the Unigram model segments input strings by maximizing the joint probability under the Viterbi algorithm:

x* = argmax_{x in S(X)} prod_{i=1}^{M} P(x_i)

Where $S(X)$ is the set of all valid segmentations of input $X$, and $P(x_i)$ is the probability of subword token $x_i$. This probabilistic approach provides two key advantages for Sindhi:

1. Linguistically Sound Morphological Boundaries: Inflections, case markers, and clitic suffixes are detached cleanly rather than creating unnatural cross-morpheme concatenations.
2. Subword Regularization: SentencePiece Unigram natively supports sampling alternative segmentations during pretraining, improving downstream neural network robustness to typos and dialectal spelling variants.

Training Configuration

Repository Structure

The project repository is structured cleanly with separated concerns for training, evaluation, data preparation, documentation, and model artifacts:

Corpus Architecture & Preprocessing

The training dataset comprises 1,490,000 deduplicated sentences (~28.4 million words) collected and curated specifically to prevent domain overfitting. The corpus employs a 70 / 15 / 15 register distribution:

Text Normalization Pipeline

Perso-Arabic script in digital media frequently contains encoding ambiguities resulting from inconsistent keyboard layouts (e.g., Arabic Kaf ك vs. Sindhi Kaf ڪ, and Arabic Yeh ي vs. Farsi Yeh ی). The preprocessing pipeline enforces standard Sindhi orthography:

import unicodedata
import re

def normalize_sindhi(text: str) -> str:
    # 1. Unicode NFKC normalization
    text = unicodedata.normalize("NFKC", text)

    # 2. Harmonize common regional character collisions
    REPLACEMENTS = {
        "\u0643": "\u06aa",  # Arabic Kaf -> Sindhi Kaf (ڪ)
        "\u06cc": "\u064a",  # Farsi Yeh -> Arabic/Sindhi Yeh (ي)
        "\u06be": "\u0647",  # Do-chashmi Heh -> Heh (where inappropriate)
        "\u200c": "",        # Remove superfluous Zero-Width Non-Joiners
        "\u200d": "",        # Remove Zero-Width Joiners
    }
    for old, new in REPLACEMENTS.items():
        text = text.replace(old, new)

    # 3. Collapse duplicate whitespace
    text = re.sub(r"[ \t]+", " ", text)
    return text.strip()

Evaluation Methodology

To avoid benchmark bias, evaluation was conducted across four distinct textual registers totaling 40,000 sentences (10,000 sentences per register):

1. Formal Sindhi Prose: Literary publications, historical archives, and academic texts.
2. Conversational / Social Media: Informal Facebook, Twitter/X, and forum posts exhibiting colloquial spelling variations.
3. Technical / Educational: Science, IT, and medical documents containing domain-specific loanwords.
4. FLORES+ Sindhi Benchmark: Standard Wikipedia translation test split from Meta's FLORES+ suite.

Evaluated Metrics

• Subword Fertility (tokens/word) ↓: Ratio of total generated subwords to whitespace-delimited words. Ideal value approaches 1.0.
• Single-Token Word % ↑: Percentage of words in the vocabulary represented as a complete single subword token.
• Out-of-Vocabulary (UNK) Rate % ↓: Percentage of input words that trigger an <unk> token. Must be 0.000%.
• Byte-Fallback Rate % ↓: Percentage of tokens emitted as raw UTF-8 byte pieces (e.g., <0xD8>).
• Round-Trip Character Loss % ↓: Character deletion rate between original string $S$ and $S' = \text{decode}(\text{encode}(S))$ after NFKC canonicalization: $\frac{\text{len}(S) - \text{len}(S')}{\text{len}(S)}$. Evaluated unconditionally across all registers.

Macro Benchmark Matrix

The table below shows the macro-averaged results across all four evaluation registers for all 17 evaluated tokenizers. Use the filter tabs or search input to isolate specific model classes:

* Note: TinyLlama_Sindhi and FahadQazi_BPE display apparent 0.000% UNK rates because both tokenizers silently drop unsupported Unicode characters from the output rather than emitting an explicit UNK symbol. SindhiLM_v2 is architecturally and numerically identical to Qwen2.5 across all metrics; both are listed for completeness.

Key Takeaways from the Matrix

1. Efficiency Sweep at ≥64k: CustomSindhi (64k, 100k, 150k) outperforms all 13 baseline tokenizers across all 4 registers in subword fertility and single-token retention simultaneously. (On byte fallback, Gemma 3 achieves 0.105% vs Custom 64k's 0.138% due to Gemma's 256k multilingual vocabulary, though Custom maintains far superior fertility: 1.147 vs 1.344). At 32k, only TinyLlama_Sindhi records lower fertility on the FLORES+ split, but it carries a fatal character-deletion defect.
2. Drastic Multilingual Fragmentation: LLaMA 3.2 fragments Sindhi at 2.896 tokens per word with 28.5% byte fallback, while BLOOM fragments at 2.627 with 31.7% byte fallback. Our 64k model improves efficiency by 60.4% over LLaMA 3.2.
3. Zero Data Loss: CustomSindhi achieves an absolute 0.0000% character error rate across all tested corpora.

Defect Analysis: Character Deletion in Baselines

During evaluation, automated round-trip consistency tests ($S \to \text{encode} \to \text{decode} \to S'$) uncovered critical silent data-corruption defects in prominent open-source Sindhi tokenizers:

In contrast, CustomSindhi models guarantee 100% character preservation. Every valid Unicode codepoint is either matched by a dedicated subword piece or cleanly decomposed into reversible UTF-8 byte tokens.

Downstream NLP & LLM Impact

Subword fertility directly governs the efficiency and latency of modern transformer architectures:

Reproducibility Guide

All scripts, configurations, and evaluation harnesses required to reproduce our reported findings are provided in the repository:

1. Clone Repository and Set Up Environment

git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -r requirements.txt

2. Retrain SentencePiece Models

# Train 64k flagship model on processed corpus
python training/train.py --vocab_size 64000 --model_type unigram --output_dir tokenizers/custom_unigram_64k/

3. Execute the Full Benchmark Matrix

# Run comparative evaluation across all 17 models and 4 registers
python evaluation/benchmark.py --output results/macro_benchmark_results.json

Paper & Citation

If you utilize this tokenizer, datasets, or benchmark suites in your research, please cite our technical paper (read the complete paper preprint):

@article{nawaz2026sindhitokenizer,
  title={Sindhi Tokenizer: High-Coverage SentencePiece Unigram Subword Tokenization for the Perso-Arabic Script},
  author={Nawaz, Ali and Fareed, Muqadas and Sadaf, Soonh and Sahito, Attaullah},
  journal={arXiv preprint},
  year={2026},
  publisher={Shaikh Ayaz University, Shikarpur},
  url={https://github.com/alinawazcode/sindhi-tokenizer}
}

Frequently Asked Questions

Team & Academic Supervision

This project was conducted as an academic natural language processing research initiative at the Department of Computer Science, Shaikh Ayaz University, Shikarpur.

License & Attribution

The Sindhi Tokenizer codebase, models, and documentation are published under the open-source Apache License 2.0. You are free to use, modify, distribute, and commercialize the tokenizers with appropriate academic attribution.