But why does AI struggle with Arabic? · light

But why does AI struggle with Arabic?

a joint workshop by Future Data & Intellectual Systems

Artiom Lipinski

researcher at the A.A. Markov Laboratory for Computer Science, Mathematics and Artificial Intelligence · Saint Petersburg State University

Chief Technical Officer at Intellectual Systems

4+ years in AI research and deep learning · 2+ years in Arabic NLP

BSc in Data Analytics, Faculty of Mathematics and Computer Science, SPbU · now an MSc student and teaching faculty member there

Chat

Generative generates text

Pre-trained first trained, then used

Transformer

what is it?

an architecture that generates text autoregressively

= each new word is predicted from all the words written so far

at each step, the model looks at everything written so far and predicts the next word

I like drinking tea.

cycle 1

drinking38%

to24%

playing15%

tea8%

…

cycle 2

tea46%

coffee31%

water12%

juice6%

…

I40 like1424 drink7305 ing278 tea15668 .13

tokens: the pieces of text the model actually reads

the model never sees letters, only numbers

tokenizer: cuts text into tokens and looks up their numbers

⋮

tap15665

task15667

tea15668

teach15669

team15671

⋮

vocabulary: every token the model knows · ≈ 50,000–250,000 entries

I like drinking tea .

the simplest vocabulary: one token = one letter

✗ inefficient: 17 tokens instead of 6, one cycle each

✗ hard to learn: a single letter means nothing

who chooses the pieces?

nobody: the pieces are learned from a lot of text

ten teachers teach ten teens tea

ten

teachers

teach

ten

teens

tea

cycle 1: count neighbouring pairs

t + e×6

e + n×3

e + a×3

c + h×2

cycle 2: count again

te + a×3

te + n×2

a + c×2

c + h×2

cycle 3: ties are broken by rule

c + h×2

tea + c×2

te + n×2

h + e×1

cycle 4: pieces grow

tea + ch×2

te + n×2

ch + e×1

e + n×1

vocabulary

tena chrs

te tea ch teach teacher drink ing

⋮ ~50,000 more

repeat ~50,000 cycles → the vocabulary

task15667

tea15668

teach15669

neighbours by number, strangers in meaning

[0.21−1.340.72…0.05]

embedding: a list of numbers that stores a token’s meaning

where do these numbers come from? → training · part 4

two numbers = a place on a map · a list like this is called a vector

real embeddings: hundreds of numbers · same idea, bigger space

meaning = location · similar meanings sit close together

this arrow is the direction «man → woman»

the same arrow, started from «king», lands on «queen»

[ 3.2 , 1.8 ]

tea

coffee

juice

cat

dog

kitten

king

queen

man

woman

[ 0.4 , 1.6 ]

king − man + woman ≈ queen

this genuinely works in real embeddings

king3.42.8

−

man3.01.2

woman5.01.5

≈

queen5.43.1

«man» is itself a vector: an arrow from the origin to his point

«woman»: the same thing, her own arrow from the origin

− man · 3.4 − 3.0 = 0.4 · 2.8 − 1.2 = 1.6 → the point moves to [ 0.4 , 1.6 ]

+ woman · 0.4 + 5.0 = 5.4 · 1.6 + 1.5 = 3.1 → and lands on queen

tea→[0.21−1.340.72…0.05]

this vector, not the letters, is what actually enters the model

tokenize embed RoPE layers un-embed softmax

I like drinking ⎕

one cycle under the microscope: what happens between the text and the next word?

step 1: the tokenizer splits the text and looks up the ids

step 2: every id is mapped to its embedding; together they form the input matrix

the rows come from E, the embedding matrix: id = row number · learned in training

step 3: RoPE rotates every vector by an angle that encodes its position

step 4: the matrix (one row per token) arrives at the first layer

into layer 1… and out: same rows, new values

again, layer after layer, ×32 · at the end we keep the last row: h

step 5: h times the same matrix E → one score for every token in the vocabulary

one column at a time: multiply the pairs, add them up → one score

the same for every column; the highest score is the model’s best guess → tea

step 6: softmax squashes scores into probabilities · the numbers you saw in part 1

pick a token, append it, run the whole thing again: that is the cycle

I40

like1424

drink7305

ing278

40 1424 7305 278

→→→→

0.120.87−0.330.05⋯

−0.510.220.68−0.09⋯

0.630.90−0.440.18⋯

0.33−0.750.100.59⋯

embedding dimensions · the length of a list of numbers

every id fetches its embedding → a matrix · one vector per token

E · the embedding matrix

row 00.31−0.620.080.77⋯

row 1−0.140.290.91−0.35⋯

⋮⋮

row 1424−0.510.220.68−0.09⋯

⋮⋮

row 50,2560.440.05−0.710.13⋯

50,257 rows · one per token

12,288 numbers per row · GPT‑3

Iposition 1

likeposition 2

drinkposition 3

ingposition 4

layer 1 attention · MLP

layer 2 attention · MLP

⋯
×32

Ilikedrinking

0.120.87−0.330.05

−0.510.220.68−0.09

0.630.90−0.440.18

0.33−0.750.100.59

0.340.51−0.020.44

−0.180.770.25−0.31

0.950.12−0.600.72

0.58−0.220.830.11

1.02−0.350.660.90

0.411.15−0.580.27

−0.120.741.31−0.44

1.70−0.402.200.60

↓

[1.7−0.42.20.6]

h · the last row: «ing»’s vector

1.7−0.42.20.6…

h · 12,288 numbers

teacoffeewaterjuice

…0.21…0.40…0.20…0.30…

…−1.34…−0.80…−0.90…0.60…

…0.72…0.50…0.40…0.15…

…0.05…0.00…−0.30…0.50…

the same E, flipped · one column per token

12,288 rows × 50,257 columns · only a slice shown

tea2.5 coffee2.1 water1.4 juice0.9 …

scores · 50,257 of them

score(tea) = 1.7 · 0.21 + (−0.4) · (−1.34) + 2.2 · 0.72 + 0.6 · 0.05 = 2.5

2.5

softmax(z_i) = e^z_i Σ_j e^z_j

softmax → probabilities

tea46%

coffee31%

water12%

juice6%

I like drinking tea↻

three ingredients

a GPU server

where everything runs

a dataset

billions (nowadays trillions) of tokens

training code

loss function · optimizer · scheduler · …

the model sees

Ilikedrinkingtea.

it should say

Ilikedrinkingtea.

p(next | «I»)

am28%

Oman19%

Apple14%

茶9%

like4%

… and 50,000 more

is 4%should be 100%−log 0.04 = 3.2

is 12%should be 100%−log 0.12 = 2.1

is 88%should be 100%−log 0.88 = 0.1

is 46%should be 100%−log 0.46 = 0.8

is 62%should be 100%−log 0.62 = 0.5

the loss function · cross-entropy

cost = −log p(right answer)

at p = 100% the cost is 0

loss = ( 3.2 + 2.1 + 0.1 + 0.8 + 0.5 ) / 5 ≈ 1.3

0.21−1.340.72

0.120.87−0.33

0.630.900.05

↓

E · the embedding matrix

0.411.15−0.58

−0.120.741.31

↓

inside every one of the 32 layers

…

↓

and the rest

the weights

every number inside the model · GPT‑3: 175 billion of them

loss ≈ 1.3

the gradient: one exact slope per weight

0.210.19

−1.34−1.31

0.720.71

0.050.07

0.870.83

−0.33−0.32

−0.02

+0.03

−0.01

+0.02

−0.04

+0.01

w ← w − lr · gradient

the full math, beautifully told → 3Blue1Brown · «Gradient descent, how neural networks learn» · youtu.be/IHZwWFHWa-w

L(w) = ¼ w⁴ − w² + 0.15 w

L′(w) = w³ − 2w + 0.15

the blue line: the tangent at w = 2

y = L(2) + L′(2) · (w − 2) = 0.3 + 4.15 (w − 2)

w ← w − lr · L′(w) · lr = 0.2

L′(2.00) = 4.15 → step −0.83 → w = 1.17

L′(1.17) = −0.59 → step +0.12 → w = 1.29

L′(1.29) = −0.28 → step +0.06 → w = 1.35

L′(1.35) ≈ 0 → stop

from the dataset, take one sample: this one, and billions like it

shift it by one: every position’s right answer is just the next token

the model sees «I» and scores every token in the vocabulary, sensible or not · the right one got 4%

at every position: how much probability did the model give the right answer?

bad guesses are expensive, good ones almost free; the average is the loss

the IOU from part 2 · nobody writes these numbers: they start random, training must find them

from the loss, compute the gradient · for every weight: which way, and how far

every weight takes a tiny step downhill: the loss shrinks · that is learning

freeze the other 175 billion weights: the loss becomes a curve over one single weight w

the curve is just a formula: what is its derivative?

the derivative L′(w) is a formula too: the exact slope at any point · and it is cheap to compute

the model never sees this curve, it is too expensive to draw: the tangent is all it knows at w = 2

slope 4.15: steep, rising to the right → take a big step left

overshot the bottom: the slope flips sign and pulls us back

slope ≈ 0: settled in a local minimum · a deeper valley exists, we never saw it · that’s fine

every root spawns hundreds of forms: Arabic has ~20× more words than English

but the vocabulary budget goes to English: Arabic gets ~4,000 whole words

the same sentence: Arabic says it in 3 words

…yet the tokenizer shatters them: more tokens, and the pieces are not real morphemes

building your own: balance the tokenizer from day one · the Jais way

or take an open model: add Arabic tokens, then teach it to use them

we trained an Arabic tokenizer and merged it into Qwen3’s vocabulary, in three steps

new tokens need embeddings: place them where they belong, not at random

first, only the embedding matrix learns: the frozen model cannot be damaged

then unfreeze everything: the whole model learns to accept the new tokens

Arabic normally writes no short vowels: the diacritics (حَرَكات) are left out

the same three letters: a flag, knowledge, or a whole verb · the reader decides from context

diacritics are ordinary Unicode characters, woven between the letters

marked sequences are rare, so BPE never merges them: vocalized text shatters

train the tokenizer on marked text and, at best, the mark glues to its letter: 3 tokens where the bare word took 1

one root, one family, but after tokenization the model never sees it

the marks leave the token sequence: they ride inside the embeddings, the way the script rides them on letters

step 1: pin every mark to its letter, then strip them from the text

step 2: fold them into a reserved corner of the embedding · under 200 extra parameters

this rotation gently perturbs the embedding, and that perturbation injects the diacritic’s meaning

the full method, written up: submitted to AAAI

step 3: a tiny head reads the marks back from the hidden states

all diacritics restored in one pass, zero extra tokens, and reading them makes the model better

three ways to get Arabic training data: one found, two synthetic

the web gives real Arabic, but it is scarce and noisy

translation scales infinitely, but the Arabic comes out translated, not spoken

when an answer can be checked, correct reasoning can be farmed, and taught to our model

a scanned page: cursive text, marks above and below the letters

step 1: detect the text lines

step 2: cut the picked line into letters · a dedicated model reads every crop

step 3: boxes that overlap enough (IoU) are chained into a line

step 4: big gaps split the chain into words

step 5: the diacritics arrive by their own separate channel

built with an oil-company partner: route the oil optimally from suppliers to customers

the real thing: the whole network, every pipe colored by its load

zoom in: a computed flow on every single pipe

for a major financial authority: raw cash-register reports become daily price histories

the app: fact vs model for every single series · the days that escape the corridor get flagged

and the countrywide view: a heatmap of where the anomalies concentrate

today

1✓How an LLM works?

1How an LLM generates text

2How an LLM understands words

3Overview of the architecture

4How the model is trained

2✓Why Arabic is hard for LLMs?

1Morphological Complexity

2Orthographic Ambiguity

3Data Scarcity

3✓What else do we do with AI?

1OCR · text recognition

2Flow Distribution

3Price Anomaly Detection

1 · Morphological Complexity

how many words are there? …and inside one vocabulary · Qwen3*, 152k tokens

English

≈ 600,000≈ 70,000 words

Arabic

≈ 12,000,000≈ 4,000 words

rough dictionary estimates: roots × patterns × affixes multiply fast * whole words counted in the Qwen3 tokenizer vocabulary

English

Ilikedrinkingtea.

6 tokens

Arabic

أحبشربالشاي.

8 tokens

tokens per word · English ≈ 1.2 · Arabic ≈ 2.5

how do we fix it?

1train from scratch

English

Arabic

build your own tokenizer: give Arabic as many words as English, then train the model on it

expensive · only if you have the resources

2extend the vocabulary

inject Arabic tokens into an existing model’s vocabulary, then train: introduce the model to them

affordable · builds on an open model

our case · extending Qwen3

1 · initialize the embeddings → 2 · train E only → 3 · train the whole model

tea coffee water king queen الشاي قهوة

a good start is the average of its old pieces: init(الشاي) = mean( E[الش], E[اي] )

E · the embeddings with the new Arabic rows

🔥 training

the rest of the model all the transformer layers

❄️ frozen🔥 training

this exact recipe, run for real

Gulinar

our Arabic model · continuously pretrained and fine‑tuned from Qwen3‑1.7B and Qwen3‑8B

2 · Orthographic Ambiguity

علم three consonants: ʿ‑l‑m · no vowels written · how do you read it?

عَلَم ʿalam a flag

عِلْم ʿilm knowledge, science

عَلِمَ ʿalima he knew

عِلْم · as the computer stores it

عU+0639ain

◌ِU+0650kasra

لU+0644lam

◌ْU+0652sukun

مU+0645meem

plain

علم

1 token

vocalized

ع◌ِل◌ْم

5 tokens

tokenizer trained
on marked text

عِلْم

3 tokens

one root · ع ل م · one family

عِلْمknowledge

ع◌ِل◌ْم

5 pieces

مُعَلِّمteacher

م◌ُع◌َل◌ّ◌ِم

8 pieces

the gold letters are all there, but never together: the root is shredded

our fix · Fold-In

عِلْم splits into two layers

علم the bare text: tokenized as usual · still 1 token

◌ِ ◌ْ the marks: carried inside the embeddings, not in the sequence

◌ِkasraع

◌ْsukunل

noneم

the text itself becomes bare: علم · still 1 token

علم → its token embedding

◌ِ ◌ْ

2,048 numbers, untouched

64 · for marks

the write is gated, starting at zero: the wrapped model begins bit-identical

the letter’s reserved numbers, drawn as an arrow

→

◌ِ the mark turns it by its own learned angle

→

2,048 numbers+

the turned arrow is added into the embedding · the gate starts at 0

a tiny, learned perturbation: the mark’s meaning, injected into the embedding

علم →

the modelone forward pass

→

DiaHeada tiny 2-layer head

→ عِلْم

zero extra tokens, and the letters themselves can never be changed

3 · Data Scarcity

1mine the webfound data

2translate from Englishsynthetic

3ask a teacher modelsynthetic

1mine the web

scrape websites forums, news, blogs: Arabic as people actually write it

ready-made datasets collected and cleaned by the community

after cleaning, the entire Arabic web is ≈ 100B tokens; frontier models eat trillions

+ natural, real Arabic − a lot of garbage · and there simply isn’t much of it

2translate from English

I like drinking tea. →machine translation أحبُّ شربَ الشاي.

+ any English data can be translated: unlimited scale − unnatural · meaning drifts · sounds robotic

3ask a teacher model

checkable tasksmath, code, quizzes: the answer can be verified

→

teacher modela big LLM tries each task many times

→

verifier ✓keep only the attempts that check out, with their reasoning

→

our modeltrain on the kept reasoning and solutions

+ hard tasks, correct answers, full chains of thought − only works when the answer is checkable

0extra tokens generated

1.000text fidelity: the words cannot be corrupted

8.4%diacritic error: beats every open LLM we measured

−15%perplexity when the marks are folded in: the model truly reads them

case 1 · OCR: how it reads a page

القِرَاءَةُ نُورٌ وَالعِلْمُ كَنْزٌ

أُحِبُّ شُرْبَ الشَّايِ كُلَّ صَبَاحٍ

الخَطُّ العَرَبِيُّ فَنٌّ جَمِيلٌ

تَتَدَفَّقُ الأَنْهَارُ نَحْوَ البَحْرِ

ح →

letter modela small net, trained on millions of letter crops

→ ح · 0.97

IoU = overlap / unionboxes that overlap enough belong together: chain them in order

أحبشربالشاي

◌ُ ◌ِ ◌ُّ ◌ُ ◌ْ ◌َ ◌َّ

case 2 · flow distribution: oil in pipelines

network load right now customer demand pipe capacities supplier production

case 3 · price anomalies · for a major financial authority

a desktop app that watches consumer prices across 85 regions × 32 products, and flags the days that look wrong

1 · importthe user drops in raw cash-register price reports · no technical setup at all

→

2 · build the seriesthe app turns them into one clean daily price history for every region × product pair

→

3 · learn & flageach series learns its own normal (level, trend, seasonality) and the days that escape it get flagged

the extra tell: flags that cluster on reporting days hint at manipulated reporting, not real price moves

price series: fact vs model with anomalies

What else?

Contact centers

Knowledge-base RAG chatbotCall analytics & transcriptionAutomated QA scoringReal-time agent assist

Fintech

Regulatory RAG assistantKYC & document automationBehavioral fraud scoringAlternative-data credit scoring

E-commerce & retail

Recommendation engineSemantic catalog searchDemand forecasting & inventoryDynamic pricing

Legal & documents

Contract clause extractionLegal RAG assistantTender documentation parserCompliance checker

Logistics & supply chain

Demand forecastingRoute optimizationTransport-document NLPPredictive fleet maintenance

EdTech

Course-material AI assistantLearning-path recommendationsAutomated short-answer gradingOnboarding knowledge AI

Computer vision

OCR & document digitizationVideo analytics & safety monitoringDefect detection on the lineImage search & tagging

Healthcare

Medical document structuringPatient-flow forecastingInsurance claims processingDiagnostic decision support

∞

Everything.

wherever there is data, there is a model to build

Technical challenges create interesting opportunities

we are ready to share our experience and knowledge with Oman

key takeaways

1an LLM does one thing: it guesses the next token · trained by being wrong billions of times

2Arabic demands language-first engineering: from the tokenizer up

3with creativity and research drive, there is no field AI cannot enter

Artiom Lipinski

CTO & Founder, Intellectual Systems

+7 952 284 21 77

contact@intellectual-systems.com

intellectual-systems.com

all the materials & contact information

Amna AlSinani

CEO & Founder, Future Data

amna@futuredata.me

+968 9989 9420

futuredata.me

→ advance · ← back · F fullscreen

But why does AI struggle with Arabic?

0 / 7