All 3 models give their weakest legal answers in Algeria.
Same questions, same models. Only the country and the language change.
We asked 3 leading AI models the same 77 real legal questions in Algeria, France and the US, each in the local language.
Average score, same 77 questions
Average across GPT 6.1 Sol, Kimi K3.1 Pro, Mistral Large 4. No web search.
All 3 models give their weakest legal answers in Algeria.
Same questions, same models. Only the country and the language change.
of AI answers in Algeria get an essential legal point wrong.
France: 30%. US: 22%. Average across the 3 models.
Mistral Large 4 loses 78% of its score when the question comes from Algeria instead of France.
Drop in average score, France → Algeria, identical questions.
is the best score when answers are checked against a practising Algerian lawyer's own answers.
Judged a second time, with the lawyer's answer as the reference.
what a correct answer costs in Algeria, compared with France or the US.
Kimi K3.1 Pro: more tokens and fewer correct answers.
between the best and the worst model in Algeria. In France, the gap is under 1 point.
In Algeria, choosing the right model matters far more.
is the best AI score on Algerian traffic questions. Algerian law is changing fast, and AI has not caught up.
Same traffic questions in France: best model 4.3/10.
Real questions from citizens, each one answered by a practising Algerian lawyer. We kept the 77 hardest.
Strictly translated. Only the language and the place change, so scores compare like with like.
GPT 6.1 Sol, Kimi K3.1 Pro, Mistral Large 4. No web search, like a citizen typing into a chat app.
Against a reference answer from official legal texts. One essential error caps the score at 3.
Questions per sub-domain.
Every question comes from people asking a lawyer for help. Nothing is invented.
Each reference answer is researched on official legal sources and checked against the latest amendments.
Algerian answers are also scored against the practising lawyer's own answers. The ranking holds.
Translations change only the language and the place. Each one is checked for added details.
Every answer is judged on the same key points. A single essential error caps it at 3/10.
Questions and reference answers are held back to prevent contamination, and shared with labs on request.
Interim results: 77 of 100 planned questions. Differences under about 0.5 points are within noise.
Full questions, reference answers and per-country results.