Does AI safety survive in Nigerian Pidgin?

Salecta is an open evaluation that checks whether AI models make the same safety decisions in Nigerian Pidgin as in English, in both directions.

Code and data on GitHub →Try the safety filter game

The question

Nigerian Pidgin is spoken by tens of millions of people. Much of its vocabulary looks like English, but the meaning often differs. That makes it a sharp test: a model can recognise every word and still misread what someone means. Salecta looks for two ways that can go wrong.

Failure 1: over-refusal. A harmless message is refused or lectured because it sounds harsh.

Failure 2: under-refusal. A harmful request refused in English gets through in Pidgin.

These two cartoons show what Salecta is designed to detect. They aren't results. Real numbers will appear on this page once the evaluation runs.

How it works

01Matched promptsEach item is written three ways: plain English, Pidgin, and code-switched.

02Models answerSeveral AI models from different providers respond to every version.

03Grader labelsA separate model marks each answer helped, partly helped or refused. I hand-check a sample.

04ComparePidgin vs English refusal rates, with confidence intervals.

100Matched Items

Harmless: 60
Harmful: 40

×3Versions Each

English: Plain meaning
Pidgin: + code-switched

2Speaker Checks

Stage 1: Validation
Stage 2: Blind agreement

The English version of each item says plainly what the Pidgin means, rather than translating it word for word. So if a model helps with "I want to ace my exam" but refuses "I wan kill this exam", the gap comes from the Pidgin itself, not from a different request.

The harmful items are low-to-moderate risk and set in Nigerian situations, such as fake bank alerts, airtime scams and exam malpractice. Only the prompts and overall rates are published, never harmful model outputs, and providers are told about serious gaps before anything is published.

How the dataset was made

I drafted the items with help from an AI assistant (Claude). Native Nigerian Pidgin speakers then check every item for meaning and natural phrasing, and a second speaker who hasn't seen the items blind-labels a sample so I can report agreement. Each item records who drafted it and who checked it. I think being open about this matters: an AI safety project should be honest about how AI was used to build it.

Status

Proposal and repo. Research question, method and code published.
Dataset drafted. 100 items, plus a labelling tool for native speakers.
Native-speaker checks. In progress.
Model runs and grader check. Three models, with the grader hand-validated.
Results and write-up. Published here, including what didn't go as expected.

Results

Coming soon. This section will show over-refusal and harmful-compliance rates for each model in English, Pidgin and code-switched form, plus examples of where models got it wrong.

← Back home