Research Findings

Robustness of alignment-eval performance to prompt perturbations — a principled, significance-gated read of the sweep. RQ1 how much does phrasing move alignment behaviour · RQ2 which perturbation classes move it most · RQ3 does model capability change susceptibility.

Scope & method. Trusted full-roster round (bulbasaur1-4 + charizard1-4): 14 capability-ranked models (ECI 115–157, 8 families) × 3 single-turn evals (MASK honesty · APE persuasion · sycophancy) × 5 generations × 13 perturbation classes spanning 7 taxonomy tiers (orthographic→framing). One cell = (model, eval, class, pole); dose magnitude pooled by sign (|dose|≥2). Each cell's Δ is measured against the control floor (tier −1, faithful reproduction = the paraphrase/sampling noise floor). Significance: control-floor permutation test + BH-FDR across the grid; capability/family via Spearman/Wilcoxon. Each eval rests on only 4–6 base scenarios — see the power section: the independent unit is the scenario, so the design is item-limited, not sample-limited.

RQ1 · How much does alignment depend on phrasing? Mixed

Two-part answer. (a) Absolute sensitivity is high. Even a faithful re-rendering of the same prompt moves the headline metric substantially — MASK lie-rate swings ≈8.3pp from paraphrase alone (APE/sycophancy ≈5.0–4.5pp), and the largest single class shifts metrics 12.3–23.8pp. Alignment-eval scores are materially phrasing-fragile. (b) But class-specific signal beyond that floor is mostly undetectable at this power. After correcting for the paraphrase floor and multiple comparisons, only 1/38 model×eval combinations show a perturbation class moving the metric beyond noise at q<.05 — the rest are 81% underpowered (per-cell minimum detectable effect ≈23–36pp ≫ typical effect ≈4–9pp; see Power below). The largest genuine class effect that does clear the bar is llama-4-maverick on MASK (+16.8pp).

Takeaway: phrasing demonstrably moves alignment metrics, and most of the movement is generic paraphrase/sampling variance rather than identifiable perturbation classes. We cannot resolve individual model×class effects below ~23pp — so this is "absence of evidence", not evidence that specific classes have no effect.

RQ2 · Which perturbation classes move behaviour most? (H2: higher tiers larger) Not supported

Across-model mean |Δ| is essentially flat across tiers (6.3–8.1pp from orthographic to framing) with no monotonic ordering: tier does not predict effect size (Spearman ρ=-0.00, p=0.88; tier-permutation p=0.60). Even the coarse surface (tiers 0–1) vs semantic (tiers 2–6) contrast is not significant in the trusted set (semantic +0.6pp, Wilcoxon p=0.11; it reached p≈.05 only when legacy high-repeat runs were pooled in). The top movers (hypothetical, politeness, formality, whitespace) are a mix of framing/register and a surface class (whitespace), not a clean semantic>surface story. H2 is not supported: the tier taxonomy does not rank classes by how much they move alignment behaviour in this data.

perturbation class	tier	mean \|Δ\| pp	median pp	n cells
hypothetical	framing	8.2	6.0	69
politeness	register	8.1	6.0	138
formality	register	8.0	6.0	138
whitespace	orthographic	7.8	7.3	69
clause_reorder	syntactic	7.4	5.3	69
sentiment	affect	7.3	6.0	138
third_person	framing	6.8	5.9	69
leading	framing	6.7	4.4	69
active_passive	syntactic	6.5	4.1	69
typos	char-noise	6.3	5.0	69

RQ3 · Does model capability change susceptibility? (H1: larger = less affected) Not supported

No significant relationship between capability (ECI) and noise-corrected susceptibility (Spearman ρ=0.21, p=0.46, n=14; raw-susceptibility ρ=0.06, p=0.83). Per-eval correlations are all non-significant. The point estimate is, if anything, weakly positive — more-capable models are marginally more sensitive, the opposite of H1. Model capability does not buy robustness to phrasing in this data.

model	ECI	family	corrected susceptibility pp	raw pp	mean max swing pp	# evals q<.05
llama-3.1-8b-instruct	115	llama	-0.5	4.7	24.3	0
llama-3.1-70b-instruct	126	llama	-1.5	3.9	12.3	0
gpt-4o-mini	127	gpt4	0.4	6.8	18.8	0
llama-4-maverick	133	llama	6.9	12.6	41.0	1
deepseek-chat-v3-0324	137	deepseek	-2.9	2.4	11.9	0
qwen3-235b-a22b	139	qwen	-1.3	4.9	20.1	0
gemini-2.5-flash	140	gemini	0.2	7.2	18.0	0
gpt-5-mini	146	gpt5	0.1	2.7	9.5	0
claude-sonnet-4.6	147	claude	-0.4	2.2	12.6	0
deepseek-v4-pro	150	deepseek	-0.3	7.4	26.0	0
grok-4.20	154	grok	-1.1	4.7	13.5	0
gemini-3.5-flash	155	gemini	-0.6	5.7	23.7	0
glm-5.2	155	glm	0.3	3.6	11.7	0
claude-opus-4.8	157	claude	3.5	7.5	19.8	0

H3 · Are model families hit by the same perturbation classes? Not supported

Same-family models are slightly more alike in their tier-susceptibility profile than cross-family pairs (within r=0.22 over 6 pairs vs across r=0.06 over 85 pairs), but the difference is not significant (Mann–Whitney p=0.21) and rests on only 6 within-family pairs. We cannot conclude families share a susceptibility fingerprint; the profile looks largely idiosyncratic to the individual model. (Underpowered — see below.)

Do we have enough samples? Mixed

The honest answer is level-dependent, and the limit is base scenarios, not sample count. Each eval has only 4–6 base scenarios (items); because the independent unit is the scenario (not the sampled generation), item-clustering deflates the per-cell effective n to ≈4–9, so the per-cell minimum detectable effect is ≈23–36pp — far above the typical observed effect (≈4–9pp). 724 of 897 cells are formally underpowered. Adding generations (this round already has 5) barely helps; more diverse scenarios per eval is what's needed.

Well powered → trust these: the absolute paraphrase-sensitivity numbers (control floor estimated from 13 models × ~40–120 control samples each) and the population-level tier/class rankings (n=69–276 cells/tier).
Under-powered → treat as bounded: individual model×class effects below ~23pp; H2's tier gradient. "Not supported" here = no detectable effect at this power, not proven zero.
Design-limited (n=14 models): H1 & H3 can only detect strong capability/family relationships (|ρ|≳0.5); a weak true effect would be invisible. These need more models, not samples.

Method caveats

Conservative effect sizes. pooling |dose|≥2 by sign scores each class at its typical, not maximal, strength; degenerate control bands (rate 0/1, undefined noise floor) are excluded, not counted as signal.
Eval idiosyncrasies. MASK has the largest paraphrase floor (~8.3pp) and highest item-heterogeneity (so its effective n collapses most). bias & impossible_coding are not in the bulbasaur/charizard round, so they sit outside this grid.
Significance is gated. every Δ is measured against the faithful-reproduction floor and BH-FDR'd across the grid; we report signal, not raw movement.

tier	class	mean \|Δ\| pp	median pp	n cells
0	orthographic	7.8	7.3	69
1	char-noise	6.3	5.0	69
2	syntactic	7.0	4.9	138
4	register	8.1	6.0	276
5	affect	7.3	6.0	138
6	framing	7.2	5.6	207

Prompt-perturbation robustness — results