Documentation

Project Overview

Intelligence Arbitrage is a systematic experiment testing whether small, locally-running language models (2–14B parameters), when optimized with DSPy's MIPROv2 prompt optimizer, can match the performance of frontier API models (GPT-4o, GPT-5.2).

The core idea: frontier models are priced uniformly across all tasks, but many production tasks don't require frontier intelligence. If optimization can close the gap on specific tasks, teams can route work to cheap local models and pay frontier prices only where they're genuinely necessary.

Hypothesis

DSPy's optimization pipeline (teacher bootstrapping + MIPROv2) can close the performance gap between local open-weight models (3B–14B parameters) and frontier API models (GPT-4o, GPT-5.2) on structured NLP tasks.

The "arbitrage" framing: if a $0/eval local model can match a $0.01+/eval API model on a specific task after optimization, you capture the accuracy without paying the inference cost. The larger the performance gap that optimization closes, the larger the arbitrage opportunity.

Key Findings

1. Optimization is a force multiplier for weaker models. qwen2.5:7b gained +14.2pp average (Phase 1, n=50), gpt-4o +7.9pp. The smaller the model, the more prompt optimization helps.
2. 14 rank inversions. After optimization, local models beat API models in 14 of 36 possible matchups. The most dramatic: mistral classification (54pp swing past gpt-4o).
3. MIPROv2's all-or-nothing demo selection is suboptimal. It always chose 0 or 5 demos, never intermediate. Phase 3 proved phi4 peaks at n=1–2 on most tasks. A granular strategy could recover 2–8pp.
4. Two failure modes. Instruction damage (optimized instructions hurt: phi4 rag −8pp) and demo interference (too many examples: phi4 analysis inverted-U peaking at n=2). These are independent and can compound.
5. Optimization response is task-specific. Tasks explain 2× more variance than models (ratio 0.50). Code always 0pp, math always big gains — regardless of model.
6. Demo count optima are model-specific. On math, llama3.2 needs all 5 demos (monotonic), phi4 gets +20pp from demo #1 alone. Same task, different optimal strategy.
7. Extra compute per inference doesn't help (Phase 6a). Best-of-3, Majority Class Confidence, and Iterative Refinement showed no statistically significant improvement over vanilla CoT (all McNemar's p > 0.05, n=200).
8. Two confirmed significant findings (Bonferroni-corrected, n=200). Mistral + MIPROv2 on Agentic: +13.5pp (p<0.0001). GPT-5.2 + MIPROv2 on Synthesis: +12.0pp (p<0.0001). All other unified experiment results were within noise.

Optimization Lift by Model (Phase 1)

Intelligence Arbitrage Scoreboard

Local optimized vs GPT-5.2 baseline:

Experimental Setup

• 9 task categories: agentic (StrategyQA), analysis (BoolQ), classification (Banking77), code (HumanEval), extraction (CoNLL-2003), math (GSM8K), QA (HotPotQA), RAG (SQuAD), synthesis (XSum)
• 8 models across phases: Local (Phase 1): llama3.2 (3B), mistral (7B), qwen2.5:7b (7B), phi4 (14B). API (Phase 1): gpt-4o, gpt-5.2. Reasoning (Phase 4): qwen3:4b (4B), deepseek-r1:7b (7B).
• Optimizer: MIPROv2 with 3 seeds (10, 20, 30), best-of-3 selected by dev score. GPT-4o as teacher model.
• Evaluation: 50 examples per task (Phase 1), 200 examples (Unified experiment). Temperature=0, separate train/eval splits verified.
• Total evaluations: 108 (Phase 1) + 42 (Phase 3) + 17 (Phase 4) + 8 (Phase 5) = 175 runs

Task Domains

Phase 1: Baseline vs Optimized (6 Models × 9 Tasks)

MIPROv2 optimization applied to 6 models across all 9 task domains. Each condition evaluated on n=50 examples.

Optimization Delta Matrix

Percentage-point change from optimization (positive = improved). Large positives (math, extraction, QA) represent genuine gains; deltas ≤±4pp are within the noise floor at n=50.

Rank Inversions

14 cases where an optimized local model outperformed an optimized API model. Most dramatic examples:

Phase 2: Variance Analysis — Task-Specific or Model-Specific?

This means the task type predicts whether optimization will help more reliably than the model choice. Code always shows 0pp delta. Math always shows big gains. These patterns hold regardless of which model you use.

Task Consistency Tiers

MIPROv2 also showed an all-or-nothing demo count pattern: 30 programs used 5 demos, 24 used 0 demos. No intermediate counts were ever chosen — suggesting MIPROv2 uses a binary heuristic rather than fine-grained demo selection.

Phase 3: Demo Curve Experiment

For 6 (model, task) pairs where Phase 1 showed negative or near-zero deltas, we progressively evaluated performance at n=0, 1, 2, 3, 4, 5 demonstrations — decomposing optimization into instruction effect and demo effect.

Key Decomposition Results

Demo count optima are also model-specific: on math, llama3.2 shows monotonic improvement through all 5 demos, while phi4 gets +20pp from demo #1 alone and shows diminishing returns after that.

Phase 4: Reasoning Model Baselines

Two reasoning models — qwen3:4b and deepseek-r1:7b — evaluated as unoptimized baselines across all 9 tasks.

Complementary Profiles

deepseek-r1:7b's classification score (64%) matches GPT-5.2's baseline without any optimization. However, its per-example latency is 13–18 seconds vs qwen3:4b's ~95ms — a critical production tradeoff.

Phase 5: Optimization Tipping Point

Tests: at what number of labeled examples does optimization start working? Uses BootstrapFewShotWithRandomSearch on qwen3:4b with GPT-4o as teacher.

These three tasks represent the three possible outcomes of optimization: responsive (math jumps +14pp from a single example), resistant (classification: 3.7 hours, zero gain), and broken (QA program failed to compile at n=0 and n=1).

Methodology & Limitations

Optimization Protocol

• MIPROv2 with GPT-4o as teacher model
• Three seeds (10, 20, 30) run independently; best-of-3 by dev score selected
• Each optimized program contains rewritten task instructions + 0 or 5 curated demonstrations
• Binary demo count is MIPROv2's decision, not experimenter-chosen

Statistical Limitations

Two Confirmed Significant Findings (Unified Experiment, n=200)

Technical Stack

Optimization Loop

1. Teacher Bootstrapping: GPT-4o generates high-quality few-shot examples for the target task
2. MIPROv2: Optimizes task instructions and selects demonstrations for the student model
3. Variance Reduction: 3 seeds (10, 20, 30); best-of-3 selected by dev set accuracy

Interactive Dashboards

The experiment results are visualized across three interactive pages: