# Anthony Cho — AI Agent Portfolio

> Anthony designed an AI agent that researched and ranked 15 LLMs using real execution data — not marketing scores. The agent chose its own statistical methods, refused to fabricate missing data, and proved the naive "intelligence ÷ price" formula distorts rankings by 17× when cache efficiency is included.

## Key points

- **Result:** 15 LLMs ranked from 30 days of real fleet execution data (success rates, aborts, cache hit rates, token volumes) + real-time scraped intelligence scores.
- **17× distortion:** Naive score/price = 370 for DeepSeek V4 Flash; effective intelligence-per-dollar with 99% cache = 2,087. Cache is the ranking axis for agentic workloads.
- **Missing data handled honestly:** 9/15 models had no public intelligence score ("not yet scored"). Rejected mean imputation (MNAR bias against new models). Used Manski bounds: ranking recomputed at observed min/max intelligence. Where rank flips → honest "undecided".
- **Rejected weighted-sum and TOPSIS:** weighted-sum lets cheap price offset a 48% success rate; TOPSIS reverses ranks when models are added. Used Pareto filter + hard abort-rate gate (≤15%) + 3-axis normalized score.
- **Fleet default re-validated:** deepseek-v4-flash ranked #1 (157 runs, 100% success, 0 aborts, 99% cache).
- **Hidden cost exposed:** Kimi K3 was intelligence #1 (59.70) but had 21% abort rate — silently the most expensive to run.

## Final ranking (abort gate survivors)

1. deepseek-v4-flash — IQ 51.77, eff. cost $0.025/M, success 90.7%, score 0.794
2. hy3-paid — IQ n/a, eff. cost $0.076/M, success 87.0%, score 0.699
3. laguna-s-2.1-free — IQ n/a, eff. cost $0.000/M, success 86.4%, score 0.692
4. gpt-5.6-luna — IQ 52.32, eff. cost $0.099/M, success 87.3%, score 0.620

Gate-failed: Kimi K3 (21% aborts), Gemini 3.7 Flash (33%), GLM-5.2 (100%).
Undecided (no execution data): 8 models — reported honestly rather than imputed.

## Links

- [Full portfolio page](/)
- [Full story (Korean, 1st person)](story)
- [Raw ranking data](data.csv)
- [Ranking script](ranking.py)

## Method

- Missing intelligence scores → Manski bounds analysis (no imputation)
- MCDA → Pareto filter + abort-rate hard gate + 3-axis normalized score (execution weighted 2×)
- Cost → effective per-token cost with 90% cache: C_eff = 0.05·P_in + 0.90·(h·P_cache + (1-h)·P_in) + 0.05·P_out
- Success → Beta posterior shrinkage: p̂ = (k+α)/(n+α+β)
- Retry/abort → λ·(1-p̂)/p̂ human recovery cost

## Contact

Anthony Cho · Calgary, AB · anthony.cho@solutoconsulting.com