Anthony Cho — AI Agent Portfolio

Result — 30-second read

I designed an AI agent that ranked 15 LLMs for our fleet — not from marketing scores, but from real execution data: 30 days of actual runs, success rates, abort counts, cache hit rates, token volumes.

It proved the naive "intelligence ÷ price" formula distorts the true ranking by 17× when cache efficiency is included. And when 9 of 15 models had no public intelligence score, it refused to invent numbers — it reported an honest "ranking undecided" instead.

The result: the fleet's default model was re-validated as optimal, and a hidden cost was exposed — the #1 intelligence model (Kimi K3) had a 21% abort rate that silently inflated real cost.

Problem

"Rank these 15 models for my workload. Some have scores, some don't. There's execution data, prices, cache rates... how do we combine this honestly?"

This isn't a model recommendation — it's a decision system design problem:

6/15 models had public intelligence scores; 9 were "not yet scored"
Metrics don't combine: intelligence, cost, success rate, cache efficiency, abort rate
The easy formula ("score ÷ price") ignores cache and success — and we proved how badly

Method (chosen by the agent, not dictated)

1. Missing scores → Manski bounds, not mean imputation

"Not yet scored" correlates with recent release (MNAR). Mean imputation systematically penalizes new models. Instead: recompute the ranking twice, with intelligence set to the observed min and max. If the rank doesn't flip, the missing data doesn't matter. If it does, say "undecided."

2. MCDA → rejected weighted-sum and TOPSIS

Weighted-sum lets cheap price offset a 48% success rate. TOPSIS reverses ranks when models are added — fatal in a fleet that changes monthly. Instead: Pareto filter (eliminate dominated models, no weights needed) → hard gate: abort rate ≤ 15% (aborts are mostly quota/rate-limit, but they explode retry + human-recovery cost) → 3-axis normalized score (intelligence, cost, success; execution weighted 2×).

3. Cost per task, not score ÷ price

With 99% cache hit, effective intelligence-per-dollar is 17× higher than naive score/price. Cache is the ranking axis for agentic workloads.

Effective cost: C_eff = 0.05·P_in + 0.90·(h·P_cache + (1−h)·P_in) + 0.05·P_out
Success shrunk via Beta posterior: p̂ = (k+α)/(n+α+β)
Abort cost: λ·(1−p̂)/p̂ human recovery

Evidence (real data, 2026-08-16)

Rank	Model	Public IQ	Eff. cost $/M	Success	Score
1	deepseek-v4-flash	51.77	$0.025	90.7%	0.794
2	hy3-paid	n/a	$0.076	87.0%	0.699
3	laguna-s-2.1-free	n/a	$0.000	86.4%	0.692
4	gpt-5.6-luna	52.32	$0.099	87.3%	0.620

Gate-failed: Kimi K3 (intelligence #1 at 59.70, but 21% aborts), Gemini 3.7 Flash (33%), GLM-5.2 (100%)
Undecided (no execution data): 8 models — reported honestly rather than imputed
Hidden cost found: the #1 "smartest" model was silently the most expensive to run

Why This Belongs in a Portfolio

The agent decides. It chose and rejected methods — it didn't just compute what it was told.
Statistical honesty. It refused to fabricate scores; it reported bounds and "I don't know."
Real cost thinking. Cache, success rate, retries — not list prices or benchmark bragging.

The real output isn't a ranking table. It's a methodology for trustworthy decisions under incomplete data — the exact problem every AI-first company faces at scale.