Why Goldman Tiered Its Models Instead of Buying the Biggest

Last Updated: July 16, 2026

AI technology in banking hit an inflection point when Goldman Sachs rolled its GS AI Assistant out to 46,000 employees and began piloting autonomous banking agents this year — and notably, it didn't buy the biggest model on the market. It built a tiered stack where small, domain-tuned models handle 80% of the volume and a frontier LLM handles the rest. Most financial services firms are copying the wrong half of that decision, and the mistake is expensive.

Most AI technology workflows are solving the wrong problem entirely. The question dominating banking and fintech boardrooms — 'custom Small Language Model or off-the-shelf LLM?' — is real and urgent, driven by OpenAI's GPT-4o pricing, Anthropic's Claude enterprise tiers, and open-weight models like Llama and Mistral that now fine-tune for a few thousand dollars. By the end of this article you'll have a decision framework, real cost math, and a reference architecture you can take to your risk committee.

Overview: Why the SLM-vs-LLM Question Is the Wrong Starting Point

The instinct in most financial services organizations is to treat this as a binary procurement decision: license a hosted frontier model like GPT-4o or Claude, or build a custom Small Language Model (SLM) fine-tuned on proprietary data. Framed that way, you'll almost certainly pick wrong — because the real cost, risk, and performance drivers of any AI technology stack live in the layer between the models, not in the models themselves.

An SLM is a language model in the roughly 1B–15B parameter range — think Microsoft's Phi-3, Mistral 7B, Llama 3.1 8B, or Gemma 2 — small enough to run on a single GPU or even CPU, cheap enough to fine-tune on domain data, and private enough to deploy inside your own VPC. An off-the-shelf LLM is a hosted frontier model accessed via API: OpenAI's GPT-4o, Anthropic's Claude 3.5, or Google's Gemini, offering the deepest reasoning at the highest per-token cost and least control.

Why now? The economics flipped in 2025. Fine-tuning an open-weight 8B model on a bank's document corpus dropped below $10,000. Inference on a self-hosted SLM now costs a fraction of a cent per thousand tokens versus 2–5 cents on frontier APIs. And regulators — from the SEC to the European Banking Authority — began issuing explicit guidance on model explainability and data residency that hosted-only strategies struggle to satisfy. The Federal Reserve and the Office of the Comptroller of the Currency have both signaled that model-risk-management expectations (SR 11-7 lineage) now extend to generative systems.

Here's the counterintuitive truth: the firms winning with AI technology in finance aren't the ones who picked the 'right' model — they're the ones who solved the handoff between models, systems, and humans. A JPMorgan compliance workflow doesn't use one model; it uses a classifier SLM, a retrieval layer, a reasoning LLM, and a human-in-the-loop gate — and the value or the disaster lives in how those hand off.

This article maps that gap. We'll break the decision into a named framework with distinct layers, show how each works in a real financial services deployment, quantify the ROI, and walk through the reference architecture. By the end, you'll be able to design a stack — not just pick a model. For broader context on where this fits, see our overview of enterprise AI deployment patterns.

In financial services, you don't lose money on the model that's wrong 3% of the time. You lose it on the handoff nobody designed, where a 97%-accurate classifier feeds a 97%-accurate extractor and your end-to-end reliability quietly falls to 83%.

What Is a Custom SLM and When Does It Beat a Frontier LLM?

A custom SLM isn't a weaker LLM. It's a specialized one. When you fine-tune Llama 3.1 8B on ten years of your firm's KYC documents, loan memos, and compliance filings, it develops a density of domain knowledge that a general-purpose frontier model doesn't carry by default. Microsoft's Phi-3 research demonstrated that a 3.8B parameter model, trained on curated data, matched models 10x its size on targeted benchmarks.

For financial services, the SLM advantage compounds across four axes: cost per inference, latency, data residency, and auditability. A self-hosted 8B model responding to a transaction-classification query costs a fraction of a cent and returns in under 200ms inside your VPC. The equivalent GPT-4o call costs 20–60x more, adds network latency, and sends your data to a third party — a non-starter for many regulated workloads.

But the SLM fails exactly where the frontier LLM shines: novel, multi-step reasoning over ambiguous inputs. Ask an 8B model to reason through a complex derivative structuring question or an unusual fraud pattern it's never seen, and it will confabulate. This is why the answer is almost never 'SLM or LLM' — it's 'SLM and LLM, with a routing layer that decides which handles what.' If you want to skip the plumbing, you can explore our AI agent library for pre-built tiered routing blueprints.

The Five Layers of the AI Coordination Gap Framework

Stop looking at models. Start looking at the coordination surface. Every production financial services AI technology system is composed of five layers — and the reliability, cost, and compliance of the whole are determined by how cleanly these hand off to each other. This is where deployments live or die.

Layer 1: The Routing Layer

The routing layer decides which model handles each request. A cheap SLM classifier reads the incoming query — 'Is this a routine transaction categorization, or a novel compliance judgment?' — and routes accordingly. Get this wrong and you either burn frontier-model money on trivial tasks or send high-stakes reasoning to a model that will hallucinate.

In practice, this is a small fine-tuned model (often the SLM itself, or a dedicated 1–3B classifier) running at the edge of your pipeline. Latency here matters enormously because it sits in front of every request. Tools like LangChain routers and LangGraph state machines make routing explicit and auditable — critical when a regulator asks 'why did this request go to the autonomous agent?'

Layer 2: The Retrieval Layer (RAG)

Retrieval-Augmented Generation grounds the model in your firm's actual data. Instead of relying on a model's frozen training knowledge, the retrieval layer pulls relevant documents — regulatory filings, account histories, policy documents — from a vector database like Pinecone and injects them into the prompt. This is the single most important layer for reducing hallucination in finance.

The coordination gap here is brutal. A great retrieval system feeding a great model still fails if the chunking strategy is wrong, if embeddings are stale, or if the retrieved context contradicts itself. I've seen teams spend months debugging 'model errors' that were actually retrieval errors — bad chunking, mismatched embedding models, indexes that hadn't refreshed in weeks. Most 'the model got it wrong' incidents in banking are actually 'the retrieval layer fed it the wrong context.' Our deeper walkthrough of RAG pipelines covers the chunking and embedding-refresh patterns that prevent this.

Layer 3: The Reasoning Layer

This is where the SLM-vs-LLM decision actually gets made — per task, not globally. Routine extraction and classification stay on the SLM. Complex multi-step reasoning escalates to a frontier LLM. The reasoning layer is where you spend your frontier-model budget deliberately, on the 15–20% of tasks that genuinely need it.

Layer 4: The Orchestration Layer

Orchestration is the sequencing logic — the code that says 'first classify, then retrieve, then reason, then validate, then hand to a human if confidence is low.' This is where multi-agent systems live. Frameworks like LangGraph, AutoGen, and CrewAI define these flows as graphs with explicit state, retries, and fallbacks.

Layer 5: The Human-in-the-Loop Gate

In regulated finance, no autonomous decision above a risk threshold ships without a human gate. This layer defines confidence thresholds, escalation paths, and audit logging. It's not a fallback — it's a compliance requirement. The design question is where the gate sits and what triggers it, because a gate that fires on every request destroys the ROI, and one that never fires destroys the bank.

You cannot fine-tune your way out of the Coordination Gap. A perfect model in a badly sequenced pipeline will still fail — because the failure was never in the model. It was in the seam.

Real Deployments: How Financial Firms Are Actually Building This

Theory is cheap. Here's what the framework looks like in production at named institutions.

Goldman Sachs: Tiered Agents at Scale

Goldman's GS AI Assistant, deployed to 46,000 employees, is explicitly tiered. Marco Argenti, Goldman's CIO, has described the firm's approach as using targeted models for high-volume developer and document tasks while reserving frontier reasoning for complex analysis. The autonomous banking agents piloted in 2025–2026 sit inside strict orchestration and human-review gates — the Coordination Gap made institutional policy. Reuters coverage of the rollout underscores the tiered design.

JPMorgan Chase: LLM Suite and Grounded Retrieval

JPMorgan's internal LLM Suite reached over 200,000 employees. Their public engineering commentary emphasizes retrieval-grounding and access controls over raw model size — a direct acknowledgment that the retrieval and orchestration layers, not the model, carry the compliance burden. Teresa Heitsenrether, JPMorgan's Chief Data & Analytics Officer, has framed the rollout around governed, grounded deployment rather than model maximalism. That framing matters. It's an institution of that scale publicly saying the seams are the product.

Morgan Stanley: RAG Over Research Corpus

Morgan Stanley's wealth-management assistant, built with OpenAI, is a canonical RAG deployment: a frontier model grounded in the firm's proprietary research library so advisors get answers sourced from vetted internal content, not model memory. It's a reminder that even a frontier-LLM deployment lives or dies on its retrieval layer.

How to Implement the Framework: A Step-by-Step Path

Here's the practical build sequence for an operations or engineering leader starting from zero. You can prototype most of this in weeks, not quarters.

Step 1: Instrument before you build

Log a representative month of the requests your AI technology system will handle. Cluster them by complexity. This tells you your SLM/LLM split — most financial services workloads land at 70–85% SLM-suitable. Skip this step and you'll over-provision frontier-model capacity and wreck your ROI before you've shipped anything.

Step 2: Stand up the retrieval layer first

Before you fine-tune anything, build the RAG pipeline. Use Pinecone or a self-hosted vector DB, embed your document corpus, and validate retrieval quality with a held-out question set. A frontier LLM on a good retrieval layer will outperform a custom SLM on a bad one every time — I've watched teams learn this the hard way after six weeks of fine-tuning that fixed nothing. If you want production-ready components to start from, explore our AI agent library for pre-built retrieval and orchestration blueprints.

# Route between SLM and frontier LLM based on classifier confidence
from langgraph.graph import StateGraph

def route_request(state):
    # Cheap SLM classifier returns complexity + confidence
    result = slm_classifier.invoke(state['query'])
    if result['complexity'] == 'standard' and result['confidence'] > 0.85:
        return 'slm_reasoning'   # in-VPC, ~$0.0002/req
    return 'llm_reasoning'       # frontier API, escalate

graph = StateGraph(dict)
graph.add_node('classify', route_request)
graph.add_node('slm_reasoning', run_slm)
graph.add_node('llm_reasoning', run_frontier_llm)
graph.add_conditional_edges('classify', route_request)
# Every transition logged with model version for audit
app = graph.compile()

Step 3: Fine-tune the SLM on your highest-volume task

Pick the single most frequent task — usually classification or extraction. Fine-tune Llama 3.1 8B or Mistral 7B on a few thousand labeled examples. This is the $2K–$10K step that permanently removes that task's cost from your frontier bill. Compare against a GPT-4o baseline on the same held-out set before committing. If the fine-tuned SLM doesn't beat GPT-4o on your specific task, you don't have enough labeled data yet — don't ship it.

Step 4: Wire orchestration with explicit gates

Use workflow automation tooling and LangGraph to sequence the steps. For lighter-weight, non-critical automations, n8n gives you a visual layer to connect models to internal systems. Bake in validation gates at every seam — this is where you reclaim the reliability the Coordination Gap steals. See our guide to n8n AI workflows for connector patterns.

Step 5: Add MCP for tool and data access

Adopt Model Context Protocol (MCP) to give models standardized, auditable access to internal tools and data sources. MCP turns ad-hoc integrations into a governed interface — increasingly the expectation in enterprise AI compliance reviews. Explore more patterns in our AI agents and orchestration guides, or browse ready-made connectors when you explore our AI agent library.

What Most Companies Get Wrong About SLM vs LLM Decisions

The failure patterns are remarkably consistent across financial services deployments. Here are the ones that cost the most.

What Comes Next: The 18-Month Outlook

The next competitive advantage in financial services AI technology isn't a bigger model. It's the firm that can show a regulator exactly which model handled a decision, why, and where a human signed off — in a diagram that fits on one page.

About the Author

Rushil Shah

AI Systems Builder & Founder, Twarx

Rushil Shah is the founder of Twarx and an AI systems builder who has spent years designing autonomous workflows, multi-agent architectures, and AI-powered business tools. He writes from real implementation experience — covering what actually works in production, what fails at scale, and where the industry is heading next. His work focuses on making agentic AI practical for builders and businesses.

LinkedIn · Full Profile