Short answer: put one thin gateway that you own between your product and every model, and keep everything that is hard to move (prompts, tool schemas, eval sets, logs, routing rules) in your own repository. Route in this order: data sensitivity first, task difficulty second, cost third, availability last. Let fallbacks move traffic only between models that are allowed to see the same data. Before you shift real traffic to a new model, run it against your own eval set. Lock-in rarely lives in the API call. It lives in prompts tuned to one model, embeddings you cannot swap and tests you never wrote.
This guide is for engineering leads who already ship an LLM feature on one provider and want a second model, or a private one, without a rewrite. It is engineering guidance, not legal advice. The legal reading of the DPDP Act for your business goes to counsel.
Why use more than one model?
A single model is the simplest design, and for a prototype it is the right one. Teams add a second model for four practical reasons.
- Cost. Public list prices span a range of about 80 times between small and top-tier models (see the table below). Most requests do not need the top tier.
- Availability. Providers have outages, rate limits and quota walls. One provider is one point of failure.
- Data class. Some data should not leave your own servers or your chosen region. Other data can go to a public API. One feature often handles both.
- Capability. Models differ on long documents, code, extraction, multilingual text and tool use. The best model for one task is rarely the best for all of them.
Each extra model adds evals and failure modes. Add one when a reason above is real for you.
What does a multi-model LLM architecture look like?

Think of four layers. Your product code talks only to the first one.
| Layer | What it does | What belongs here |
|---|---|---|
| 1. Model interface | What it does: One internal function such as `complete(task, messages)`. No provider names in product code. | What belongs here: Task names, not model names. Structured output schemas. Timeouts. |
| 2. Gateway | What it does: Translates the call to each provider, applies keys, limits and logging. | What belongs here: Provider adapters, virtual keys per team or tenant, spend caps, request logs. |
| 3. Router | What it does: Picks the model for this request and decides what happens on failure. | What belongs here: Policy file: data class, task tier, fallback order, cooldowns. |
| 4. Models | What it does: Private open-weight models you host, and third-party APIs. | What belongs here: Each with a data class it is cleared for, a version pin and an owner. |
Two rules keep this honest. Product code asks for a task ("summarise_ticket"), and the router maps it to a model, so changing a model is one line of policy. Every model also carries a data class label, such as "public", "internal" or "personal data", and the router refuses to send a request to a model labelled lower than the request.
Where does lock-in really come from?
Most teams think lock-in is the SDK import. An OpenAI-compatible interface is common across gateways, vLLM and several providers, so swapping the HTTP call is the easy part. These are the harder ones.
| Lock-in point | Why it hurts | How to loosen it |
|---|---|---|
| Prompts tuned to one model | Why it hurts: The same prompt can behave differently on another model, in format, length and refusals. | How to loosen it: Keep prompts as versioned files per task, with the expected output as a schema. |
| Embeddings | Why it hurts: Vectors from different embedding models cannot be compared, so switching means re-embedding the corpus. | How to loosen it: Store the embedding model and version on every chunk, and keep the source text so you can rebuild. |
| Tool and function-call formats | Why it hurts: Providers differ in how they describe tools and return calls. | How to loosen it: Define your tool schemas once in your own format. Let the gateway translate. |
| Evals that do not exist | Why it hurts: Without a test set you cannot tell if a new model is better, so you never switch. | How to loosen it: Build a small eval set from real, redacted cases. This is your real switching asset. |
Our guide to LLM observability lists the fields to log on every call, in your own store and not only in a vendor's console. The same fields let you compare models later.
How should a router choose between models?
Use a short policy a reviewer can read. Four checks, in order.
- 1. Data class. Does this request contain personal data, client secrets or regulated data? If yes, only models cleared for that class are candidates. This check is a hard filter, never a score.
- 2. Task tier. Is the task easy (classify, extract, rewrite), medium (summarise, draft, answer from retrieved text) or hard (multi-step reasoning, difficult code)? Map each task to a tier in config. Start with static rules per task, and add a difficulty classifier only if evals show static rules waste money.
- 3. Cost and latency. Within the allowed tier, choose the cheapest model that meets your latency target and your eval score.
- 4. Availability. If the first choice is down or rate limited, go to the next model on the list, still inside the same data class.
Here is why the tier matters. These are public list prices, checked on 20 November 2026, in USD per 1 million tokens, ex-tax. Anthropic prices are for prompts of 100K tokens or less. OpenAI prices are for context under 272K tokens.
| Tier | Model (provider) | Input | Output |
|---|---|---|---|
| Small | Model (provider): Haiku 5.5 (Anthropic) | Input: $0.10 | Output: $0.50 |
| Small | Model (provider): GPT-6 Luna (OpenAI) | Input: $0.10 | Output: $0.50 |
| Mid | Model (provider): Sonnet 5.5 (Anthropic) | Input: $2 | Output: $10 |
| Mid | Model (provider): GPT-6.2 Sol (OpenAI) | Input: $2 | Output: $10 |
| Mid | Model (provider): Gemini 4 Carbon (Google) | Input: $2 through 30 November 2026, $4 from 1 December 2026 | Output: $10 through 30 November 2026, $20 from 1 December 2026 |
| Top | Model (provider): Opus 5.5 (Anthropic) | Input: $4 | Output: $20 |
| Top | Model (provider): Fable 5.2 (Anthropic) | Input: $8 | Output: $40 |
| Top | Model (provider): GPT-6.2 Astra (OpenAI) | Input: $8 | Output: $40 |
Take one request of 1,500 tokens in and 300 tokens out. At top-tier rates ($8 in, $40 out) it costs about $0.024. At mid-tier rates ($2, $10) about $0.006. At small-tier rates ($0.10, $0.50) about $0.0003. Suppose, as an illustration and not a benchmark, that 70% of traffic is easy enough for the small tier, 25% needs the mid tier and 5% the top tier. The blend is about $0.0029 a request: roughly half the all-mid cost and one eighth of the all-top cost. The saving holds only if the small tier passes your evals on the easy tasks. For the wider cost picture, see why LLM bills grow faster than usage.
Which data should go to a private model, and which to an API?
Decide this per data class, not per feature. A practical split looks like this.
| Data class | Typical examples | Where it can go |
|---|---|---|
| Public or synthetic | Typical examples: Marketing copy, public docs, test data | Where it can go: Any approved API, cheapest model that passes evals |
| Internal, low risk | Typical examples: Internal wikis, non-personal tickets | Where it can go: Approved APIs with zero-retention or no-training terms, or private models |
| Personal data | Typical examples: Customer names, contacts, account details, HR data | Where it can go: Private models in your own VPC or servers, or an API only after redaction and a signed data processing agreement |
| Highly sensitive | Typical examples: Health, financial account data, credentials, client secrets | Where it can go: Private models only, or redact before any external call |
The DPDP Act does not require data to stay in India and does not ban foreign AI APIs. Section 16 lets the government restrict transfers to notified countries, and none are notified as of November 2026. Section 16(2) keeps stricter sector laws, such as RBI payment data storage rules. Hosting in India avoids future Section 16 restrictions and makes sector rules easier to meet. The core DPDP duties start on 13 May 2027, so build the routing and logging now. See our guides to DPDP-compliant AI architecture and PII redaction for LLMs.
A hosted router is another processor that sees prompts. For DPDP-sensitive data, use a self-hosted gateway in front of India-hosted models.
How do fallbacks and failover work without breaking your data rules?
Fallbacks are where multi-model systems quietly break their own rules. A request cleared for a private model fails, the router retries on a public API, and personal data has left. Fix this in the policy.
- Fallback lists are per data class. A personal-data request falls back to another private model, or fails with a clear error. It does not fall back to a public API unless that API is cleared for the class.
- Fail closed for sensitive classes. A visible error beats a silent leak. Show a retry message and alert the team.
- Handle context length separately. Move an over-long prompt to a larger-window model inside the same class. LiteLLM has `context_window_fallbacks` and `content_policy_fallbacks` alongside ordered fallbacks.
- Use cooldowns. In LiteLLM's Router the default cooldown is 5 seconds, a 429 triggers an immediate cooldown, and so does a failure rate above 50% in a minute.
- Log every fallback. Record the model first chosen, the model that answered, and the reason.
OpenRouter, a hosted router, tries the entries of a `models` array in order on errors such as rate limits, downtime, moderation and context length, and prices the request by the model that answered. Its `allow_fallbacks` setting is true by default, so restrict providers with `only`, `order` or `data_collection: "deny"` when data matters.
Should you build a gateway, use LiteLLM, or use OpenRouter?

| Option | What it is | Good for | Watch for |
|---|---|---|---|
| Your own thin wrapper | What it is: A small internal service or library with an adapter per provider | Good for: One or two providers, simple routing, full control | Watch for: You maintain every provider change and edge case yourself |
| LiteLLM (self-hosted) | What it is: An open-source AI gateway that calls 100+ providers in the OpenAI format, as an SDK or a proxy. The proxy adds virtual keys, spend tracking, guardrails and load balancing. MIT licence, except the `litellm/enterprise/` folder. Latest release as of 20 November 2026: v1.110.2. | Good for: Several providers plus your own vLLM server, cost attribution per team, running inside your own network | Watch for: You run and patch it. Read the licence of the enterprise folder before using those features |
| OpenRouter (hosted) | What it is: One API and one bill for many models. No markup on model prices, a 5.5% fee ($0.80 minimum) on credit purchases, and bring-your-own-key free up to $25,000 a month on pay-as-you-go, then 5%. | Good for: Fast experiments and low-sensitivity workloads | Watch for: It is another processor that sees your prompts. Not a fit for personal data without a clear data processing review |
The LiteLLM Router's default strategy, `simple-shuffle`, is marked "Recommended for Production" in its docs. A common path: a hosted router for prototypes, a self-hosted gateway before customer data goes through, and the same internal interface throughout.
How do you add a private model to the mix?
The usual route is an open-weight model served with vLLM. `vllm serve
`--gpu-memory-utilization` (share of GPU memory, 0 to 1), `--max-model-len` (prompt plus output length) and `--tensor-parallel-size` (GPUs to split across, default 1) decide whether a model fits. vLLM supports quantisation such as AWQ, GPTQ and FP8, which "trades off model precision for smaller memory footprint". As a rough estimate, weights take about parameters times bytes per parameter: 2 bytes in BF16, 1 in FP8 or INT8, about 0.5 at 4-bit. A 31B model is about 62 GB in BF16 and about 16 GB at 4-bit, before the KV cache and overhead, which grow with context and concurrency.
A fixed GPU bill replaces a per-token bill. E2E Networks' public list shows an NVIDIA H100 80GB at ₹379.05 an hour on demand, ex-GST, checked 20 November 2026. Running one for 720 hours is about ₹2.7 lakh before staff, storage and networking, busy or idle. A private tier suits steady, sensitive traffic. Our private LLM vs API cost model shows the break-even.
In India you can host on AWS Mumbai (`ap-south-1`) or Hyderabad (`ap-south-2`, opt-in), Azure Central India (Pune), or GCP `asia-south1` (Mumbai) and `asia-south2` (Delhi). GPU types differ by zone, so check before planning. The IndiaAI compute portal offers subsidised rates for approved projects as a monthly, per-request subsidy, so do not plan production capacity on it. For help with the private tier, see Own Your AI.
Which open-weight model licences affect a multi-model setup?
Licences are set per model, not per vendor. Record the licence next to each model in the policy file. This is a summary, not legal advice. Read the licence on each model card.
| Model family | What the public terms say (checked 20 November 2026) |
|---|---|
| Llama 4 (Scout, Maverick) | What the public terms say (checked 20 November 2026): Llama 4 Community License. A company with more than 700M monthly active users in the preceding calendar month must request a licence from Meta. Distributors must display "Built with Llama" and start derivative model names with "Llama". The Acceptable Use Policy applies, and for multimodal models it withholds the licence grant from EU-based individuals and companies. |
| Gemma 4 | What the public terms say (checked 20 November 2026): Apache 2.0. Earlier Gemma versions use Google's Terms of Use, which pass restrictions downstream. |
| Qwen | What the public terms say (checked 20 November 2026): Mixed. Qwen3.8-27B is Apache 2.0. Qwen3.8-Flash-Next uses the Qwen Community License 1.0. Qwen-Image-2.1 is non-commercial. |
| Mistral | What the public terms say (checked 20 November 2026): Mixed. Mistral Small 4 is Apache 2.0. Mistral Medium 3.5 is Modified MIT and may not be used above $20M global monthly revenue in the previous month. Mistral Large 4 needs a commercial licence for a Model-as-a-Service business. |
Never assume a whole family shares one licence. Call Llama "open-weight under Meta's community licence", not open source. Our guide to open-weight model licences covers the checks.
How do you test a new model before you switch traffic?
A switch is a release, so treat it like one.
- Build the eval set first. Collect 50 to 200 real, redacted examples per task, each with a pass or fail rule.
- Run the candidate offline. Compare pass rate, format errors, refusals, cost and latency. Tune the prompt for the new model before you judge it.
- Shadow, then canary. Send a copy of live requests to the candidate within the same data class, compare, then move 5% of one task and step up.
- Pin versions. Name the exact model version in policy. A changed provider alias is a silent release.
- Keep the rollback. The old model stays in the policy until the new one has run cleanly for an agreed period.
What should you do first?
- List every LLM call in your product and give each a task name and a data class.
- Put one internal interface in front of the calls, and move prompts into versioned files.
- Build a first eval set of 50 real, redacted cases for your two most expensive tasks.
- Stand up a gateway that logs every call, and add a second model for one low-risk task with a fallback list inside its data class.
- Add a private model for the data class that cannot leave your servers, once the rest is in place.
Mono does not certify anyone's AI systems. For help with this layer, see AI product development and Own Your AI. Adding a first AI feature to a live product? Start with adding AI to an existing SaaS product. The legal reading of data rules goes to counsel.
Sources: Digital Personal Data Protection Act, 2023, Section 16. LiteLLM repository and Router documentation. OpenRouter documentation (model fallbacks, provider selection, FAQ). vLLM documentation and releases page. Llama 4 Community License and Acceptable Use Policy (dev.meta.ai/llama/llama4). Gemma 4 licence and Gemma Terms of Use. Qwen and Mistral model cards and LICENSE files on Hugging Face. Public price pages of OpenAI, Anthropic and Google and the E2E Networks price list, checked 20 November 2026. AWS, Azure and Google Cloud region documentation. IndiaAI compute portal. Last reviewed 20 November 2026.
FAQ
What is a multi-model LLM architecture?
It is a design where your product sends requests through one internal interface and a router chooses among several models, such as a small API model, a top-tier API model and a privately hosted open-weight model. The router decides by data sensitivity, task difficulty, cost and availability, and falls back to another allowed model on failure.
How do you avoid LLM vendor lock-in?
Keep product code free of provider SDK calls and talk to one internal interface. Keep prompts, tool schemas and eval sets in your own repository, log every call in your own store, and record the embedding model on every vector so you can re-index. Test any new model against your own eval set before moving traffic.
Should I use LiteLLM or OpenRouter?
LiteLLM is an open-source gateway you run yourself, so prompts stay inside your network, and it adds virtual keys, spend tracking and load balancing. OpenRouter is a hosted service with one API and one bill, which suits prototypes. A hosted router is another processor that sees prompts, so use a self-hosted gateway for sensitive data.
Can a fallback send personal data to a public API by mistake?
Yes, if the fallback list ignores data class. A request cleared only for a private model can retry on a public API and expose personal data. Define fallback lists per data class, fail closed for sensitive classes, and log which model answered every request.
Do I need a private model to run multiple models?
No. You can route between several API models without any private hosting. A private open-weight model, served for example with vLLM, becomes worth adding when some data should not leave your own servers or region, or when steady volume makes a fixed GPU bill cheaper than per-token pricing.
Does the DPDP Act stop me from routing prompts to foreign model providers?
The Act does not ban foreign AI APIs or require data to stay in India. Section 16 lets the government restrict transfers to notified countries, and none were notified as of November 2026. Hosting in India avoids future restrictions and sector localisation rules. Take the legal reading for your business to counsel.
