Short answer: treat the move as a controlled cutover, not a model swap. Put a gateway in front of your app, build an eval set from your own traffic, pick an open-weight model whose licence fits your business, size the GPU from the model's memory footprint, serve it with vLLM behind your own network, run it in shadow mode, then shift traffic in steps with the old API kept as a rollback. Most of the work is not the serving. It is the eval set, the prompt changes, the capacity plan and the evidence that the new setup is safe to run.
This guide is for engineering leads with a working AI feature on a hosted API such as OpenAI, Anthropic or Gemini. It is engineering guidance, not legal advice. Licence and contract questions go to counsel.
Should you migrate from a hosted API to an open-weight model?
Often the answer is no, or not yet. Open-weight models give you control of where data goes, a model version that does not change under you, and a bill that tracks GPU time instead of tokens. They cost you operations: capacity, patching and on-call. Our cost model in rupees shows that cost alone rarely justifies the move at modest volumes. Migrate when one of these is true:
- Data rules. A customer contract, a regulator or your own policy says prompts must stay on infrastructure you control. Note that the DPDP Act does not require data to stay in India. Hosting in India avoids future restrictions under Section 16 and sector rules on data location, but it is a choice, not a mandate.
- Version control. You need the same model behaviour for years, and hosted models are retired on the provider's schedule.
- Predictable load. Steady traffic keeps GPUs busy, so a fixed monthly cost beats per-token pricing.
- A narrow task. A smaller open model can match a large hosted one on one job. You only know by testing.
If none applies, keep the API. If the data rule covers only part of your traffic, migrate that part.
What are the steps to migrate an AI feature to an open-weight model?
Run the work in this order. Each phase ends with evidence you can show a reviewer.
| Phase | What you do | Evidence at the end |
|---|---|---|
| 1. Inventory | What you do: List every call to the hosted API: prompts, tools, formats, context sizes, volumes | Evidence at the end: A call register |
| 2. Gateway | What you do: Route all calls through a gateway you own, so the model is a config value | Evidence at the end: No direct API calls in app code |
| 3. Eval set | What you do: Build test cases from real traffic, personal data removed, with a pass mark per task | Evidence at the end: A versioned eval set and the API's scores |
| 4. Model and licence | What you do: Shortlist two or three models, read each licence, test them | Evidence at the end: A licence note and score table |
| 5. Host and size | What you do: Choose the host, size the GPUs, pin versions | Evidence at the end: A capacity plan and monthly cost |
| 6. Serve | What you do: Run vLLM in a private network with auth and health checks | Evidence at the end: An endpoint behind the gateway |
| 7. Shadow | What you do: Copy live requests to the new model and compare, without returning its answers | Evidence at the end: A diff report |
| 8. Canary and cutover | What you do: Shift traffic in steps with a tested rollback | Evidence at the end: A signed-off go-live checklist |
How do you choose an open-weight model, and what do the licences allow?

Pick on your eval set first, but check the licence before you invest in a model. Open-weight does not mean open source, and licences are set per model, not per vendor. Qwen and Mistral both publish models under very different terms. A few examples, as checked on 20 November 2026 (read the licence on each model card):
- Apache 2.0: Gemma 4, Qwen3.8-27B and Mistral Small 4.
- Llama 4 (Scout, Maverick): open-weight under Meta's Llama 4 Community License. Companies with more than 700M monthly active users need a licence from Meta, and for multimodal models the Acceptable Use Policy withholds the grant from EU-based individuals and companies.
- Custom conditions: Qwen3.8-Flash-Next restricts Model-as-a-Service and "AI Work Assistant" businesses, Mistral Medium 3.5 has a revenue cap, and Mistral Large 4 restricts Model-as-a-Service use.
- Research or non-commercial only: Qwen-Image-2.1 is non-commercial only, and Mistral-Large-2411 and Codestral-22B are under research or non-production licences.
Keep a copy of the licence text next to the weights you deploy, and re-read it when you change models, because a new release can carry new terms. Our open-weight model licences guide has the full per-model table and a review checklist, and evaluating open-weight models covers the test plan. This is not legal advice; have counsel check your use case against the licence text.
Where can you host an open-weight model in India?
Hosting in India is a design choice you can explain to customers, not a DPDP requirement. Facts below were checked on 20 November 2026.
| Option | India locations and GPUs | Notes |
|---|---|---|
| AWS | India locations and GPUs: Mumbai (ap-south-1, 3 availability zones) and Hyderabad (ap-south-2, 3 zones, opt-in required) | Notes: Check GPU availability per region |
| Azure | India locations and GPUs: Central India (Pune, 3 zones), India South Central (Hyderabad, 3 zones), South India (Chennai) and West India (Mumbai), both without zones | Notes: Zones help replicas |
| Google Cloud | India locations and GPUs: asia-south1 (Mumbai) and asia-south2 (Delhi) | Notes: GPU types differ by zone; check before planning |
| E2E Networks | India locations and GPUs: Its site shows "MeitY Empaneled" and lists B200, H200, H100, A100 and RTX PRO 6000 | Notes: Public list price for an H100 80GB was ₹379.05 an hour on demand and ₹347.70 committed for 3 months or more, ex-GST |
| Yotta Shakti Cloud | India locations and GPUs: NVIDIA H100 and L40S, B200 announced, data centres in Navi Mumbai and Greater Noida | Notes: No public price. The MeitY-empanelled label on its site belongs to Yntraa Cloud, not Shakti. |
At the E2E list price, one H100 running all month (730 hours) costs about ₹2.8 lakh before GST. That is a rough estimate, not a quote. Spot capacity can be taken back, so use it for restartable batch jobs, not live traffic.
What is the IndiaAI compute subsidy, and can you rely on it?
The IndiaAI compute portal offers subsidised rates for approved projects. Users pay the rest. Eligibility follows the portal's criteria. Startups and MSMEs need DPIIT recognition, AI or ML experience, and either ₹50 lakh annual revenue or ₹100 lakh funding. Requests are filed between the 1st and 25th of each month. Requests under 5,000 GPU-hours are auto-approved, and larger ones go to a committee. Approvals are valid for 30 days. The portal lists 14 empanelled providers, and PIB reported 38,000+ GPUs onboarded in March 2026.
Treat it as a monthly, per-request subsidy. It suits a pilot, an eval run or a fine-tune. Do not plan production capacity on it, because a 30-day approval is not a service commitment. Our guide to VPC, on-premises and GPU cloud compares the hosting models, and private AI in India covers local options.
How much GPU memory do you need?

Start with the weights, then add the working memory. A rough estimate for weights is the parameter count times the bytes per parameter: about 2 for BF16, about 1 for FP8 or INT8, and about 0.5 for 4-bit. The KV cache and overhead come on top and grow with context length and requests in flight, so a model that fits at idle can fail under load.
| Model size | BF16 weights | FP8 or INT8 weights | 4-bit weights |
|---|---|---|---|
| 8B parameters | BF16 weights: about 16 GB | FP8 or INT8 weights: about 8 GB | 4-bit weights: about 4 GB |
| 31B parameters | BF16 weights: about 62 GB | FP8 or INT8 weights: about 31 GB | 4-bit weights: about 16 GB |
| 70B parameters | BF16 weights: about 140 GB | FP8 or INT8 weights: about 70 GB | 4-bit weights: about 35 GB |
These are rough estimates, before the KV cache and overhead. We give no speed figures. Measure throughput and latency on your own prompts.
Quantisation shrinks the footprint but trades precision for memory, as the vLLM docs put it. vLLM supports AWQ, GPTQ, bitsandbytes, LLM Compressor (FP8 W8A8, INT8 W8A8 and INT4 W4A16), NVIDIA ModelOpt, AMD Quark, TorchAO and a quantised KV cache. FP8 W8A8 needs Ada, Hopper or AMD GPUs. Run your eval set on the exact quantised build you will deploy.
To size a first deployment:
- Fit the weights on the GPU you can get, with room left. If they only just fit, go to a smaller quantisation or add GPUs.
- Set the context you really need. Use `--max-model-len` to cap prompt plus output. A cap you never reach wastes memory.
- Estimate concurrency from peak requests per second and response time, then load test.
- Split across GPUs with `--tensor-parallel-size` only when the model does not fit on one. The default is 1.
- Plan at least two replicas for production. One GPU is one failure away from an outage.
How do you serve the model with vLLM?
vLLM's latest release was v0.34.0 on 16 November 2026. Pin the version in your container image and upgrade after re-running your eval set. `vllm serve
A first command looks like `vllm serve
| Flag | What it does | Notes |
|---|---|---|
| `--gpu-memory-utilization` | What it does: Share of GPU memory vLLM may use, from 0 to 1 | Notes: Default 1.0 with the extensible KV cache, 0.92 otherwise |
| `--max-model-len` | What it does: Context length, prompt plus output | Notes: Set it to what your feature needs |
| `--tensor-parallel-size` | What it does: Splits the model across GPUs | Notes: Defaults to 1 |
| `--kv-cache-dtype` | What it does: Stores the KV cache in a smaller type such as fp8 | Notes: Test quality; it lowers precision |
| `--enable-prefix-caching` | What it does: Reuses the computed prefix of repeated prompts | Notes: Helps with long shared system prompts |
| `--max-num-seqs` | What it does: Caps how many sequences run at once | Notes: Trades throughput against latency |
Treat the endpoint as an internal service. Run it in a private subnet, require an API key, and let only the gateway reach it. Do not expose the port to the internet. See our guide to LLM observability for what to log and alert on.
What changes in your application code?
The API call barely changes. The behaviour does. Check each of these before the cutover:
- Prompts. Prompts tuned for one model often work worse on another. Expect to rewrite and re-test them.
- Chat template. Each model has its own template and special tokens. Use the one from its model card.
- Tool calling and structured output. Support depends on the model and on the parser settings in vLLM. Test your exact schemas, measure the format failure rate, and add validation and a retry.
- Tokenisation. Token counts differ between models. Recount your longest prompts.
- Embeddings. A new embedding model means re-embedding and re-indexing the whole corpus. Treat it as a separate project.
- Safety features. Hosted APIs may include moderation you used without noticing. Decide what replaces it.
- Personal data. Your logs now hold the prompts. Apply redaction, access limits and retention. See where personal data lives in an AI feature.
Which gateway should sit in front of the model?
Use one, and own it. A gateway lets you change the model by config, split traffic, set fallbacks and log every call in one place. It is also your rollback switch. LiteLLM is an open-source AI gateway that calls 100 or more providers in the OpenAI format. It ships as an SDK and a proxy. The proxy adds virtual keys, spend tracking, guardrails and load balancing. It is MIT licensed except for the `litellm/enterprise/` folder, and its latest release was v1.110.2 on 20 November 2026. Its router supports ordered fallbacks, `context_window_fallbacks` and `content_policy_fallbacks`.
Hosted routers such as OpenRouter offer fallbacks and provider controls, including `data_collection: "deny"` and `zdr`, but a hosted router is another processor that sees prompts. For DPDP-sensitive data, use a self-hosted gateway in front of India-hosted models. Whatever you choose, configure fallbacks so traffic only moves to models that are allowed to see the same data. See our multi-model architecture guide for routing rules.
How do you roll out the new model safely?
Move in four stages and keep the old route working until the last:
- Offline eval. Score the candidate on your eval set against the current API. Set a pass mark per task before you look at the results. Include long inputs and adversarial prompts.
- Shadow mode. Copy a share of live requests to the new model. Log its answers, but return the API's. Compare quality, format failures, latency and cost.
- Canary. Return the new model's answers for a small slice of traffic, such as internal users first. Raise the share in steps, with the rollback trigger set up front.
- Cutover. Move all traffic, but keep the gateway route to the old API ready for an agreed period. If the old API may not see the data, your fallback is a second replica of the new model.
If your feature uses retrieval, test access control as part of the eval, since a new model can quote what it is given. See our note on permission-aware retrieval for the checks.
What is the go-live checklist for a private AI deployment?
Use this list as a gate. Each line needs an owner and evidence.
| Area | Check |
|---|---|
| Scope | Check: Call register complete; all calls routed through the gateway |
| Quality | Check: Eval set versioned; pass marks met on the exact build you will deploy |
| Licence | Check: Licence text saved with the weights; use case checked against it |
| Capacity | Check: Weights plus KV cache fit at peak; two or more replicas; load test passed |
| Serving | Check: vLLM version pinned; settings recorded; health check and restart policy in place |
| Network | Check: Private subnet; API key required; only the gateway can reach it |
| Data | Check: Logs redacted; retention set; access limited |
| Fallbacks | Check: Targets allowed to see the same data; tested by taking the primary down |
| Monitoring | Check: Latency, queue depth, errors, GPU memory and format failures, with alerts |
| Rollback | Check: Route back to the old model tested; trigger and owner named |
| Operations | Check: On-call named; patching plan for vLLM, drivers and base image |
| Evidence | Check: Shadow and canary reports saved; sign-off recorded |
Staffing is part of the plan: someone must own GPU capacity, upgrades and incidents after go-live. If you want a partner to build the private deployment, see our Own Your AI service. Mono does not certify anyone, and this checklist is engineering evidence, not an audit opinion.
Sources: vLLM documentation (online serving, engine arguments, quantisation) and vLLM releases, checked 20 November 2026. Model licences from the model cards and licence files of Meta (Llama 4 Community License and Acceptable Use Policy), Google (Gemma 4 licence, Gemma Terms of Use), Qwen and Mistral on Hugging Face. AWS, Microsoft Azure and Google Cloud region documentation. E2E Networks price list and site, Yotta Shakti Cloud site, IndiaAI compute portal (last updated 14 November 2026) and PIB press note of 25 March 2026. LiteLLM repository and Router documentation, OpenRouter documentation. Digital Personal Data Protection Act, 2023, Section 16. Last reviewed 20 November 2026.
FAQ
How do I migrate from the OpenAI API to a self-hosted LLM?
Route every call through a gateway you own, build an eval set from real traffic, pick an open-weight model whose licence fits your use, size the GPU, and serve it with vLLM, which exposes an OpenAI-compatible API. Then run the new model in shadow mode, shift traffic in steps, and keep a tested rollback route until you are sure.
Is an open-weight model the same as an open-source model?
No. Open-weight means you can download the weights, but the licence decides what you may do with them. Llama is released under Meta's community licence. Qwen and Mistral publish some models under Apache 2.0 and others under custom or restricted terms. Read the licence on each model card before you build on it.
How much GPU memory do I need for an open-weight model?
As a rough estimate, weights need about the parameter count times the bytes per parameter: about 2 for BF16, 1 for FP8 or INT8 and 0.5 for 4-bit. A 31B model is about 62 GB in BF16 and about 16 GB at 4-bit. The KV cache and overhead come on top and grow with context length and concurrency, so always load test.
Does DPDP require me to host my AI model in India?
No. The DPDP Act does not require data to stay in India. Section 16 lets the government restrict transfers to notified countries, and none were notified as of November 2026. Sector rules, such as RBI payment-data storage, can be stricter. Hosting in India is a sensible way to avoid future restrictions and answer customer questions.
Can I use the IndiaAI compute subsidy for production inference?
It is better suited to pilots, evals and training runs. Requests are filed between the 1st and 25th of each month, and approvals are valid for 30 days. Subsidised rates apply to approved projects, and eligibility follows the portal's criteria. Do not plan production capacity on a monthly, per-request approval.
