← All news and guides
Private AI

How to Migrate an AI Feature From a Hosted API to an Open-Weight Model

A step-by-step plan to move an AI feature off a hosted API: model licences, GPU sizing, vLLM settings, India hosting, shadow tests and a go-live checklist.

Mono Team ·

A long aisle between rows of server racks in a quiet data hall, cool blue-white light on a polished floor, bundled cables overhead, no screens lit.

Short answer: treat the move as a controlled cutover, not a model swap. Put a gateway in front of your app, build an eval set from your own traffic, pick an open-weight model whose licence fits your business, size the GPU from the model's memory footprint, serve it with vLLM behind your own network, run it in shadow mode, then shift traffic in steps with the old API kept as a rollback. Most of the work is not the serving. It is the eval set, the prompt changes, the capacity plan and the evidence that the new setup is safe to run.

This guide is for engineering leads with a working AI feature on a hosted API such as OpenAI, Anthropic or Gemini. It is engineering guidance, not legal advice. Licence and contract questions go to counsel.

Should you migrate from a hosted API to an open-weight model?

Often the answer is no, or not yet. Open-weight models give you control of where data goes, a model version that does not change under you, and a bill that tracks GPU time instead of tokens. They cost you operations: capacity, patching and on-call. Our cost model in rupees shows that cost alone rarely justifies the move at modest volumes. Migrate when one of these is true:

  • Data rules. A customer contract, a regulator or your own policy says prompts must stay on infrastructure you control. Note that the DPDP Act does not require data to stay in India. Hosting in India avoids future restrictions under Section 16 and sector rules on data location, but it is a choice, not a mandate.
  • Version control. You need the same model behaviour for years, and hosted models are retired on the provider's schedule.
  • Predictable load. Steady traffic keeps GPUs busy, so a fixed monthly cost beats per-token pricing.
  • A narrow task. A smaller open model can match a large hosted one on one job. You only know by testing.

If none applies, keep the API. If the data rule covers only part of your traffic, migrate that part.

What are the steps to migrate an AI feature to an open-weight model?

Run the work in this order. Each phase ends with evidence you can show a reviewer.

PhaseWhat you doEvidence at the end
1. InventoryWhat you do: List every call to the hosted API: prompts, tools, formats, context sizes, volumesEvidence at the end: A call register
2. GatewayWhat you do: Route all calls through a gateway you own, so the model is a config valueEvidence at the end: No direct API calls in app code
3. Eval setWhat you do: Build test cases from real traffic, personal data removed, with a pass mark per taskEvidence at the end: A versioned eval set and the API's scores
4. Model and licenceWhat you do: Shortlist two or three models, read each licence, test themEvidence at the end: A licence note and score table
5. Host and sizeWhat you do: Choose the host, size the GPUs, pin versionsEvidence at the end: A capacity plan and monthly cost
6. ServeWhat you do: Run vLLM in a private network with auth and health checksEvidence at the end: An endpoint behind the gateway
7. ShadowWhat you do: Copy live requests to the new model and compare, without returning its answersEvidence at the end: A diff report
8. Canary and cutoverWhat you do: Shift traffic in steps with a tested rollbackEvidence at the end: A signed-off go-live checklist

How do you choose an open-weight model, and what do the licences allow?

A tidy wooden workbench with a row of identical sealed metal tins of different sizes, each with a blank plain lid, warm side light from a window, shallow depth of field.

Pick on your eval set first, but check the licence before you invest in a model. Open-weight does not mean open source, and licences are set per model, not per vendor. Qwen and Mistral both publish models under very different terms. A few examples, as checked on 20 November 2026 (read the licence on each model card):

  • Apache 2.0: Gemma 4, Qwen3.8-27B and Mistral Small 4.
  • Llama 4 (Scout, Maverick): open-weight under Meta's Llama 4 Community License. Companies with more than 700M monthly active users need a licence from Meta, and for multimodal models the Acceptable Use Policy withholds the grant from EU-based individuals and companies.
  • Custom conditions: Qwen3.8-Flash-Next restricts Model-as-a-Service and "AI Work Assistant" businesses, Mistral Medium 3.5 has a revenue cap, and Mistral Large 4 restricts Model-as-a-Service use.
  • Research or non-commercial only: Qwen-Image-2.1 is non-commercial only, and Mistral-Large-2411 and Codestral-22B are under research or non-production licences.

Keep a copy of the licence text next to the weights you deploy, and re-read it when you change models, because a new release can carry new terms. Our open-weight model licences guide has the full per-model table and a review checklist, and evaluating open-weight models covers the test plan. This is not legal advice; have counsel check your use case against the licence text.

Where can you host an open-weight model in India?

Hosting in India is a design choice you can explain to customers, not a DPDP requirement. Facts below were checked on 20 November 2026.

OptionIndia locations and GPUsNotes
AWSIndia locations and GPUs: Mumbai (ap-south-1, 3 availability zones) and Hyderabad (ap-south-2, 3 zones, opt-in required)Notes: Check GPU availability per region
AzureIndia locations and GPUs: Central India (Pune, 3 zones), India South Central (Hyderabad, 3 zones), South India (Chennai) and West India (Mumbai), both without zonesNotes: Zones help replicas
Google CloudIndia locations and GPUs: asia-south1 (Mumbai) and asia-south2 (Delhi)Notes: GPU types differ by zone; check before planning
E2E NetworksIndia locations and GPUs: Its site shows "MeitY Empaneled" and lists B200, H200, H100, A100 and RTX PRO 6000Notes: Public list price for an H100 80GB was ₹379.05 an hour on demand and ₹347.70 committed for 3 months or more, ex-GST
Yotta Shakti CloudIndia locations and GPUs: NVIDIA H100 and L40S, B200 announced, data centres in Navi Mumbai and Greater NoidaNotes: No public price. The MeitY-empanelled label on its site belongs to Yntraa Cloud, not Shakti.

At the E2E list price, one H100 running all month (730 hours) costs about ₹2.8 lakh before GST. That is a rough estimate, not a quote. Spot capacity can be taken back, so use it for restartable batch jobs, not live traffic.

What is the IndiaAI compute subsidy, and can you rely on it?

The IndiaAI compute portal offers subsidised rates for approved projects. Users pay the rest. Eligibility follows the portal's criteria. Startups and MSMEs need DPIIT recognition, AI or ML experience, and either ₹50 lakh annual revenue or ₹100 lakh funding. Requests are filed between the 1st and 25th of each month. Requests under 5,000 GPU-hours are auto-approved, and larger ones go to a committee. Approvals are valid for 30 days. The portal lists 14 empanelled providers, and PIB reported 38,000+ GPUs onboarded in March 2026.

Treat it as a monthly, per-request subsidy. It suits a pilot, an eval run or a fine-tune. Do not plan production capacity on it, because a 30-day approval is not a service commitment. Our guide to VPC, on-premises and GPU cloud compares the hosting models, and private AI in India covers local options.

How much GPU memory do you need?

A brass balance scale with small weights on a plain workshop bench, soft window light from the left, a blurred shelf of tools behind, shallow depth of field.

Start with the weights, then add the working memory. A rough estimate for weights is the parameter count times the bytes per parameter: about 2 for BF16, about 1 for FP8 or INT8, and about 0.5 for 4-bit. The KV cache and overhead come on top and grow with context length and requests in flight, so a model that fits at idle can fail under load.

Model sizeBF16 weightsFP8 or INT8 weights4-bit weights
8B parametersBF16 weights: about 16 GBFP8 or INT8 weights: about 8 GB4-bit weights: about 4 GB
31B parametersBF16 weights: about 62 GBFP8 or INT8 weights: about 31 GB4-bit weights: about 16 GB
70B parametersBF16 weights: about 140 GBFP8 or INT8 weights: about 70 GB4-bit weights: about 35 GB

These are rough estimates, before the KV cache and overhead. We give no speed figures. Measure throughput and latency on your own prompts.

Quantisation shrinks the footprint but trades precision for memory, as the vLLM docs put it. vLLM supports AWQ, GPTQ, bitsandbytes, LLM Compressor (FP8 W8A8, INT8 W8A8 and INT4 W4A16), NVIDIA ModelOpt, AMD Quark, TorchAO and a quantised KV cache. FP8 W8A8 needs Ada, Hopper or AMD GPUs. Run your eval set on the exact quantised build you will deploy.

To size a first deployment:

  • Fit the weights on the GPU you can get, with room left. If they only just fit, go to a smaller quantisation or add GPUs.
  • Set the context you really need. Use `--max-model-len` to cap prompt plus output. A cap you never reach wastes memory.
  • Estimate concurrency from peak requests per second and response time, then load test.
  • Split across GPUs with `--tensor-parallel-size` only when the model does not fit on one. The default is 1.
  • Plan at least two replicas for production. One GPU is one failure away from an outage.

How do you serve the model with vLLM?

vLLM's latest release was v0.34.0 on 16 November 2026. Pin the version in your container image and upgrade after re-running your eval set. `vllm serve ` starts an OpenAI-compatible server that covers the Completions, Chat Completions, Responses and Embeddings APIs. That is why many apps move with a small change: point the client's base URL at your server and change the model name.

A first command looks like `vllm serve --port 8000 --max-model-len 16384 --gpu-memory-utilization 0.90`, with the context length and memory share set from your sizing. These are the flags that matter most:

FlagWhat it doesNotes
`--gpu-memory-utilization`What it does: Share of GPU memory vLLM may use, from 0 to 1Notes: Default 1.0 with the extensible KV cache, 0.92 otherwise
`--max-model-len`What it does: Context length, prompt plus outputNotes: Set it to what your feature needs
`--tensor-parallel-size`What it does: Splits the model across GPUsNotes: Defaults to 1
`--kv-cache-dtype`What it does: Stores the KV cache in a smaller type such as fp8Notes: Test quality; it lowers precision
`--enable-prefix-caching`What it does: Reuses the computed prefix of repeated promptsNotes: Helps with long shared system prompts
`--max-num-seqs`What it does: Caps how many sequences run at onceNotes: Trades throughput against latency

Treat the endpoint as an internal service. Run it in a private subnet, require an API key, and let only the gateway reach it. Do not expose the port to the internet. See our guide to LLM observability for what to log and alert on.

What changes in your application code?

The API call barely changes. The behaviour does. Check each of these before the cutover:

  • Prompts. Prompts tuned for one model often work worse on another. Expect to rewrite and re-test them.
  • Chat template. Each model has its own template and special tokens. Use the one from its model card.
  • Tool calling and structured output. Support depends on the model and on the parser settings in vLLM. Test your exact schemas, measure the format failure rate, and add validation and a retry.
  • Tokenisation. Token counts differ between models. Recount your longest prompts.
  • Embeddings. A new embedding model means re-embedding and re-indexing the whole corpus. Treat it as a separate project.
  • Safety features. Hosted APIs may include moderation you used without noticing. Decide what replaces it.
  • Personal data. Your logs now hold the prompts. Apply redaction, access limits and retention. See where personal data lives in an AI feature.

Which gateway should sit in front of the model?

Use one, and own it. A gateway lets you change the model by config, split traffic, set fallbacks and log every call in one place. It is also your rollback switch. LiteLLM is an open-source AI gateway that calls 100 or more providers in the OpenAI format. It ships as an SDK and a proxy. The proxy adds virtual keys, spend tracking, guardrails and load balancing. It is MIT licensed except for the `litellm/enterprise/` folder, and its latest release was v1.110.2 on 20 November 2026. Its router supports ordered fallbacks, `context_window_fallbacks` and `content_policy_fallbacks`.

Hosted routers such as OpenRouter offer fallbacks and provider controls, including `data_collection: "deny"` and `zdr`, but a hosted router is another processor that sees prompts. For DPDP-sensitive data, use a self-hosted gateway in front of India-hosted models. Whatever you choose, configure fallbacks so traffic only moves to models that are allowed to see the same data. See our multi-model architecture guide for routing rules.

How do you roll out the new model safely?

Move in four stages and keep the old route working until the last:

  • Offline eval. Score the candidate on your eval set against the current API. Set a pass mark per task before you look at the results. Include long inputs and adversarial prompts.
  • Shadow mode. Copy a share of live requests to the new model. Log its answers, but return the API's. Compare quality, format failures, latency and cost.
  • Canary. Return the new model's answers for a small slice of traffic, such as internal users first. Raise the share in steps, with the rollback trigger set up front.
  • Cutover. Move all traffic, but keep the gateway route to the old API ready for an agreed period. If the old API may not see the data, your fallback is a second replica of the new model.

If your feature uses retrieval, test access control as part of the eval, since a new model can quote what it is given. See our note on permission-aware retrieval for the checks.

What is the go-live checklist for a private AI deployment?

Use this list as a gate. Each line needs an owner and evidence.

AreaCheck
ScopeCheck: Call register complete; all calls routed through the gateway
QualityCheck: Eval set versioned; pass marks met on the exact build you will deploy
LicenceCheck: Licence text saved with the weights; use case checked against it
CapacityCheck: Weights plus KV cache fit at peak; two or more replicas; load test passed
ServingCheck: vLLM version pinned; settings recorded; health check and restart policy in place
NetworkCheck: Private subnet; API key required; only the gateway can reach it
DataCheck: Logs redacted; retention set; access limited
FallbacksCheck: Targets allowed to see the same data; tested by taking the primary down
MonitoringCheck: Latency, queue depth, errors, GPU memory and format failures, with alerts
RollbackCheck: Route back to the old model tested; trigger and owner named
OperationsCheck: On-call named; patching plan for vLLM, drivers and base image
EvidenceCheck: Shadow and canary reports saved; sign-off recorded

Staffing is part of the plan: someone must own GPU capacity, upgrades and incidents after go-live. If you want a partner to build the private deployment, see our Own Your AI service. Mono does not certify anyone, and this checklist is engineering evidence, not an audit opinion.

Sources: vLLM documentation (online serving, engine arguments, quantisation) and vLLM releases, checked 20 November 2026. Model licences from the model cards and licence files of Meta (Llama 4 Community License and Acceptable Use Policy), Google (Gemma 4 licence, Gemma Terms of Use), Qwen and Mistral on Hugging Face. AWS, Microsoft Azure and Google Cloud region documentation. E2E Networks price list and site, Yotta Shakti Cloud site, IndiaAI compute portal (last updated 14 November 2026) and PIB press note of 25 March 2026. LiteLLM repository and Router documentation, OpenRouter documentation. Digital Personal Data Protection Act, 2023, Section 16. Last reviewed 20 November 2026.

FAQ

How do I migrate from the OpenAI API to a self-hosted LLM?

Route every call through a gateway you own, build an eval set from real traffic, pick an open-weight model whose licence fits your use, size the GPU, and serve it with vLLM, which exposes an OpenAI-compatible API. Then run the new model in shadow mode, shift traffic in steps, and keep a tested rollback route until you are sure.

Is an open-weight model the same as an open-source model?

No. Open-weight means you can download the weights, but the licence decides what you may do with them. Llama is released under Meta's community licence. Qwen and Mistral publish some models under Apache 2.0 and others under custom or restricted terms. Read the licence on each model card before you build on it.

How much GPU memory do I need for an open-weight model?

As a rough estimate, weights need about the parameter count times the bytes per parameter: about 2 for BF16, 1 for FP8 or INT8 and 0.5 for 4-bit. A 31B model is about 62 GB in BF16 and about 16 GB at 4-bit. The KV cache and overhead come on top and grow with context length and concurrency, so always load test.

Does DPDP require me to host my AI model in India?

No. The DPDP Act does not require data to stay in India. Section 16 lets the government restrict transfers to notified countries, and none were notified as of November 2026. Sector rules, such as RBI payment-data storage, can be stricter. Hosting in India is a sensible way to avoid future restrictions and answer customer questions.

Can I use the IndiaAI compute subsidy for production inference?

It is better suited to pilots, evals and training runs. Requests are filed between the 1st and 25th of each month, and approvals are valid for 30 days. Subsidised rates apply to approved projects, and eligibility follows the portal's criteria. Do not plan production capacity on a monthly, per-request approval.

Planning a move off a hosted AI API?