← All news and guides
Private AI

Do You Need an MLOps Team to Self-Host an LLM? What Private AI Really Takes

Do you need an MLOps team to self-host an LLM? The nine jobs, GPU sizing, India hosting, licence checks and when a managed route is the better choice.

Mono Team ·

A single narrow rack of servers in a small, tidy equipment room, a rolling stool pushed in beside it, soft cool light.

Short answer: no, most teams that self-host one open-weight model for internal or moderate-volume use do not need a dedicated MLOps team. They do need named owners for about nine jobs, and at least two people who can fix serving at 2 a.m. One platform or DevOps engineer who is comfortable with GPUs, plus a backup, can run a single-model private deployment. You need a real team when you fine-tune or train, run many models, serve customers under an uptime promise, or operate a fleet of GPUs. If you cannot staff the jobs, buy them as a service rather than leave them unowned.

This guide is for CTOs and engineering leads who like the idea of private AI but worry about who will run it. It lists the work, shows how staffing changes with the setup, covers the serving layer, GPU sizing, hosting in India and licence checks, and sets out the build, hire or managed choice. It is engineering guidance, not legal advice. Legal readings for your business go to counsel.

What does it take to self-host an LLM?

Self-hosting means you run the model server, not just call one. Most teams serve an existing open-weight model. They do not train one. That removes the hardest ML work, but it leaves ordinary production work with a GPU attached. Here are the jobs, and who in a normal engineering team can own each.

JobWhat it involvesWho can own it
1. ServingWhat it involves: Run the inference server (usually vLLM), pin its version, set memory and context flags, handle restarts.Who can own it: Platform or DevOps engineer
2. CapacityWhat it involves: Size GPUs for the model and the number of users, choose quantisation, watch idle cost.Who can own it: Same engineer, with a budget owner
3. Access and gatewayWhat it involves: API keys per team, rate limits, routing, fallbacks, spend limits.Who can own it: Backend or platform engineer
4. Model changesWhat it involves: Move to a new model or version only after running your own eval set. Version your prompts.Who can own it: Backend engineer with eval discipline, or an ML engineer
5. SecurityWhat it involves: Patch the OS, GPU drivers, container images and the serving software. Block outbound traffic you do not need. Rotate secrets.Who can own it: DevOps or security engineer
6. Observability and logsWhat it involves: Latency, errors, queue depth, GPU memory. Prompt and response logs with personal data masked.Who can own it: Platform engineer
7. Data and retrievalWhat it involves: If you add retrieval over your documents: the index, who may see which document, deletion.Who can own it: Backend or data engineer
8. IncidentsWhat it involves: A runbook, an escalation path and a rota. Rehearse a model that is down, slow or wrong.Who can own it: Engineering lead
9. Licences and vendor termsWhat it involves: Check the licence of each model you serve and the terms of your GPU provider.Who can own it: Engineering lead with counsel

None of these needs a research degree. They are the jobs you already do for a database or a payment service. The new parts are GPUs, model changes and evals.

Do you need an MLOps team, or one good engineer?

A wall-mounted wooden key rack beside a doorway with several sets of plain keys hanging from brass hooks, soft daylight from the side, shallow depth of field.

It depends on what you run, not on how clever the model is. This is a rule of thumb from engineering judgement, not a survey.

Your setupStaffing pattern that usually worksMain risk
An internal assistant for staff, one model, one or two GPUsStaffing pattern that usually works: One platform or DevOps engineer owns it part-time. A second person is the named backup.Main risk: The one person who knows it goes on leave or leaves.
A customer-facing feature with an uptime promiseStaffing pattern that usually works: A rota of two to three people who can fix serving. A failover path: a second node, or a fallback model cleared for the same data.Main risk: An outage that you cannot fix out of hours.
Retrieval over company documentsStaffing pattern that usually works: As above, plus a backend or data engineer who owns the index and the permission checks.Main risk: A person sees a document they should not.
Fine-tuning or training your own modelsStaffing pattern that usually works: Add ML engineering: data preparation, training runs, evals, a model registry.Main risk: Evals that do not exist, so nobody knows if the new model is better.
Several models, several tenants or a GPU fleetStaffing pattern that usually works: A small platform team with a clear owner.Main risk: Cost and drift that nobody tracks.

The pattern: staff for the failure you cannot tolerate. A fixed pool of GPUs bills whether anyone uses it or not, and it does not fix itself. If an outage hurts customers, you need people on a rota. If it only slows your own staff, a careful part-time owner is enough.

What does an MLOps or LLM engineer cost in India?

There is no clean market price for "the person who runs GPU inference". The closest public figure is for the broad title "AI Engineer". AmbitionBox's self-reported data shows a typical salary of ₹14.8 to 16.4 lakh a year for 1 to 6 years of experience, from about 5,600 salaries, updated 19 November 2026. That is salary only, before employer costs, and someone who can run GPU inference in production may cost more.

Take the midpoint, ₹15.6 lakh a year, which is about ₹1.3 lakh a month. Two such people come to about ₹2.6 lakh a month in salary. For comparison, E2E Networks' public price list shows an NVIDIA H100 80GB at ₹379.05 an hour on demand, ex-GST, checked 20 November 2026. Running one all month (730 hours) is about ₹2.77 lakh. So a minimum two-person cover can cost about as much as one GPU, and that is before storage, networking, monitoring and tax. Our private LLM vs API cost model puts these numbers into a break-even calculation.

When you hire, look for a platform or DevOps engineer who has run production services and is willing to learn GPUs. Add ML skill if you fine-tune.

What does the serving layer actually look like?

The usual route is an open-weight model served with vLLM, whose latest release at our check was v0.34.0 on 16 November 2026. Running `vllm serve --port 8000` starts an OpenAI-compatible server for the Completions, Chat Completions, Responses and Embeddings APIs. Your applications and gateway then treat it like any hosted provider.

A few flags decide whether the model fits and how it behaves:

  • `--gpu-memory-utilization` sets the share of GPU memory vLLM may use, from 0 to 1.
  • `--max-model-len` sets the context length, which is the prompt plus the output.
  • `--tensor-parallel-size` splits the model across several GPUs. It defaults to 1.
  • `--kv-cache-dtype`, `--enable-prefix-caching` and `--max-num-seqs` matter once you tune for real load.

vLLM supports quantisation formats such as AWQ, GPTQ, bitsandbytes and FP8 or INT8 through LLM Compressor. Its docs say quantisation "trades off model precision for smaller memory footprint", so test quality on your own tasks before you adopt it. FP8 W8A8 needs Ada, Hopper or AMD GPUs.

In front of the server, most teams add a gateway. LiteLLM is an open-source gateway that exposes 100+ providers in the OpenAI format, with virtual keys, spend tracking, guardrails and load balancing in its proxy. It is MIT licensed, except the `litellm/enterprise/` folder, and its latest release at our check was v1.110.2 on 20 November 2026. A hosted router is another processor that sees your prompts, so for DPDP-sensitive data use a self-hosted gateway in front of India-hosted models. Our multi-model architecture guide covers routing and fallbacks.

How do you size GPUs without a benchmarking team?

Start with arithmetic, then confirm with a load test. This is a rough estimate. Weights take about the number of parameters times the bytes per parameter: 2 bytes in BF16, 1 byte in FP8 or INT8, and about 0.5 bytes at 4-bit.

Model sizeBF16 (2 bytes)FP8 or INT8 (1 byte)4-bit (about 0.5 bytes)
31B parametersBF16 (2 bytes): about 62 GBFP8 or INT8 (1 byte): about 31 GB4-bit (about 0.5 bytes): about 16 GB

These are weights only. The KV cache and overhead come on top, and they grow with context length and the number of users at once. That is why `--max-model-len` and `--max-num-seqs` are capacity settings, not just tuning. We publish no throughput or latency figures, because they depend on your prompts and traffic. Run your own test before you commit to a GPU count. Our VPC, on-premise and GPU cloud comparison has a fuller sizing table and monthly GPU rental costs.

What must be monitored, patched and logged every week?

A close view of a row of analogue dials on a plain grey industrial panel, needles resting at different positions, no numbers or text legible, soft side light, shallow depth of field.

This is the recurring work that a part-time owner has to fit in. It is small if it is automated and large if it is not.

  • Health. Alert on errors, latency, queue depth and GPU memory. A model that is up but slow is an outage to users.
  • Version pins. Pin the model and the vLLM version. Upgrade on purpose, in staging first, with your eval set.
  • Patching. Keep the OS, GPU drivers, base images and serving software current. Scan images before you deploy.
  • Egress. Allow the model server to reach only what it needs. A private model that can call the internet is less private.
  • Logs. Rule 6(1)(e) of the DPDP Rules, which applies from 13 May 2027, asks fiduciaries to retain such logs and personal data for a period of one year, unless another law requires otherwise, so that unauthorised access can be detected and investigated. Mask personal data before it is written, or your log store becomes a second copy of customer data. See our guides to LLM observability and PII redaction for LLMs.
  • Incidents. CERT-In's reporting window is 6 hours, and its directions ask for logs to be kept in India for 180 days. Whether and how these apply to you is a question for counsel. Write down who decides and who calls.

If your private AI also searches company documents, add the retrieval layer to the list. Treat access control as a query-time check. Postgres row-level security can restrict which rows each user can see, but superusers and `BYPASSRLS` roles always bypass it, and owners bypass it unless `FORCE ROW LEVEL SECURITY` is set. With pgvector's approximate indexes, filtering is applied after the index is scanned, so a filtered query can return too few rows. Iterative scans, added in 0.8.0, address this.

Where can you host in India, and does it change the staffing?

It does. The more of the stack you own, the more people you need. Rented GPUs in an Indian cloud region (AWS, Azure and Google Cloud all have them) or from an Indian GPU cloud such as E2E Networks or Yotta Shakti Cloud leave the hardware to someone else, and your team runs the serving stack, security and monitoring. Your own servers add procurement, power, cooling, spares and repairs to the list. GPU types differ by region and zone, so check before planning.

IndiaAI's compute portal offers subsidised rates for approved projects through empanelled providers. It is a monthly, per-request subsidy, so do not plan production capacity on it. Eligibility follows the portal's criteria. Our VPC, on-premise and GPU cloud comparison covers the regions, providers, IndiaAI process and trade-offs in detail, and the Own Your AI India page explains how we think about hosting for DPDP.

On the law: DPDP does not require data to stay in India. Section 16 lets the government restrict transfers to notified countries, and none were notified as of November 2026. Hosting in India avoids future restrictions and sector rules, such as RBI payment-data storage.

Which model licence checks does someone have to own?

Licences are set per model, not per vendor, and they change between releases. Someone must check the licence before a model goes into production, and again before each upgrade. Read the licence on each model card. This table is a summary, not legal advice.

FamilyWhat to check
Llama 4 (Scout, Maverick)What to check: Llama 4 Community License, effective 5 April 2025. Companies with more than 700M monthly active users in the preceding calendar month must request a licence from Meta. Distributors must display "Built with Llama", and derivative model names must start with "Llama". The Acceptable Use Policy applies, and for multimodal models it withholds the licence grant from EU-based individuals and companies, though end users of products built on them are exempt. It is open-weight under Meta's community licence, not open source.
Gemma 4What to check: Apache 2.0. Earlier Gemma versions use the Gemma Terms of Use, which pass restrictions downstream.
QwenWhat to check: Mixed. Qwen3.8-27B is Apache 2.0. Qwen3.8-Flash-Next uses the Qwen Community License 1.0, which adds display and separate-licence conditions. Qwen-Image-2.1 is non-commercial only.
MistralWhat to check: Mixed. Mistral Small 4 is Apache 2.0. Mistral Medium 3.5 is Modified MIT and may not be used if your company's global monthly revenue was above $20M in the previous month.

So never say "Qwen is Apache 2.0" or "Mistral is open source" as a blanket statement. Name the model and its licence. Our open-weight model licence guide goes further.

Should you build it, hire for it, or buy it managed?

There are three honest routes. Pick by who will answer the pager.

RouteGood whenWatch for
Run it yourselfGood when: You already have a strong platform team, steady traffic and a reason to own every layer.Watch for: Staff dependency. A fixed GPU bill. An operations job that never ends.
Run it with a partner on your infrastructureGood when: You want the model in your own accounts or servers but not the operations. The partner builds it and hands it over, and may stay on to run it.Watch for: Clear exit terms: documentation, access, and what happens if you leave.
Use a hosted APIGood when: Your data allows it, traffic is bursty or low, and you want no GPUs to run.Watch for: Prompts leave your network. Prices and model versions change on the vendor's schedule.

Mono's Own Your AI service is the middle route. We build private AI in your own cloud account or on your servers, and hand it over with training and documentation. Your team can run it from there, or Mono can stay on with a monthly support plan covering monitoring, model updates and fixes. Whichever route you pick, ask these questions in writing: where does each part run, which model under what licence, who updates it, and what happens when we leave.

What should you do first?

  • Write down the nine jobs and put a name next to each. Gaps are your hiring plan.
  • Name a backup for every critical job. A single owner is a risk, not a plan.
  • Decide the outage you cannot tolerate. That sets whether you need a rota and a failover path.
  • Run a short pilot on rented GPUs with real prompts. Measure memory, concurrency and quality before you buy or commit.
  • Write a one-page runbook: how to restart, how to roll back a model, who to call, what to log.

This article is general information, not legal advice. How the DPDP Act and sector rules apply to your hosting and logging choices is a question for your lawyer.

Sources: vLLM documentation: online serving, engine arguments and quantisation (docs.vllm.ai), and the vLLM releases page, checked 20 November 2026. LiteLLM repository (github.com/BerriAI/litellm). Llama 4 Community License and Acceptable Use Policy (dev.meta.ai/llama/llama4). Gemma 4 licence (ai.google.dev/gemma/docs/gemma_4_license) and Gemma Terms of Use. Qwen and Mistral model cards and LICENSE files on Hugging Face. Cloud region lists: AWS, Microsoft Azure and Google Cloud documentation. E2E Networks (e2enetworks.com, including pricing.json) and Yotta Shakti Cloud (shakticloud.ai), checked 20 November 2026. IndiaAI compute portal (compute.indiaai.gov.in) and PIB press note, 25 March 2026. AmbitionBox, AI Engineer salary in India, self-reported, updated 19 November 2026. PostgreSQL documentation on row security and the pgvector repository. Digital Personal Data Protection Act, 2023, Section 16. Digital Personal Data Protection Rules, 2025, Rule 6. CERT-In directions on incident reporting. Prices and licences change often. Last reviewed 20 November 2026.

FAQ

Do you need an MLOps engineer to self-host an LLM?

Not always. A single-model private deployment for internal use can be run by a platform or DevOps engineer who learns GPUs, with a named backup. You need a proper MLOps or platform team if you fine-tune models, run many models or tenants, or serve customers under an uptime promise. The work is mostly ordinary production operations with a GPU attached.

What does it take to run a private LLM in production?

Nine jobs need owners: serving, capacity, gateway and access, model changes with evals, security patching, observability and logs, retrieval and permissions, incident response, and licence checks. Most teams serve an existing open-weight model with vLLM and do not train one. The recurring effort is monitoring, patching, upgrading models safely and keeping a backup person who can fix it.

How much GPU memory does a self-hosted model need?

As a rough estimate, weights take about parameters times bytes per parameter: 2 bytes in BF16, 1 in FP8 or INT8 and about 0.5 at 4-bit. A 31B model is about 62 GB in BF16 and about 16 GB at 4-bit. The KV cache and overhead come on top and grow with context length and concurrent users, so run a load test with your own prompts.

Can I host a private LLM in India without buying hardware?

Yes. You can rent GPUs in Indian cloud regions such as AWS Mumbai and Hyderabad, Azure Central India and India South Central, and GCP Mumbai and Delhi, or from Indian GPU clouds such as E2E Networks and Yotta Shakti Cloud. GPU types differ by region and zone, so check availability before you plan. IndiaAI offers subsidised rates for approved projects, but treat it as a monthly, per-request subsidy.

Does the DPDP Act require me to self-host an LLM?

No. DPDP does not require data to stay in India or require you to run your own model. Section 16 lets the government restrict transfers to notified countries, and none were notified as of November 2026. Self-hosting in India is a design choice that reduces transfer and sector-rule risk. You still need safeguards, contracts and logs under the Rules. Your counsel should confirm how they apply to you.

Is a managed private AI service better than hiring an MLOps team?

It can be, if you want the model in your own accounts without building an operations team. A partner builds and hands over the system, and may keep monitoring and updating it under a support plan. A hired team gives you more control but brings staff dependency and a fixed GPU bill. Choose by who will answer the pager when the model is down.

Not sure who would run a private model? Talk to us before you hire.