Short answer: evaluate an open-weight model with a fixed, written test plan, and run the same plan on every candidate and every later version. Write down the job and the pass marks first. Shortlist on licence, size and language. Build a golden set of 150 to 300 real, redacted cases with expected results. Score with deterministic checks first, a rubric second and a calibrated LLM judge last. Run the tests on the exact quantisation, context length and serving stack you will deploy, then add safety, data-flow and operations tests. Keep the scores and the decision as a record. Public leaderboards help you shortlist. They do not tell you whether a model works on your documents.
This guide is for engineering and product leads who have a use case and want to know whether an open-weight model can replace, or sit beside, a hosted API. It publishes no benchmark results, because the right answer depends on your data. It gives you the method. It is engineering guidance, not legal advice. Licence terms and the legal reading of the DPDP Act for your business go to counsel.
What are you actually testing when you evaluate an open-weight model?
Most failed evaluations test the wrong thing: a public benchmark, a good number, and a decision. A benchmark measures someone else's tasks. You need four separate answers.
| Question | What you check | Typical evidence |
|---|---|---|
| Does it do the job? | What you check: Accuracy, format, tone and refusals on your own cases | Typical evidence: Golden-set scores per task type |
| Can it handle our data? | What you check: Document lengths, languages, tables, scanned text, personal data | Typical evidence: Scores on the hard slice of the set; leak and redaction tests |
| Can we run it? | What you check: GPU memory, latency under load, upgrades, restarts, cost per task | Typical evidence: Load test on your hardware; a cost sheet |
| Are we allowed to use it? | What you check: Licence terms for your product and customers | Typical evidence: A licence note per model, reviewed by counsel |
Write a one-page requirement sheet before you touch a model. It names the task, the input types, the languages, the longest input, the output format, the data class (does it contain personal data?), where it must run, and the pass mark for each measure. Setting the pass marks first stops the team from choosing marks that fit the model it already likes.
How do you shortlist open-weight candidates?
Do not test everything. Cut the field with cheap filters, then test three to five models properly.
- Licence, per model. Licences are set per model, not per vendor. Read the licence on each model card before you invest time. Our licence comparison goes through the main families.
- Size against your hardware. As a rough estimate, weights take about parameters times bytes per parameter: 2 bytes in BF16, 1 in FP8 or INT8, about 0.5 at 4-bit. A 31B model is about 62 GB in BF16 and about 16 GB at 4-bit, before the KV cache and overhead, which grow with context length and concurrency. Our migration guide covers GPU sizing in full.
- Languages and context. If you serve Hindi or other Indian-language text, or documents of 100 pages, test those first. A model that fails them is out.
- Capabilities you need. Tool calling, structured output, vision or long context are yes-or-no filters.
Licences differ a lot inside one family, so a blanket statement is wrong. Per the model cards and licence files checked on 20 November 2026, here are examples of how far they differ.
| Model | Licence (summary) | What to check |
|---|---|---|
| Llama 4 Scout and Maverick | Licence (summary): Llama 4 Community License. Companies with more than 700M monthly active users need a licence from Meta. Distributors must display "Built with Llama" | What to check: Naming rules for derivatives; the Acceptable Use Policy, which withholds the grant for multimodal models from EU-based individuals and companies |
| Gemma 4 | Licence (summary): Apache 2.0 | What to check: Earlier Gemma versions are under Google's Gemma Terms of Use, which differ |
| Qwen3.8-27B | Licence (summary): Apache 2.0 | What to check: Other Qwen models use the Qwen Community License, a Max License or a non-commercial Research License |
| Mistral Small 4 | Licence (summary): Apache 2.0 | What to check: Medium 3.5 is Modified MIT with a revenue cap; Large 4 has its own open-weight licence |
We describe Llama as open-weight under Meta's community licence, not as open source. Save the licence text for each finalist in the evaluation folder. It is part of the record.
How do you build a golden set that survives model changes?

A golden set is a fixed collection of real inputs with agreed correct outputs. It is the most valuable asset in the whole process, because it outlives every model you test. Build it so it does not depend on any one model.
- Use real inputs. Take them from tickets, documents, emails or forms your system will see. Synthetic cases can add edge cases, but a set made only of them tests a fiction.
- Redact personal data first. The set is a data store. Redact or replace names, numbers and addresses before it enters a shared folder, and add it to your data map. Our post on PII redaction for LLMs covers the tools. If a case cannot be redacted without losing its meaning, keep it in a restricted store.
- Tag every case. Task type, language, length, difficulty and data class. Tags let you see that a model scores well overall but fails on long scanned documents.
- Store the expected result as data, not as a prompt. Keep a reference answer, a list of required facts, a required JSON schema or a pass/fail rule. If the expected result is written in one model's style, the next model will be unfairly penalised.
- Include the awkward cases. Ambiguous inputs, missing fields, contradictory documents, prompt-injection text inside a document, and requests the system should refuse. Aim for roughly a fifth of the set.
- Split it. Keep about a third as a held-out set that nobody uses while tuning prompts. Look at it once, at the end. Never use golden cases as examples in a prompt or as fine-tuning data.
How many cases do you need?
Enough that a few wrong answers do not swing the result. This is arithmetic, not a rule of the industry. The table shows the 95% Wilson confidence interval around an observed 90% pass rate at different set sizes.
| Cases | Observed pass rate | Plausible true rate (95% interval) |
|---|---|---|
| 30 | Observed pass rate: 90% | Plausible true rate (95% interval): 74.4% to 96.5% |
| 50 | Observed pass rate: 90% | Plausible true rate (95% interval): 78.6% to 95.7% |
| 100 | Observed pass rate: 90% | Plausible true rate (95% interval): 82.6% to 94.5% |
| 200 | Observed pass rate: 90% | Plausible true rate (95% interval): 85.1% to 93.4% |
| 300 | Observed pass rate: 90% | Plausible true rate (95% interval): 86.1% to 92.9% |
With 50 cases, a model scoring 90% could really be at 79% or at 96%. Two models that differ by three points on a set of 50 cannot be told apart. That is why 150 to 300 cases is a sensible target for a task, and why you should only call a winner when the gap is clearly larger than the interval. A smaller set is still better than none, if you report the interval.
How should you score the outputs?
Use the cheapest reliable method for each case, in this order.
- Deterministic checks. Is the output valid JSON? Does it match the schema? Does the extracted amount equal the reference? Is a required field present? Is a forbidden word absent? These are fast, free and unarguable. Many enterprise tasks (extraction, classification, routing) can be scored entirely this way.
- Rubric scoring. For text that cannot be matched exactly, write a short rubric: facts that must appear, facts that must not, tone, length. Score each point yes or no. Yes-or-no points are easier to agree on than a score out of ten.
- LLM-as-judge. Use a strong model to grade the rest. The research on this (Zheng et al., 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena") reports that judges can agree with human preferences well, and also names position, verbosity and self-enhancement biases. So calibrate it: have two people grade a sample of 30 to 50 outputs, compare the judge to them, and fix the rubric until they agree. Swap the order in pairwise comparisons. Do not let a model judge its own outputs.
- Human review. Read every case where the judge and the deterministic check disagree, and every case that lands near the pass mark.
Report scores per tag, not only as one number. A single average hides the slice that matters. Weight failures by cost: a wrong tone in a draft email is not a wrong amount in an invoice.
Why must you test at your real serving settings?

Quantisation, context length and the serving engine all change behaviour. The vLLM documentation says quantisation "trades off model precision for smaller memory footprint". So a 4-bit build that fits your GPU is a different candidate from the BF16 build in the model card's table.
Evaluate the build you will deploy. Pin and record each of these in the run log:
| Setting | Why it matters | Where it appears in vLLM |
|---|---|---|
| Model name and exact revision | Why it matters: Weights can be updated under the same name | Where it appears in vLLM: The model path or ID you pass to `vllm serve` |
| Quantisation format | Why it matters: Changes accuracy and memory. FP8 W8A8 needs Ada, Hopper or AMD GPUs | Where it appears in vLLM: The model files and quantisation settings |
| Context length (prompt plus output) | Why it matters: Long inputs fail silently if the cap is too low | Where it appears in vLLM: `--max-model-len` |
| Share of GPU memory | Why it matters: Sets the room left for the KV cache and so for concurrency | Where it appears in vLLM: `--gpu-memory-utilization` |
| GPUs per model | Why it matters: Splitting across GPUs changes cost and failure modes | Where it appears in vLLM: `--tensor-parallel-size` |
| KV cache type | Why it matters: A quantised KV cache saves memory and may change output | Where it appears in vLLM: `--kv-cache-dtype` |
| Serving engine version | Why it matters: Fixes and changes land often. v0.34.0 was the latest vLLM release on 20 November 2026 | Where it appears in vLLM: Your lockfile or container tag |
`vllm serve
Run each candidate more than once. Sampling makes outputs vary, so run three times, or set temperature to zero for tasks that allow it, and report the spread. Run the tests on the same cloud or hardware you plan to use. If you plan to host in India, test in an Indian region from the start: AWS has Mumbai and Hyderabad, Azure has Central India (Pune) and others, and GCP has Mumbai and Delhi, and GPU types differ by zone, so check before planning. Our guide on VPC, on-premises and GPU cloud compares the options.
What else should you test besides accuracy?
A model that scores well and then leaks data, falls over under load or costs three times the forecast is still the wrong model. Add these tests to the plan.
- Latency and throughput under your real load. Measure time to first token and total time at the number of simultaneous users you expect, using your real prompt lengths. We publish no figures here, because they depend entirely on your hardware, model and settings.
- Cost per completed task. Divide the cost of the GPU hours by the tasks finished, at realistic load, and compare with the API bill for the same tasks. Private LLM vs API cost in rupees shows the arithmetic.
- Prompt-injection and refusal behaviour. Put hostile instructions inside test documents and check that the model does not follow them or reveal the system prompt. Check that it refuses what it should.
- Data-flow test. Trace one request end to end and list every place the text lands: logs, traces, caches, vector stores, eval tools and any third-party hosted judge. Self-hosting a model does not help if the prompts are then sent to an outside service.
- Permissions. If the model reads internal documents, test that one user cannot get another user's content through it.
- Operations. Restart the server, roll to a new version and fill the context window. Note what breaks and how you would know.
The multi-model architecture guide shows where a gateway and routing rules fit once you have more than one model in service.
What does the decision record look like?
Finish each evaluation with a short, dated record. It protects you in three ways. The next engineer knows why the model was picked. A buyer's security team can see how you chose and checked it. And a later model change has something to be compared with.
| Section | What goes in it |
|---|---|
| Use case and requirement sheet | What goes in it: Task, data class, pass marks, set before testing |
| Candidates and licences | What goes in it: Model names, revisions, licence text, counsel's note |
| Golden set | What goes in it: Size, tags, redaction method, who approved it, version hash |
| Serving configuration | What goes in it: Quantisation, context length, engine version, hardware, region |
| Results | What goes in it: Scores per tag with intervals, judge calibration, failures reviewed |
| Safety and operations tests | What goes in it: Injection, leak, data-flow, load and restart results |
| Decision and owner | What goes in it: The pick, the reasons, the open risks, the date of the next review |
Enterprise buyers can ask how you selected and tested a model. A tidy record answers that in one attachment. See our page on AI security questionnaires.
How do you re-run the plan when the model changes?
Models change in four ways: you upgrade to a new version, the vendor reissues weights under the same name, you change quantisation, or you change the prompt. Each can move your scores. Treat the golden set as a regression suite.
- Automate the run. One command should load the golden set, call the endpoint, score and print the table. If it takes a week of manual work, nobody will re-run it.
- Gate the change. A new model or prompt goes live only if it meets the pass marks and does not drop on any tag by more than a margin you set in advance.
- Version everything. Golden set, scorer, prompts, model revision and engine version, each with an ID. A score is only meaningful with these attached.
- Add production failures. Each time a real case goes wrong, redact it and add it to the set. The set grows from the failures that matter.
- Watch live traffic. Sample production outputs for review. See LLM observability setup for what to log without logging personal data.
Run the full plan on a schedule too, for example each quarter, even when nothing has changed.
What are the common mistakes?
- Picking by leaderboard rank. Public benchmarks are a way to shortlist. Their questions are not your documents.
- Tuning prompts on the test set. Then the scores measure the prompt, not the model. Keep the held-out set sealed.
- Testing a bigger build than you will run. The 4-bit build on your GPU is the real candidate.
- Ignoring the licence until launch. A licence problem found after integration costs far more than one found in week one.
If you want a second pair of eyes on the plan, our Own Your AI work covers choosing and running models on your own servers or cloud.
Sources: vLLM documentation (online serving; engine arguments; quantisation) and the vLLM v0.34.0 release notes, checked 20 November 2026. Model cards and licence files on Hugging Face and Meta's Llama 4 licence and use policy, and Google's Gemma 4 licence page, checked 20 November 2026. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv:2306.05685. EleutherAI lm-evaluation-harness README. The confidence intervals are our own arithmetic (Wilson interval, 95%). Cloud region lists from AWS, Microsoft Azure and Google Cloud documentation. Last reviewed 20 November 2026.
FAQ
How do you evaluate an open-weight LLM for enterprise use?
Write a requirement sheet with pass marks first. Shortlist three to five models on licence, size, language and capability. Build a golden set of 150 to 300 real, redacted cases with expected results. Score with deterministic checks, a rubric and a calibrated LLM judge. Run the tests on the exact quantisation and serving settings you will deploy, add safety, data-flow and load tests, and keep the results as a dated decision record.
How big should a golden set be?
For one task, 150 to 300 cases is a sensible target. As arithmetic, a 90% pass rate on 50 cases has a 95% interval of about 79% to 96%, while on 300 cases it is about 86% to 93%. Smaller sets still help, but report the interval and do not call a winner on a small gap.
Can I trust public leaderboards to pick a model?
Use them to build a shortlist, not to decide. A leaderboard measures a model on other people's questions, usually with the vendor's best settings. Your documents, languages, output formats and quantised build can give very different results, so test the shortlist on your own golden set.
Should I test the quantised model or the full-precision one?
Test the build you will deploy. The vLLM documentation says quantisation trades off model precision for a smaller memory footprint, so a 4-bit or FP8 build can behave differently from the BF16 weights. Record the quantisation format, context length and engine version in the run log, and re-test if any of them change.
Can an LLM judge replace human reviewers?
Not fully. Research on LLM judges reports good agreement with human preferences and also position, verbosity and self-enhancement biases. Calibrate the judge against two human graders on 30 to 50 outputs, swap answer order, avoid using a model to judge itself, and have people read the cases near the pass mark and where checks disagree.
Does the licence belong in the evaluation?
Yes. Open-weight licences are set per model, and they differ inside one family: some Qwen and Mistral models are Apache 2.0, others carry revenue limits, service restrictions or non-commercial terms. Save each finalist's licence text in the evaluation record and ask counsel to review it for your product.
