LLM Judge Rubric Builder
LLM Judge Rubric Builder
Compose weighted criteria, scoring scales, and a self-validating judge prompt — exportable as YAML or JSON.
Criteria (5)
name: Customer Support Response Rubric
task: Score how well the assistant resolves a customer support ticket using only
the provided knowledge base context.
judge_model: claude-sonnet-4.6
output_format: json
criteria:
- id: factuality
name: Factual accuracy
description: All claims are correct and verifiable from the source material; no
hallucinated facts.
weight: 0.35
scale: likert5
pass_threshold: 4
- id: completeness
name: Completeness
description: Response addresses every part of the user request without omission.
weight: 0.25
scale: likert5
pass_threshold: 4
- id: groundedness
name: Groundedness
description: Response cites or references provided context where relevant; no
extrapolation beyond context.
weight: 0.2
scale: likert5
pass_threshold: 3
- id: format
name: Output format compliance
description: Response strictly matches the requested output format (JSON,
markdown headings, length).
weight: 0.1
scale: pass_fail
- id: tone
name: Tone and clarity
description: Tone matches brief; language is clear and free of jargon when not
warranted.
weight: 0.1
scale: likert5
pass_threshold: 3
What This Tool Does
The LLM Judge Rubric Builder composes weighted scoring criteria, gives each one a scale, and exports the rubric as YAML, JSON, or a ready-to-paste judge prompt. Everything runs in your browser — nothing is uploaded. The rule people miss: criterion weights must sum to exactly 1.0, so the weighted total stays a clean average. Use the Normalize button to fix any drift.
Last updated:
This tool is provided as-is for convenience. Output should be verified before use in any production or critical context.
Agent Invocation
Best Path For Builders
Browser workflow
Runs instantly in the browser with private local processing and copy/export-ready output.
Browser Workflow
This tool is optimized for instant in-browser execution with local data handling. Run it here and copy/export the output directly.
/llm-judge-rubric-builder/
For automation planning, fetch the canonical contract at /api/tool/llm-judge-rubric-builder.json.
How to Use LLM Judge Rubric Builder
- 1
Set the rubric metadata
Name the rubric, write a short task description so the judge model knows what it is grading, choose your judge model, and select an output format (JSON, XML, or structured text).
- 2
Add and weight criteria
Each criterion needs a name, an id, a description, a weight, and a scale (Likert 1-5, Likert 1-7, pass/fail, percentage, or 0/1). Use Equal weights or Normalize to 1.0 to keep weights tidy.
- 3
Validate the rubric
The validation panel flags missing fields, duplicate ids, weights that do not sum to 1.0, mixed scales, and pass-thresholds out of range. Resolve every error before exporting.
- 4
Export YAML, JSON, or judge prompt
Switch the right pane between YAML, JSON, and a generated judge prompt. The prompt renders criteria, score ranges, the procedure, and an exact output template that tracks your selected format.
Frequently Asked Questions
Which scales are supported?
How does rubric self-validation work?
What is in the generated judge prompt?
Does it send my data to a server?
Can I import an existing rubric?
How do I write a scoring rubric for an LLM judge?
Start from the task the judge is scoring, break quality into a few named criteria, give each a weight and a scale, then export a judge prompt that instructs the model to justify and score every criterion. This builder validates the rubric as you edit and writes the prompt for you, so the judge scores against explicit rules instead of vibes.
Step by step
- Write the task description — what the model under evaluation was actually asked to do. An empty task produces noisier scores, and the validator warns about it.
- Add each criterion with a short, checkable description (for example "all claims are verifiable from the provided context").
- Pick a scale per criterion and set a pass threshold where it applies.
- Assign weights, then click Normalize to 1.0 or Equal weights — the total must equal 1.000 or the rubric is flagged invalid.
- Switch the export to YAML, JSON, or Prompt and copy it into your eval harness.
Scoring scales this builder supports
| Scale | Score range |
|---|---|
| Likert 1–5 | 1 (poor) to 5 (excellent); supports a pass threshold |
| Likert 1–7 | 1 (poor) to 7 (excellent); supports a pass threshold |
| Pass / Fail | Binary verdict; no numeric threshold |
| Percentage | 0 to 100; supports a pass threshold |
| 0 / 1 | Binary numeric; no threshold |
Scale options as implemented in this tool. The generated procedure normalizes every scale to 0–1 before applying weights, and overall pass requires each criterion that has a pass threshold to meet it.
How should I weight the criteria?
Weights encode what matters most: give factual accuracy more weight than tone if a wrong answer is worse than a blunt one. The builder requires the weights to sum to 1.000 so the weighted total is a true average — Normalize rescales your current weights to hit that sum, Equal weights spreads them evenly. It also warns when you mix more than two scale kinds, because that makes the aggregate harder to reason about.
Should the judge return JSON or a prompt?
The rubric itself exports as YAML or JSON for storage in your repo. The Prompt view scaffolds the full judge instruction: task, inputs, per-criterion scoring guidance, a procedure, and a required output shape (JSON, XML, or structured text) that mirrors your criteria ids. Pin the output format so downstream parsing stays stable across runs.
Does my rubric leave the browser?
No. The builder validates and renders everything client-side; the rubric is never uploaded. Eval rubrics often encode proprietary quality bars and internal task definitions, so a local-only tool is the safe default when you are drafting evaluation criteria for real production prompts.