Compare Prompt Versions Safely
Design an experiment workspace for comparing prompt changes on a fixed evaluation set before a new version reaches production.
The brief
Understand the problem
Background
Prompt edits can improve average results while causing regressions for rare or sensitive cases. A credible experiment needs locked versions, representative inputs, blinded review, cost context, and a decision record rather than a gallery of appealing examples.
User context
Yuki maintains a customer-email summarizer. A shorter prompt reduces latency, but early checks suggest it sometimes omits refund deadlines and exposes internal notes in the customer-facing summary.
Product problem
A product team needs to compare two prompt versions under controlled conditions and decide whether to ship, revise, or stop based on explicit quality and safety criteria.
Objective
Create experiment setup, blinded result review, metric analysis, and release decision for one prompt change.
What to design
Define the experience
Required experience
- Lock a baseline, candidate, model, parameters, and evaluation set
- Define quality, safety, latency, and cost criteria before running
- Review paired outputs without version identity bias
- Record a release decision with failures and follow-up ownership
Screens and states
- Experiment setup
- Blinded comparison
- Results analysis
- Release decision
Core user flow
Follow the critical path
- 01
Yuki selects the production prompt as baseline and adds the shorter candidate
- 02
She chooses a fixed set containing refund deadlines and restricted internal notes
- 03
Reviewers compare randomized output pairs against a shared rubric
- 04
Results show lower latency but two critical privacy failures in the candidate
- 05
Yuki stops release, assigns a revision, and preserves the run for comparison
Product rules
Requirements and constraints
Requirements
- Version prompts, models, parameters, tools, evaluation inputs, and rubric together
- Prevent editing any locked experiment input after a run begins
- Randomize or blind output identity where the evaluation permits
- Show aggregate metrics alongside every severe failure and reviewer disagreement
- Require explicit ship, revise, or stop criteria and an accountable decision owner
Constraints
- Evaluation data may contain restricted customer examples
- Small samples cannot support precise claims about production performance
- Prompt text may be visible only to authorized experiment owners
Reality check
States worth considering
Ready-to-use content
Mock data
Use this content to test hierarchy and realistic data states. You can expand it when the concept needs more depth.
Success criteria
- Refund deadline recall at least 95%
- Zero internal-note exposure
- Median latency under 2.0 seconds
- Median run cost under $0.03
Evaluation set
- 40 routine cases
- 12 refund-deadline cases
- 8 restricted-note cases
- 10 multilingual cases
Finish line
What to deliver
- Four desktop screens covering a controlled prompt experiment, blind review, and stopped release
Optional direction
Visual resources
Use these as a starting constraint if you want one. They are not part of the required solution.