Workflow

Evaluator-Optimizer Translation Loop

Two LLM roles in a feedback loop: a translator generates output, an evaluator critiques it against a rubric, and the task recursively refines until the evaluator approves or a max-iteration cap is hit.

What This Builds

This recipe implements Anthropic’s evaluator-optimizer pattern: one LLM call generates a response while a second LLM call evaluates it and returns feedback, in a loop. It is the agentic analogue of an author and an editor iterating on a draft.

The concrete use case is literary translation, where nuance is easy to miss on the first pass but an evaluator can articulate what to fix. The flow:

  1. Generate a translation of the source text into the target language (optimizer role).
  2. Evaluate the translation against explicit criteria — accuracy of meaning, natural flow, preservation of style (evaluator role).
  3. If the evaluator returns exactly APPROVED, return the final translation.
  4. Otherwise the task recursively re-runs with the previous translation plus the feedback, incrementing a rejection counter.
  5. Bail out with the best-so-far translation once a maximum iteration count is reached.

The Stack

  • Trigger.dev — a single durable task that calls itself with triggerAndWait(...).unwrap() to drive the refinement loop, with the iteration counter carried in the payload so the loop is crash-safe and observable.
  • Vercel AI SDK (generateText) — separate generate and evaluate calls, each with its own system prompt and telemetry.
  • An LLM provider — Claude (via the Anthropic Startup Program) or any capable model through OpenRouter. The evaluator prompt deliberately enforces a strict response contract (APPROVED or only the issues to fix).

Step-by-Step Outline

  1. Design the payload. Include text, targetLanguage, optional previousTranslation, optional feedback, and a rejectionCount.
  2. Generate. If feedback exists, prompt the model to improve the previous translation; otherwise prompt for a first translation.
  3. Evaluate. Use a separate call with a critic system prompt that outputs either APPROVED or a terse list of must-fix issues, and tighten/loosen the quality threshold based on the iteration number.
  4. Branch. On APPROVED, return { finalTranslation, iterations, status }. Otherwise recurse with the new translation and feedback.
  5. Cap the loop. Return a MAX_ITERATIONS_REACHED result once the rejection count hits the ceiling so the job always terminates.

Why This Shape Works

This pattern shines when you have clear evaluation criteria and iterative refinement measurably improves quality. Keeping the generator and evaluator as distinct prompts prevents the model from grading its own work in the same breath. Encoding the loop as a durable, self-triggering task means each iteration is checkpointed, retryable, and visible in traces rather than buried in an in-memory while loop.

Source

Trigger.dev, Building effective AI agents with Trigger.dev (evaluator-optimizer section, full recursive TypeScript example): https://trigger.dev/blog/ai-agents-with-trigger

Pattern reference and Claude Cookbook implementation: Anthropic, Building effective agents (Workflow: Evaluator-optimizer): https://www.anthropic.com/engineering/building-effective-agents