Skip to content

Community

Case studies, adapters, and applied work.

  • 8 min read

Bridging the Subjectivity Gap: How Automated Prompt Optimization Helps Teams Build Expert-Aligned AI Functions

30annotations
+42.9ppaverage accuracy increase
+30.6ppaverage consistency increase
79%cheaper at peak accuracy

Most applied AI organizations build systems for recurring tasks, often taking the form of scorers, judges, classifiers, summarizers, matchers, and more. These tasks are typically subjective in nature, and the goal is to align the judgment of a model with that of a domain expert.

While foundation models have a lot of general world knowledge, they don't know how your organization wants a particular decision made. That gap between general intelligence and organization-specific judgment is the subjectivity gap.

We refer to models that make repeated decisions as AI Functions. They make up a less-visible category of intelligence relative to agentic coding, world models, and robotics - but are perhaps just as important, if not more so.

  • 15 min read

Confidence-Aware Prompt Optimization for LLM Classification

ConfidenceAdapter is a custom GEPA adapter that uses token-level log-probabilities from LLM structured output to score prompt candidates on a continuous scale instead of binary correct/wrong. By feeding the reflection LLM rich feedback about model uncertainty — including probability distributions and top alternatives — it produces better disambiguation rules and converges to higher-accuracy prompts. In experiments across AG News (4-class), Emotion (6-class), and Rotten Tomatoes (binary), ConfidenceAdapter matched or beat DefaultAdapter on all three datasets, with gains of +2.10pp and +1.80pp on the multiclass tasks.

Subscribe