Release tests should cover the jobs the agent is meant to perform, the actions it must refuse, and the evidence an operator needs to publish or hold the candidate.
Write tests around user jobs and control boundaries
Choose a compact set of top tasks from actual product use, recent regressions, support issues, and high-consequence actions. Each test should specify starting state, approved input, expected observable behavior, forbidden behavior, tool calls, data boundary, and a pass or hold rule. Include at least one refusal or denied-action path rather than evaluating only helpful responses.
For stochastic output, test stable properties instead of exact prose: required facts, cited source identifiers, tool choice, argument schema, authorization decision, absence of prohibited data, and completion state. Preserve representative outputs for human comparison, but do not let a single subjective score hide a failed permission or missing required field.
Compare candidate and accepted version
Run the same fixture set against the candidate and current accepted release under controlled settings. Record model identifier, parameters, component versions, timestamps, and external dependencies. A difference report should show changed pass states, tool calls, latency when relevant, and human-reviewed qualitative shifts. New behavior can be intentional, but the release note should connect it to the governing change.
LangSmith describes prompt versions, diffs, tags, and webhooks that can participate in a release workflow. Those controls are useful infrastructure, not an acceptance policy. The buyer defines which tasks matter, who reviews subjective differences, what failures block publication, and whether a model-provider change requires a new baseline.
End with an operator decision
The test receipt should identify passed, failed, held, excluded, and not-run cases. It should also name the candidate version, environment, approver, accepted exceptions, rollback target, and verification step after publish. Avoid an aggregate percentage that allows one consequential failure to disappear among many low-impact passes. The release decision needs the individual blocking states.
Release Rail installs and documents this gate through Reality Contact, LLC. The buyer reviews the evidence and chooses publish, repair, hold, or restore. Testing reduces uncertainty around specified cases but does not certify general agent safety, accuracy, security, or compliance, and it cannot cover every input or downstream system state.
Where the service stops
Reality Contact, LLC installs and tests release controls but does not certify agent safety, guarantee behavioral consistency, decide product policy, approve production releases, operate incidents indefinitely, or assume authority over customer data, credentials, model choices, or consequential agent actions. The buyer names release and rollback authorities, approves the version schema and top tasks, controls production credentials, decides whether each candidate is published, and can require repair or restoration after reviewing the evidence. This release-engineering service does not replace the buyer's product, security, legal, privacy, model-risk, or production-readiness review. The buyer controls production credentials and decides whether to publish, repair, hold, or restore every release.
Sources: LangSmith documentation for prompt versions and deployment tags; NIST Secure Software Development Framework.