# Harness Engineering: Validation, Permissions, Budgets, and Reliable Execution

**Harness Engineering · Module 3 of 16**

> **Start here:** understand that a model proposes actions and application code executes them.
>
> **Your goal:** turn a working tool loop into a system with inspectable execution boundaries.

The support assistant's answer is correct, but its trace contains an attempted refund. Would you consider the run successful? A useful harness must judge more than the final paragraph. It must control what can happen along the way.

Our fictional assistant reads a returns policy and drafts an explanation. Keeping that scope small lets us inspect six responsibilities: input contracts, tool validation, caller authorization, resource limits, result handling, and recovery.

![Requests pass through schema, access, and budget checks before execution, with failures recorded separately.](https://cdn.hashnode.com/uploads/covers/5fb3b801b589f1618af9d143/5370859c-4df7-48eb-ae92-027f5db74bec.png align="center")

*Each gate answers a different question. Passing a schema check does not grant authority.*

## 1\. Write the contract before the prompt

Define the user's requested artifact and the actions needed to produce it. “Help with returns” is too broad. “Draft an internal explanation from the authorized returns policy” gives the harness a testable boundary.

| Contract | Example | What it prevents |
| --- | --- | --- |
| Input | Nonempty question with a size limit | Unusable or unbounded requests |
| Tool | `lookup_policy(policy_id)` | Arbitrary tool dispatch |
| Identity | User and tenant from the authenticated session | Model-selected impersonation |
| Result | Status, source version, text | Success without evidence |
| Completion | Draft plus source IDs or explicit unknown | Silent evidence gaps |

Credentials belong in an approved secret mechanism outside the article and repository. Never use a generated argument as the source of the caller's identity. A tool's description helps model selection; executable validation supplies enforcement.

## 2\. Validate shape and meaning

Schema validation asks whether `policy_id` is a string and whether unexpected fields are present. Semantic validation asks whether the identifier refers to a supported policy in the caller's permitted scope. A refund amount can be a valid number and still exceed the permitted amount. A path can be a valid string and still resolve outside an approved directory.

Keep errors specific enough to diagnose without disclosing private records. For a tenant-scoped policy lookup, the public response might deliberately avoid distinguishing an inaccessible identifier from a nonexistent one. Internal audit metadata can preserve the reason under restricted access.

Here is a complete offline permission check. The grants are application-owned synthetic data; the proposed action cannot replace them.

```python
POLICIES = {"shop-a": {"returns": "Opened consumables are excluded."}}
GRANTS = {"reader-1": {"shop-a"}}


def read_policy(user, tenant, action):
    if tenant not in GRANTS.get(user, set()):
        return {"ok": False, "error": "access_denied"}
    if not isinstance(action, dict) or action.get("tool") != "lookup_policy":
        return {"ok": False, "error": "tool_not_allowed"}
    args = action.get("args")
    if not isinstance(args, dict) or set(args) != {"policy_id"}:
        return {"ok": False, "error": "invalid_arguments"}
    key = args["policy_id"]
    if not isinstance(key, str):
        return {"ok": False, "error": "invalid_arguments"}
    text = POLICIES.get(tenant, {}).get(key)
    if text is None:
        return {"ok": False, "error": "policy_unavailable"}
    return {"ok": True, "source_id": f"{tenant}/{key}/v1", "text": text}


action = {"tool": "lookup_policy", "args": {"policy_id": "returns"}}
assert read_policy("reader-1", "shop-a", action)["ok"]
assert read_policy("reader-1", "shop-b", action)["error"] == "access_denied"
assert not read_policy("reader-1", "shop-a", {"tool": "issue_refund"})["ok"]
print("3 permission checks passed")
```

This block was executed locally. It tests the listed permission cases, not an authentication service or a complete security boundary. A production service must establish the user and tenant before calling it.

## 3\. Set limits at the level that consumes resources

A prompt requesting “at most three searches” is useful guidance, but the harness needs a counter. A step limit also does not stop a single tool from hanging. Define request deadlines, individual tool timeouts, maximum result sizes, concurrent operations, and spending ceilings where supported.

Write a limit outcome as carefully as a success outcome. If the assistant reaches its search limit with conflicting policies, return a partial draft and unresolved evidence. Do not turn a budget stop into a confident conclusion.

Track input and output usage separately. A symbolic model-cost estimate is `input_tokens × input_rate + output_tokens × output_rate`, using consistent pricing units. Include retries and any charged tool or retrieval operations. Current rates and billing behavior belong in environment-specific records rather than a permanent guessed price table.

## 4\. Handle responses and streams explicitly

Model responses may contain several blocks, tool requests, structured fields, or stop reasons. A stream may expose partial text before a failure. Buffer structured output until it can be parsed and validated. Display “running” or “partial” until completion is actually observed.

| Failure | Appropriate first response |
| --- | --- |
| Invalid arguments | Return a structured correction opportunity within budget |
| Unauthorized action | Reject; do not retry with broader access |
| Authentication failure | Stop and repair configuration |
| Transient read timeout | Bounded retry if the deadline permits |
| Ambiguous write timeout | Inspect operation status before retrying |
| Malformed final output | Validate and repair within a small explicit budget |

Use call IDs to match results, especially when operations finish out of order. Record errors as errors rather than empty successful documents. The model should receive the observed failure, not an invented substitute.

## 5\. Keep approval and execution records distinct

For a draft-only application, the safest design omits sending and refunding entirely. If those capabilities are later added, bind approval to the exact action, target, arguments, policy version, and relevant draft revision. Editing the amount after approval must invalidate that approval.

An approval is still not an execution receipt. Store the provider's operation identifier and observed status after execution. If a process crashes between sending a request and recording the result, reconcile the external state. An in-memory “done” flag cannot resolve that uncertainty.

The [OWASP Excessive Agency guidance](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) explains why unnecessary capabilities, permissions, and autonomy increase exposure. A sandbox constrains execution environment access; it does not determine who is entitled to a refund.

## 6\. Follow along

Draw the assistant's data flow. Mark the boundary at which identity is established, policy access is checked, model content enters the executor, and results enter logs. Add an owner and observable test to each boundary.

Test five cases: a forbidden tool, another tenant's policy, malformed arguments, a hung lookup, and a partial model stream. **Expected behavior:** no unauthorized data reaches the model; execution stays bounded; errors remain visible; partial text is never treated as completed work.

Finish with a one-page runbook for accidental logging of a synthetic identifier: stop the affected logging path, restrict access, inspect copies, apply the retention process, and verify the corrected configuration. This is an exercise, not a report of an actual incident.

## Further reading

- [OWASP: Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) — capabilities and permissions.
- [Anthropic: building effective agents](https://www.anthropic.com/research/building-effective-agents) — choosing bounded designs.
- [Google security learning](https://www.skills.google/course_templates/1146) — defensive AI security context.

## Next module

**Context, memory, and skills:** decide what the model should see, what the application should store, and which procedures should be reusable.