Framework and Vendor Comparisons
Observation date and maintenance
Section titled “Observation date and maintenance”Observed on 2026-07-23.
Applicable surfaces: current maintainer documentation for LangGraph, OpenAI Agents SDK, Microsoft AutoGen, LlamaIndex, and CrewAI.
Maintenance state: maintained Field Note. Revisit when a framework changes its core orchestration model, persistence guarantees, human-in-the-loop support, observability model, deployment assumptions, or compatibility story.
Current observation
Section titled “Current observation”Agent frameworks and vendor harnesses do not converge on one abstraction. They emphasize different centers of gravity.
Some start from graph or workflow orchestration, where state, transitions, checkpointing, and human intervention are explicit design surfaces. Some start from agent objects, tools, handoffs, guardrails, tracing, and sessions. Some start from retrieval and data applications. Some emphasize crews, roles, tasks, and process models. These differences matter because they shape what is easy to build, test, observe, and govern.
A feature checklist hides the real decision. “Supports tools” does not say how tool arguments are validated. “Has memory” does not say who owns authoritative state. “Supports human-in-the-loop” does not say whether approval is bound to a specific action and state. “Has tracing” does not say whether traces connect to accepted outcomes, incidents, and release versions.
Durable engineering implication
Section titled “Durable engineering implication”Compare frameworks by workload and operating claim.
Begin with the work you need to support: interactive assistant, coding agent, research workflow, support case, data-analysis job, long-running business process, or multi-agent delegation. Then ask what must be true for that workload to be trustworthy in production.
Use questions like these:
| Dimension | Comparison question |
|---|---|
| Control loop | Who decides the next step, and where can the harness reject or revise it? |
| State | What state is authoritative, what is derived context, and what survives restart? |
| Tools and authority | How are tools declared, authorized, validated, sandboxed, and audited? |
| Human control | Can a person approve, edit, stop, resume, escalate, or take over at the right moment? |
| Evaluation | Can the same harness run representative tasks with versioned prompts, tools, data, and graders? |
| Observability | Can traces explain model decisions, tool calls, retries, approvals, and accepted outcomes? |
| Release | Can behavior-shaping assets be versioned, staged, rolled back, and compared? |
| Portability | Which parts are standard protocols, which are framework objects, and which are hosted product behavior? |
| Governance | Can owners, policies, risk decisions, and evidence records attach to the system? |
The answer may be “use no framework.” A small deterministic harness can be the right choice for a narrow workflow with limited tool use, strong compliance requirements, or unusual state semantics. A broad framework can be right when it removes real orchestration, persistence, evaluation, and observability work. A hosted vendor harness can be right when operational speed matters more than portability, as long as the dependency is explicit.
What not to generalize
Section titled “What not to generalize”Do not rank frameworks without a workload. A framework that is excellent for retrieval-heavy analysis may be awkward for long-running approvals. A graph runtime may be too much for a small drafting tool. A hosted SDK may provide fast integration while limiting low-level control.
Do not treat maintainer documentation as proof of production fitness. It tells you what the framework intends to expose. It does not prove your migration path, latency profile, safety controls, incident process, or user experience.
Do not let framework vocabulary replace Fieldbook vocabulary. If a framework
has an Agent, Tool, Action, Flow, Task, or Memory object, map it back
to model capability, system capability, authority, state, control loop, and
accepted outcome before making design decisions.
Operational checklist
Section titled “Operational checklist”- Define the workload, user population, consequence level, operating horizon, data sensitivity, and required evidence before comparing tools.
- Build a small representative spike that includes failure paths, not only a happy-path demo.
- Test restart, cancellation, duplicate tool calls, late callbacks, policy rejection, human approval, and audit reconstruction.
- Confirm whether state, memory, traces, prompts, tools, evaluators, and release labels can be exported or migrated.
- Identify which behavior belongs to open protocols, framework code, hosted product features, and your own harness.
- Review licensing, deployment model, dependency update policy, security model, and support expectations.
- Avoid adopting a framework feature unless it removes an actual local responsibility or makes that responsibility easier to verify.
References and how they are used
Section titled “References and how they are used”- LangGraph documentation - project-maintainer documentation used for graph, state, persistence, streaming, and human-in-the-loop comparison dimensions.
- OpenAI Agents SDK documentation - OpenAI-maintained SDK documentation used for agents, tools, handoffs, guardrails, tracing, and sessions as comparison dimensions.
- Microsoft AutoGen documentation - project-maintainer documentation used for multi-agent and conversational orchestration comparison dimensions.
- LlamaIndex documentation - project-maintainer documentation used for data, retrieval, agent, and workflow comparison dimensions.
- CrewAI documentation - project-maintainer documentation used for crew, role, task, process, and flow comparison dimensions.
- Testing, Change, and Release - Fieldbook chapter used for release and compatibility questions that maintainer documentation alone cannot answer.