Article

Keep large JSON out of AgentCore prompts

For teams building AgentCore assistants, store large tool results outside prompts, retrieve needed records and check costs and answer completeness.

Editorial illustration for Keep large JSON out of AgentCore prompts: a controlled task flows from input to output. Not documentary evidence.

AWS’s November 17, 2026 walkthrough shows teams building Amazon Bedrock AgentCore assistants how to avoid repeatedly sending large JSON tool results—structured records such as schedules or operational reports—to their model. Keep full results in S3, put references in session state, and return a small preview with a tool that can retrieve the missing details.

This is an implementation pattern, not a new service or price cut. The companion repository supplies building blocks, not a complete agent. Its offline demo measures payload size without calling a model, so it cannot reproduce the walkthrough’s token or latency results.

Check whether built-in retrieval is enough

First identify who runs your agent’s decision loop. AgentCore Runtime hosts your own orchestration; the managed harness supplies the loop. A custom LangGraph application and a harness-based assistant therefore have different starting points.

For the managed harness, check its built-in retrieve_context tool. AWS documents previews for tool results above approximately 1,500 tokens, with retrieval of the full result when needed. That may already meet your needs; the documentation does not establish field-level paging or the sample’s S3 storage design.

Use custom retrieval when you need control over selected records, summaries or storage. Before deployment, check feature-specific regional availability and access to your chosen model. Runtime availability alone does not establish that Memory and the harness are available in the same Region.

Move the data before it enters conversation history

For the custom path, the sample’s integration instructions require Python 3.12 or later and, for real storage, an AWS account and S3 bucket. Keep the bucket private and encrypted, with read/write permissions limited to the artifact prefix. Then separate three jobs:

  1. Find the oversized responses. Inspect actual tool messages and model usage records. Keep small, useful results inline; choose a preview budget based on your records and model, not a universal characters-to-tokens ratio.

  2. Save the complete result outside the prompt. Store the JSON in S3 and retain its reference in application state. The reference is a handle for your retrieval tool, not a reason to send the full file back on every turn.

  3. Return the compact response before appending the tool message. Include the total record count, a bounded preview and the handle. Register a retrieval tool that can answer the follow-up requests your application permits.

The state conversion helper preserves conversation messages. Moving a separate application payload into S3 does not remove a large result already embedded in those messages. Inspect the history you persist as well as the live application data.

At turn end, call dehydrate_state, upload its returned artifacts, fill their reference fields and write the lightweight state to Memory. On resume, restore state and load payloads only when needed. The helper emits only request_raw and responses_raw as new artifacts; other payload fields need explicit upload and lazy-loading integration. It does not perform Memory writes for you.

For the separate question of whose conversation should persist, see per-user memory for a Bedrock assistant. Persistence and prompt-size control are different jobs.

Finish the sample’s retrieval and size controls

Do not return a handle unless the model has a working way to use it. Three details in the published sample matter:

  • Dictionary retrieval is unfinished. In the paging function, field is unused and a stored dictionary returns None. Implement field or path selection before relying on dictionary previews, or limit the integration to supported list results.

  • A sample is not a size cap. Lists longer than five items get a five-item preview; dictionary compaction begins above 7,600 serialized characters. These are sample defaults, not AgentCore quotas. Large individual items or scalar strings can still make the preview too big. Validate offsets, cap page length and check the final serialized response size.

  • Paging does not limit the S3 download. The artifact store fetches and parses the complete JSON object on an uncached read, then the paging function slices it. Its in-process cache has no size bound. Cap that cache and budget for cold-read memory; for larger workloads, consider separately stored chunks or server-side filtering as extensions to the sample.

Preserve complete answers and the original permissions

The first five records cannot answer a question about every record. For example, a scheduling assistant asked whether any assignments are missing should run a tool over the full authorized dataset, not infer completeness from the preview. A useful response design would return the exceptions, the number of records examined and any processing failures. Use pages for inspection; use full-data calculations for totals and completeness checks.

The sample’s artifact references are content hashes, not ownership checks. Authorize each lookup against the authenticated user or session, including cache hits. Keep storage permissions scoped, and return explicit missing or expired-reference errors. Copying a result into S3 must not broaden who can retrieve it.

If the source is Snowflake, that access decision begins upstream: see user versus service permissions from AgentCore. Apply the relevant access boundary to the stored copy too.

Budget for retrieval, storage and extra model calls

Smaller prompts are only one part of the bill. Use current Bedrock model prices for ordinary input, output, cache reads and cache writes in both versions of the workflow. Extra retrieval turns can offset the input tokens removed. If the original input was cached, do not value its removal at the ordinary input rate.

This is distinct from when Bedrock prompt caching saves money: caching changes the cost of repeated processing, while external retrieval changes what you send to the model.

Add S3 storage and request charges, plus applicable networking costs. Stable content hashes do not eliminate all writes: the sample skips repeat uploads only when the content is already in that process’s cache. A fresh process can issue another PUT for the same content.

Include AgentCore runtime, Memory and telemetry costs for the features you use. Short-term Memory ingestion and retrieval have a 12 KB minimum billable size per event, so shrinking an already-small event does not necessarily reduce those charges. Storage is billed on actual size over its retention period.

Align handle expiry with the underlying records’ retention policy. In a versioned S3 bucket, deleting the current object can leave older versions stored; include those versions in retention and cost planning.

Validate the whole task before rollout

AWS and Flightscape report 64% fewer total tokens and about 69% lower average trace latency across 100 sample questions. Those are their workload results, not a forecast for your application or a measured reduction in the complete AWS bill.

For your rollout decision, compare the same sanitized records and questions using the same model and settings. Include questions that need one page, a nested field and the whole dataset. Check the answers against known complete results, including records beyond the preview. Record input, output and cache tokens, retrieval calls, end-to-end latency and infrastructure costs, with both cold and warm storage caches.

Start with one oversized tool response. Expand only when the full task remains correct and the total cost or response time improves. If the model repeatedly fetches the entire result, a purpose-built filtering or aggregation tool may be more useful than a smaller first message.