AI brokers run on file techniques utilizing customary instruments to navigate directories and skim file paths.
The problem, nonetheless, is that there is a number of enterprise knowledge in object storage techniques, notably Amazon S3. Object shops serve knowledge by API calls, not file paths. Bridging that hole has required a separate file system layer alongside S3, duplicated knowledge and sync pipelines to hold each aligned.
The rise of agentic AI makes that problem even more durable, and it was affecting Amazon’s personal capability to get issues performed. Engineering groups at AWS utilizing instruments like Kiro and Claude Code stored working into the identical drawback: Brokers defaulted to native file instruments, however the knowledge was in S3. Downloading it domestically labored till the agent’s context window compacted and the session state was misplaced.
Amazon’s reply is S3 Information, which mounts any S3 bucket instantly into an agent’s native setting with a single command. The info stays in S3, with no migration required. Beneath the hood, AWS connects its Elastic File System (EFS) know-how to S3 to ship full file system semantics, not a workaround. S3 Information is obtainable now in most AWS Areas.
“By making knowledge in S3 instantly obtainable, as if it is a part of the native file system, we discovered that we had a very massive acceleration with the capability of issues like Kiro and Claude Code to have the ability to work with that knowledge,” Andy Warfield, VP and distinguished engineer at AWS, advised VentureBeat.
The distinction between file and object storage and why it issues
S3 was constructed for sturdiness, scale and API-based entry at the object degree. These properties made it the default storage layer for enterprise knowledge. However additionally they created a elementary incompatibility with the file-based instruments that builders and brokers rely on.
“S3 is not a file system, and it would not have file semantics on an entire bunch of fronts,” Warfield mentioned. “You possibly can’t do a transfer, an atomic transfer of an object, and there aren’t truly directories in S3.”
Earlier makes an attempt to bridge that hole relied on FUSE (Filesystems in USErspace), a software program layer that lets builders mount a customized file system in consumer house with out altering the underlying storage. Instruments like AWS’s personal Mount Level, Google’s gcsfuse and Microsoft’s blobfuse2 all used FUSE-based drivers to make their respective object shops appear like a file system.
Warfield famous that the drawback is that these object shops nonetheless weren’t file techniques. These drivers both faked file habits by stuffing further metadata into buckets, which broke the object API view, or they refused file operations that the object retailer could not help.
S3 Information takes a special structure completely. AWS is connecting its EFS (Elastic File System) know-how instantly to S3, presenting a full native file system layer whereas holding S3 as the system of file. Each the file system API and the S3 object API stay accessible concurrently towards the identical knowledge.
How S3 Information accelerates agentic AI
Earlier than S3 Information, an agent working with object knowledge had to be explicitly instructed to obtain recordsdata before utilizing instruments. That created a session state drawback. As brokers compacted their context home windows, the file of what had been downloaded domestically was typically misplaced.
“I’d discover myself having to remind the agent that the knowledge was obtainable domestically,” Warfield mentioned.
Warfield walked by the before-and-after for a standard agent activity involving log evaluation. He defined {that a} developer was utilizing Kiro or Claude Code to work with log knowledge, in the object solely case they would want to inform the agent the place the log recordsdata are situated and to go and obtain them. Whereas if the logs are instantly mountable on the native file system, the developer can merely determine that the logs are at a selected path, and the agent instantly has entry to undergo them.
For multi-agent pipelines, a number of brokers can entry the identical mounted bucket concurrently. AWS says hundreds of compute assets can join to a single S3 file system at the identical time, with mixture learn throughput reaching a number of terabytes per second — figures VentureBeat was not ready to independently verify.
Shared state throughout brokers works by customary file system conventions: subdirectories, notes recordsdata and shared challenge directories that any agent in the pipeline can learn and write. Warfield described AWS engineering groups utilizing this sample internally, with brokers logging investigation notes and activity summaries into shared challenge directories.
For groups constructing RAG pipelines on high of shared agent content material, S3 Vectors — launched at AWS re:Invent in December 2024 — layers on high for similarity search and retrieval-augmented technology towards that very same knowledge.
What analysts say: this is not only a higher FUSE
AWS is positioning S3 Information towards FUSE-based file entry from Azure Blob NFS and Google Cloud Storage FUSE. For AI workloads, the significant distinction is not primarily efficiency.
“S3 Information eliminates the knowledge shuffle between object and file storage, turning S3 right into a shared, low-latency working house with out copying knowledge,” Jeff Vogel, analyst at Gartner, advised VentureBeat. “The file system turns into a view, not one other dataset.”
With FUSE-based approaches, every agent maintains its personal native view of the knowledge. When a number of brokers work concurrently, these views can doubtlessly fall out of sync.
“It eliminates a complete class of failure modes together with unexplained coaching/inference failures attributable to stale metadata, which are notoriously tough to debug,” Vogel mentioned. “FUSE-based options externalize complexity and points to the consumer.”
The agent-level implications go additional nonetheless. The architectural argument issues lower than what it unlocks in observe.
“For agentic AI, which thinks by way of recordsdata, paths, and native scripts, this is the lacking hyperlink,” Dave McCarthy, analyst at IDC, advised VentureBeat. “It permits an AI agent to deal with an exabyte-scale bucket as its personal native exhausting drive, enabling a degree of autonomous operational velocity that was beforehand bottled up by API overhead related to approaches like FUSE.”
Past the agent workflow, McCarthy sees S3 Information as a broader inflection level for a way enterprises use their knowledge.
“The launch of S3 Information is not simply S3 with a brand new interface; it is the elimination of the remaining friction level between large knowledge lakes and autonomous AI,” he mentioned. “By converging file and object entry with S3, they are opening the door to extra use circumstances with much less transforming.”
What this implies for enterprises
For enterprise groups which were sustaining a separate file system alongside S3 to help file-based purposes or agent workloads, that structure is now pointless.
For enterprise groups consolidating AI infrastructure on S3, the sensible shift is concrete: S3 stops being the vacation spot for agent output and turns into the setting the place agent work occurs.
“All of those API modifications that you simply’re seeing out of the storage groups come from firsthand work and buyer expertise utilizing brokers to work with knowledge,” Warfield mentioned. “We’re actually singularly centered on eradicating any friction and making these interactions go in addition to they’ll.”
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
