That one request involves both of AIStore’s security perimeters. Network path and security authority do not align hop for hop: a client may cross the public network carrying a gateway-signed handoff that is verified at the inner perimeter.

An AIS cluster has three networks and two security perimeters. The numbers do not match, and the mismatch is not an accident of naming. The networks classify where traffic arrives and keep traffic with very different performance characteristics out of each other’s way. The perimeters establish whose authority a request carries: that of an external user, or that of a current cluster member. Those are different questions, and the second one is why a client can end up holding a cluster-member signature.

Most requests fit the pattern you would guess. Clients use the public network; cluster nodes use the intra-cluster networks. The interesting cases are the ones that do not. A client following a redirect arrives directly at a storage node carrying a gateway-signed handoff. A single API call may begin on the public network, fan out over intra-control, and move its payload over intra-data. A node that crash-restarts keeps its stable node ID and acquires an entirely new cryptographic identity.

Those cases are why AIS 5.x implements network separation, request authentication, and cluster membership as a single security model rather than three independent mechanisms.

Planned staged rollout

Unless noted otherwise, this text describes AIStore 5.1 and later. Version 5.0 is covered only as the compatibility bridge: it establishes the network separation and key propagation that enforcement depends on, with signing itself disabled. Version 5.1 follows shortly.

Contents

• Architecture
  • One cluster, two node types
  • Three networks
  • Two perimeters
• Request paths
  • Get-batch: one request across all three networks
  • A normal object request
  • What the target decides, and in what order
  • Control messages and data streams
  • Transport streams use a separate signed envelope
  • S3 compatibility: another API, another set of semantics
• Per-node identity
  • From a cluster secret to per-node identity
  • Restart, rejoin, and key propagation
  • Why 5.0 is the bridge
• Future work
• Recap

One cluster, two node types

An AIS cluster consists of gateways and storage targets.

Gateways are diskless cluster coordinators. They accept public API requests, authenticate and authorize users when configured to do so, maintain and distribute cluster-level metadata, coordinate cluster-wide operations, and select storage targets. On the normal native object path a gateway redirects the client to a target rather than relaying object data itself.

Targets store objects on locally attached filesystems. They manage mountpaths, perform local I/O, and move data for replication (ais cp), erasure coding, global rebalance, ETL, get-batch, and other distributed operations.

Every gateway and target holds the cluster map, or Smap. The Smap identifies the current members and their network endpoints. It also provides the shared input for highest-random-weight placement: given the same fully qualified (bucket and object) name and the same Smap, every node independently selects the same target. There is no per-object metadata server and no object-location lookup.

That shared membership view turns out to be the foundation of the inner security perimeter, though it was not built for that.

Three networks

AIS has distinguished three traffic domains since its original architecture, and the original reason had nothing to do with security. It was bandwidth and latency. Bulk data movement between targets will starve the small messages that hold a cluster together - keepalives, membership updates, coordination - if they share a pipe, and a cluster that cannot hear its own heartbeat under load is a cluster that falls apart exactly when it is busiest. Separating the traffic classes was a performance decision circa 2018.

A production deployment commonly assigns separate interfaces and addresses to all three. They may use different subnets, VLANs, physical fabrics, or firewall domains while retaining the same port number on each interface. Smaller deployments can keep them on one host and one physical interface while still using distinct endpoint addresses. Intra-control and intra-data may also share a single HTTP endpoint; they remain distinct traffic roles even when they use the same listener.

What changed in 5.0 is that public and intra-cluster traffic can no longer collapse onto one listener. At least one intra-cluster endpoint must be configured, and it must be distinct from the public endpoint.

The reason is structural. A server can trust the listener that accepted a connection, because the server created that listener and assigned its role. It cannot obtain the same assurance from a header or a URL, both of which the caller supplies. Remote addresses are no better: proxies, NAT, containers, and shared hosts all transform or obscure them. If presenting an internal-only header is what makes a request trusted, then a public caller who presents the same header is trusted too - and that is not an exotic attack, it is the direct consequence of asking caller-controlled data to identify which side of a boundary the caller is on.

AIS therefore records the arrival network when a connection is accepted and carries that fact with every request handled on the connection. This does not identify the caller. It establishes the traffic domain within which the caller must then be authenticated.

Separate interfaces, VLANs, and firewalls can further restrict who is able to reach each network. When enabled, TLS protects confidentiality and integrity in transit and authenticates the server endpoint. Neither replaces application identity: user tokens establish user identity, node signatures establish current-member authority.

Two perimeters

The outer and inner perimeters answer different questions.

The outer perimeter handles original user requests arriving at a gateway over the public network. It validates the user’s token and applies the required cluster or bucket permission. AIStore AuthN supports JWT validation, role-based cluster and bucket ACLs, token revocation, mandatory expiration, RSA signing, and OIDC issuer discovery.

The inner perimeter handles authority derived from current cluster membership. A node signs with its own Ed25519 private key; the receiver verifies against that node’s public key in its current Smap. Some operations additionally require a particular cluster role - for example, that the signer be the primary gateway.

The inner perimeter appears in two forms:

1. A gateway or target sends a request directly to another cluster member.
2. A client presents a request envelope previously signed by a gateway, as happens after an object redirect.

The second form is the one worth pausing on. The client originates the HTTP request to the target, over the public network, with no membership credentials of its own. The authority that permits the request belongs to the gateway that signed the envelope the client is carrying. Arrival network and request authority are therefore not the same property, and a design that conflates them cannot express this case at all.

The perimeters are not concentric filters applied to every HTTP request. Each message has one provenance and follows the corresponding security path. A larger client operation can cross both, through a sequence of messages.

Get-batch: one request across all three networks

A plain read (GET) request follows a short path: the gateway authorizes the request, selects the target, and redirects the client. AIS get-batch is maybe a more revealing example: one API call uses all three networks and crosses both security perimeters without collapsing them into a single boundary.

This is a single API call to retrieve multiple objects, object ranges, and archived files - in one shot, as one assembled output stream. The requested data may belong to different buckets, may be sharded (archived), may (and most likely will) be distributed across nodes. One target is selected as the designated target, or DT. The remaining N - 1 targets act as senders: they read their local content and stream the requested payloads to the DT, which assembles the result and returns it to the client.

The operation has three phases:

Only the designated target (DT) returns data to the client; the target senders never communicate with the client directly. The dashed boundaries are conceptual. The inner perimeter is drawn inside the outer one for layout only - a request is not filtered by both in sequence. Each message crosses the perimeter corresponding to the authority it relies on.

Phase 1: initialize the designated target

The client sends the get-batch request to a gateway over the public network. The request body may contain entries such as:

{ bucket: A, objname: b/c/d.flac }
{ bucket: E, objname: f.tgz, archpath: samples/g.wav }
{ bucket: H, objname: i.tar, start: 1234, length: 5678 }

The gateway parses the request, resolves and authorizes every referenced bucket, and selects the designated target. Depending on the requested colocation level, that selection may be derived from the input objects or made independently using the current Smap.

The gateway then sends a signed intra-control POST to the DT. This is not yet the client data path. It is cluster coordination: the DT renews or creates the get-batch xaction, opens the long-lived peer-to-peer connections used by the transport subsystem, prepares its receive-side state, and returns the real xaction ID.

The gateway is therefore acting under two different authorities during this phase. It first handles the original request under the user’s authorization at the outer perimeter. It then addresses the DT as a current cluster member at the inner perimeter, signing the intra-control request with its own Ed25519 key.

No user data is returned in phase 1.

Phase 2: redirect the client to the designated target

Once the DT has initialized its receive-side state, the gateway redirects the client’s original GET to that target, normally over the public network.

A configured local-CIDR optimization may instead use the target’s intra-control endpoint.

The redirect carries the xaction ID, a per-request work ID, the expected target count, and the gateway’s signed handoff. The client does not become a cluster member and does not receive the gateway’s private key. It merely carries the signed redirect URL from one public endpoint to another.

At the DT, the request arrives over the public network from a client, not a cluster member. The DT nevertheless accepts the handoff because it carries a gateway signature. The latter identifies the signing gateway in the current Smap and verifies the signature using that gateway’s recorded verifying key before beginning assembly.

The DT then starts producing the response stream. Some requested objects (or archived files) may already be local. Others will arrive from target senders during phase 3.

Phase 3: start the target senders

After issuing the redirect, the gateway asynchronously broadcasts a signed intra-control POST to every active target except the DT.

Each target sender renews the same get-batch xaction by ID, opens its peer-to-peer transport connections, reads the local data assigned to it, and streams those payloads directly to the DT over the intra-data network.

This phase illustrates why intra-control and intra-data remain separate roles even though both are inside the inner perimeter.

The gateway’s broadcast is control-plane work: small HTTP requests that identify the xaction, the DT, and the work to start. Those requests use the same node-level HTTP signing and verification mechanism as other intra-control operations.

The resulting data streams are data-plane work: large, long-lived, and carried by the transport subsystem rather than ordinary request-response HTTP. They are authenticated as well, but by a different envelope signed when the connection opens - the subject of a later section. Both mechanisms establish current-member authority; they authenticate different protocols and different facts.

One operation, two perimeters

Get-batch is one client operation, but there is no single end-to-end trust decision that covers all of its traffic.

The gateway authorizes the original request at the outer perimeter. The DT trusts the redirected phase-2 request because it carries a valid gateway-signed handoff. The targets trust phase-1 and phase-3 coordination because the HTTP requests are signed by a current cluster member. The DT trusts incoming phase-3 streams because their transport envelopes are independently signed by the sending targets.

The three networks classify where each part of the operation belongs:

This is the architectural pattern in its fullest form: one cluster, two security perimeters, and three network roles participating in one logical request.

A normal object request

A plain-vanilla GET(object) is the same model in its simplest form, step by step:

1. The client sends the request to a gateway over the public network.
2. At the outer perimeter, the gateway authenticates the user and checks read permission for the bucket.
3. The gateway uses the current Smap and the object name to select the responsible target.
4. The gateway signs an envelope covering the request and returns a redirect.
5. The client follows the redirect to the target.
6. At the inner perimeter, the target verifies that a current gateway signed the envelope. It does not repeat the user-token and bucket-ACL checks.
7. The target reads and returns the object directly.

In normal production deployments the redirect names the target’s public endpoint, because that is the target address a client can reach (see “local-CIDR” comment above).

A PUT follows the same pattern with a method-preserving redirect. The gateway checks write permission, selects a target, and signs. The target verifies the handoff before accepting and finalizing the object. If the bucket requires replication, erasure coding, or other derived work, the target then acts under its own node identity when it talks to its peers - the user is no longer speaking to storage at that point; a cluster member is.

The mapping is therefore not one network to one perimeter:

The last row deserves a note. A target’s public endpoint must be reachable, because redirected clients have to reach it. Being reachable is not the same as being open: when gateway mediation is required, a request that names the target directly and carries no gateway envelope does not get to skip the outer perimeter by choosing a different address.

What the target decides, and in what order

The natural assumption is that the arrival network decides how a request is checked. It does not - at least, not first. A target receiving an object request asks three questions in a fixed order, and the order is the whole three-into-two problem in miniature:

   object request arrives at a target
                │
                ▼
     ┌──────────────────────┐
     │ signing enforced?    │──── no ────▶ compatibility path (5.0 bridge)
     └──────────┬───────────┘
                │ yes
                ▼
     ┌──────────────────────┐
     │ gateway envelope?    │──── yes ───▶ verify the gateway's signature
     └──────────┬───────────┘              (arrival network not decisive)
                │ no
                ▼
     ┌──────────────────────┐
     │ arrived intra-net?   │──── no ────▶ reject direct target access
     └──────────┬───────────┘              (when gateway mediation is required)
                │ yes
                ▼
       verify the peer's signature against its
       verifying key in the current Smap

A redirected client request is authorized by the gateway’s signature and deliberately not by where it landed. This matters because where it lands is genuinely variable: normally the target’s public endpoint, sometimes the intra-data one, and when a deployment collapses its two intra-cluster endpoints into one, traffic that logically belongs to intra-data legitimately arrives on the control listener. Any rule of the form “requests on this network are peer requests” would be wrong in all three configurations.

The arrival network still decides something, but only after the envelope question has been answered. A request with no gateway envelope, arriving on the public listener, is a client trying to talk to storage directly - and that is where the network check does its work.

Control messages and data streams

Many public API calls trigger several internal messages.

For a distributed list-objects request, a gateway first applies the outer perimeter to the user’s request. It then fans the operation out to the targets over intra-control. Each target verifies the gateway as a current cluster member before contributing its page of results, and the gateway merges those responses for the client.

Keepalive, metasync, primary elections, membership changes, and xaction coordination do not represent a user speaking to storage at all. They belong entirely to the inner perimeter: there is no user token, no bucket ACL, and no delegated user authority involved. Their handlers can rely on both the intra-cluster arrival domain and the verified identity of the sending node. In this case, network role and security authority coincide - unlike a redirected user GET or PUT, which normally arrives over the public network carrying a gateway-signed handoff.

Bulk intra-cluster data movement uses the same member identity in a different traffic role. Examples include offline ETL, global rebalance, and the previously described get-batch.

These operations exchange setup and coordination messages over intra-control, then move object payloads over intra-data. Long-lived transport connections are authenticated when they are opened and whenever they are re-established. The signed opening binds the transport name, sender ID, session ID, Smap (cluster map) version, and nonce. Once established, the connection carries object payloads for its lifetime.

Transport streams use a separate signed envelope

HTTP requests and intra-cluster transport streams use the same per-node Ed25519 identity, but they authenticate different things. The HTTP envelope covers one request: its method, path, sender, Smap version, content length, and nonce. It may authenticate either a direct peer request or a gateway-signed redirect carried by a client.

A transport stream is different. It is a long-lived connection between targets, accepted only on an intra-cluster listener, and may carry many objects over its lifetime. Authentication therefore happens when the connection is opened and whenever it is re-established. The signed stream opening binds the transport name, sender ID, session ID, Smap version, and nonce. It also uses a separate signature domain, preventing an HTTP signature from being replayed as proof for a transport connection.

The stream payload uses an explicit protocol domain (ais-transport-v1) to prevent cross-protocol signature reuse.

The two mechanisms thus share node identity and verification policy, but sign different facts: HTTP signing authenticates an individual request, while transport signing authenticates a member opening a long-lived intra-cluster channel.

S3 compatibility: another API, another set of semantics

Putting another API on the front of a storage system adds more than another set of URLs. S3 has its own request semantics: operations encoded in query parameters, lists of object names carried inside XML bodies, multipart state distributed across storage nodes, and clients with different redirect behavior. The gateway must translate those requests into native AIStore redirects, fanouts, and bulk operations.

Whenever the underlying security model changes, that translation layer deserves a separate review. An invariant that is obvious in the native API may no longer be obvious - or even visible - after an S3 request has been transformed.

ListMultipartUploads created one such case. Although the operation returns metadata, the target-side S3 endpoint is served over the public and intra-data networks, not intra-control. With multiple targets, the gateway queries every target over intra-data and aggregates their locally known uploads.

The common request helper, however, decided whether to stamp and sign an intra-cluster request by checking whether its destination matched the target’s intra-control URL. An explicit intra-data destination did not match, so the multipart-listing fanout went out unsigned. Every target correctly rejected it. The aggregation path treated those rejections as empty results, and the client received a successful response reporting zero uploads when three existed.

The fix was to recognize both of a member’s intra-cluster endpoints. The inner perimeter covers intra-control and intra-data alike; the network determines the traffic role, not whether member authentication is required.

Multi-object delete exposed a different translation boundary. An ordinary S3 object request carries its object name in the URL, where the target-side S3 handler can validate it before constructing local object metadata. A multi-object-delete request instead carries many keys inside an XML document. The gateway parses that document, extracts the names, converts them into an internal bulk-delete message, and broadcasts the operation.

By that point, the names are no longer part of an S3 object URL, so the normal target-side S3 path validation cannot be assumed to have covered them. Every key is now validated explicitly before it is admitted into the internal bulk-delete request.

Neither issue required novel cryptography. Both came from a semantic change at the API boundary: an intra-data call was mistaken for a non-intra call, and object names moved from a validated URL path into a decoded request body. The S3 compatibility API is therefore its own security surface, not merely another route to the same handlers.

From a cluster secret to per-node identity

AIS 4.x authenticated intra-cluster messages with one persistent symmetric key. Every node held the same key and used it to generate and verify HMACs.

That design could distinguish a cluster insider from an outsider, but it could not distinguish one member from another. A shared secret gives every node the same signet ring: a valid seal proves that someone holding the cluster secret stamped the request, and nothing more. Any node could produce a message attributed to any other node. Compromise of one copy exposed the cluster’s only secret, and rotation meant redistributing a replacement to every member.

AIS 5.0 replaces the shared secret with one Ed25519 keypair per node. A node alone holds its signing key. Its verifying key is published with its identity in the Smap. A receiver can therefore verify not merely that a request arrived with cluster credentials, but which current member authorized it.

Private keys are ephemeral: each node generates a new keypair upon startup and keeps the private half in memory only - never writes it down. This removes an entire category of secret management: no private-key files to protect, no durable copies to reconcile across mountpaths, no filesystem metadata holding key material, nothing left on disk after a node stops.

The Smap is the natural distribution vehicle for the public half. It already distributes node IDs, roles, endpoints, flags, and membership changes; adding the verifying key keeps membership and member identity in the same versioned object instead of introducing a separate key-distribution service that would need its own trust bootstrap. The consequence - and it is a large one - is that a key change is now a membership event.

Restart, rejoin, and key propagation

A node ID is stable across restart. Its (public, private) keypair is not.

When a node rejoins, the primary gateway must compare both its network information and its verifying key against the current Smap entry. A changed key requires a renewed entry even when the node ID, endpoints, role, and flags are all unchanged. The updated Smap then propagates, and once a receiver holds the new entry it verifies against the new key and stops accepting the old one.

The ordering matters during a quick crash and restart:

1. The restarted node has already generated and begun using its new signing key.
2. A peer may still hold an Smap containing the old verifying key.
3. Until the updated Smap reaches that peer, a correctly formed signature from the restarted node is cryptographically invalid from the peer’s point of view.

This is different from a missing signature during a narrowly defined bootstrap or compatibility path. An absent signature can be handled by the rules for that transitional state. An invalid signature cannot be treated as absence, because doing so would convert failed authentication into an authentication bypass - which is the one thing this entire model exists to prevent.

The same comparison is required during primary startup and cluster reconstruction. Any registration path that decides a node is unchanged based only on its ID and endpoints can leave the Smap holding a stale verifying key.

In the 5.x model, current node identity is the stable node ID, plus current membership, plus the verifying key recorded for that member in the current Smap. Key rotation has become membership propagation.

Why 5.0 is the bridge

The rollout is deliberately staged.

Version 5.0 establishes the prerequisites: separate public and intra-cluster arrival domains, and a cluster map format that carries per-node verifying keys. Signature enforcement remains disabled while the cluster moves across that shared-metadata boundary.

Version 5.1 then allows to enable signing and verification (once every member understands the new identity model).

A direct 4.x-to-5.1 upgrade is therefore unsupported (and not permitted).

Future work

The authenticated nonce already carried by signed requests is a foundation for replay handling, but replay rejection requires receiver-side state and a policy that accounts for concurrent requests, retries, out-of-order arrival, multiple destinations, memory limits, and node key epochs. A naive “reject anything not strictly newer” check would look reassuring while rejecting a great deal of valid traffic.

The signature envelope currently covers the method, path, sender, Smap version, content length, and nonce. Future extensions can add canonical query coverage, and body digests where appropriate.

The default model will likely remain: authenticate the gateway handoff, bind routing-critical fields, and let the target perform ordinary semantic validation. Consistent with AIS design and implementation, the TBD feature flag would strengthen that to cryptographically bind additional request semantics - at an added datapath cost.

A third area that, in theory, may require further investigation is a quick node restart, when the primary did not have enough time to notice the loss. Generally, a node should delay signing its control-plane messages until its newly generated verifying key has been distributed with the current Smap. The receive side already distinguishes missing startup credentials from invalid signatures; a send-side readiness barrier would make the transition explicit from both ends.

Recap

In AIS, the three networks were always separated to keep user requests, cluster coordination, and bulk data movement out of each other’s way. Eight years later, that same separation turned out to be the structure needed to apply the right security policy to each connection - this is the honest fact.

The outer perimeter authenticates users and enforces their permissions. The inner perimeter authenticates current cluster members and the handoffs they authorize. A single operation may cross from one to the other, but the two checks remain distinct, and no hop is asked to answer both questions at once.

The cluster map joins the model together. It gives every node the same view of membership and placement, distributes the verifying keys that belong to that membership, and turns a cluster member’s identity into something every peer can check for itself.

In the benchmarks, sendfile reduced CPU time per GiB transferred by up to 70% on AIS target nodes serving GET responses. When network bandwidth is not the bottleneck, those CPU savings translate directly into higher throughput, with benchmarks showing up to a 3x increase. Even during a network bottleneck, the reclaimed CPU can be used for colocated work such as ETL, jobs, and checksum validation.

Table of Contents

• Background
• Benchmarks
  • Object Size
  • Throughput Gains
• Limitations
  • TLS
  • Chunked Objects
  • Range Reads
  • Transformed Responses
• Conclusion
• References

Background

For every GET request to AIStore, it first reaches an AIS proxy, which redirects the request to the target that owns the object. On a warm GET, where the object is already local, the target serves the response through the regular transmit path:

1. Load the object’s local metadata.
2. Open a reader for the local object.
3. Construct the HTTP response headers from the object metadata.
4. Read object data from the local file into a reusable userspace buffer.
5. Perform any required processing on the buffered data.
6. Write the buffer to the response writer, which sends the bytes to the client socket as the HTTP response body.
7. Repeat until the entire object has been transmitted.

This process is illustrated in the diagram below, but a more in-depth overview is available in AIStore documentation.

By passing object data through userspace, the transmit path supports additional request handling such as HTTPS and chunked object reads, though not every GET request needs that flexibility. When a request can be served as a plain HTTP transfer of a large, local, monolithic object, copying data through a userspace buffer becomes unnecessary overhead.

The sendfile path removes that extra copy. AIS still performs the same request handling, but the CPU no longer spends additional cycles copying every byte through a userspace buffer. This reduction in CPU usage within the target’s data transfer component provides significant savings for the overall GET request.

Benchmarks

Setup

The benchmarks used a 3-node Kubernetes cluster with three AIS targets and three proxies. Each node ran Ubuntu 20.04 on a 24-core machine with about 1 TB of RAM and a 100 Gb/s NIC (corresponding to a maximum bandwidth of around 11.6 GiB/s).

The two paths were compared using separate AIS node container images: one built with sendfile enabled and one with it disabled. Before each run, the target aisnode process was traced with strace to confirm it was taking the expected path (sendfile vs userspace read/write on the response body).

All benchmark objects were prepopulated into AIS buckets before the GET tests so that the results reflect what sendfile directly impacts: warm GETs on local objects. Every GET targeted objects owned by a single chosen target node, and that node was monitored for the results reported below.

The reported CPU and throughput numbers come from target-node sar data collected and processed offline to filter out unrelated cluster activity during each run. Alongside those numbers, visualizations from the AIS monitoring stack are also included to provide a high-level overview of cluster performance during each run.

Object Size

Object size is one of the largest factors in the effectiveness of sendfile. Each GET carries the same fixed per-request overhead (object lookup and locking, metadata load, HTTP header construction) while sendfile only reduces the cost of copying the object bytes into the response body, so the benefit is easier to see when the body is large enough for that copy cost to dominate the fixed per-request work.

To measure this effect, the regular transmit path was compared with the sendfile path for object sizes ranging from 64 KiB to 4 GiB, increasing by a factor of 4 at each step. Each size was tested for 3 minutes with 120 concurrent workers.

Both the transmit and sendfile object-size runs quickly reached a network ceiling of roughly 11.3 GiB/s. Because throughput was capped by the network, this test did not reveal any significant throughput differences between the two paths. Instead, the useful comparison is CPU cost at equal throughput: at comparable network throughput, the target node consumed meaningfully less CPU with sendfile.

The plots below normalize target CPU usage as CPU-seconds per GiB transferred and show the relative CPU reduction of sendfile compared with transmit.

For 64 KiB objects, sendfile showed no benefit in this run as the measured CPU reduction was slightly negative. That is expected for very small responses because the fixed HTTP and AIS request costs dominate the body-copy savings. The CPU reduction rates peaked at around 20% for object sizes over 256 MiB, which suggests that per-request overhead is small relative to body transfer beyond that point.

There were minor run-to-run fluctuations from variations in network latency, but the data did show a consistent trend: sendfile is more CPU-efficient than the transmit path, especially for larger object sizes.

Throughput Gains

The object-size benchmark did show CPU savings, but the network ceiling was masking potential throughput gains. To eliminate the network bottleneck, a second benchmark was conducted with the AIS client colocated on the same node as the AIS proxy and target. This setup isolates the test from network limits and shows the pure performance improvement that sendfile brings.

While running the AIS client on the same node as the AIS proxy and target is not a typical AIStore deployment, such a setup simulates environments where target nodes have enough network capacity that CPU becomes the limiting resource.

In the second benchmark, each test ran for 5 minutes against a bucket of 1 GiB objects, with worker count swept from 50 to 800 (doubling at each step).

In this CPU-bound setup, the results show a significant difference in throughput. The transmit path achieved 15–17 GiB/s while the sendfile path reached 44–50 GiB/s at roughly the same high CPU utilization. These results indicate that without network limitations, the reduced CPU cost translates directly into higher GET throughput.

The normalized CPU plots show a 62-72% reduction in target CPU-seconds per GiB for sendfile compared with transmit. The reduction percentage is higher here than in the previous benchmark because the CPU-bound setup produces a compounding effect: transmit spends more CPU per delivered GiB, while sendfile keeps CPU cost lower and converts the saved CPU into additional throughput.

Limitations

The sendfile path is intentionally narrow. AIS only uses it when the request can be served as a correctly framed HTTP response whose body is a direct view of a local file. When AIS needs to inspect, transform, encrypt, or reassemble bytes before sending them, it falls back to the regular transmit path.

TLS

sendfile cannot be used for HTTPS traffic. With TLS, object bytes must be encrypted before reaching the client, which means AIS must continue reading data into userspace so the TLS stack can encrypt it before writing it to the socket.

Chunked Objects

AIS chunked objects are represented by a manifest and multiple backing chunks, not by one continuous local file that can be passed directly to sendfile. For chunked object reads, AIS has to resolve the object layout, open the relevant chunks, and stream them in the correct order. That reassembly work requires AIS-controlled readers, so the request remains on the transmit path.

Range Reads

Range reads can use sendfile when AIS only needs to validate the HTTP Range header, set the 206 Partial Content response headers, seek to the requested offset, and transfer the requested byte range directly. When AIS must compute or validate a checksum for the requested range, it may need to read the range first, materialize it, and then send it. In those cases, the transfer is no longer a pure file-backed response body, so AIS has to fall back to the transmit path.

Transformed Responses

sendfile does not apply when the response body is not the original local object file. Examples include archive extraction, ETL-transformed reads, AIS Select-style processing, and other cases where AIS must generate a response from object data. For these requests, AIS needs to inspect or transform the bytes before sending them to the client. Since the response is produced by AIS rather than being a direct view of the local file, the userspace transmit path remains necessary.

Conclusion

On eligible warm GETs, sendfile lowered target CPU-seconds per GiB by as much as 72% and raised throughput by roughly 3x. Those peaks reflect the best-case scenario from the benchmarks: CPU-bound deployments serving large, local, monolithic objects. In network-bound runs, sendfile delivered comparable throughput while reducing target CPU cost by about 20% for large objects.

Despite the performance gains, the sendfile fast path is reserved only when the response body is a direct view of the requested file and can be transferred without userspace processing. All other GET requests automatically fall back to the transmit path. That narrow eligibility is the main limitation on sendfile in AIStore today. Additionally, the benchmarks showed that even eligible GETs do not benefit equally, as object size and network saturation significantly influence the effectiveness of sendfile.

References

AIStore

• AIStore Documentation
• Parallel Download: 9x Lower Latency for Large-Object Reads
• ETL
• Jobs CLI
• AIS Monitoring Stack

Linux

• sendfile(2)
• strace(1)
• sar(1)

Symmetric signing algorithms like HMAC can be used to securely sign access tokens, but with two extremely important caveats:

1. The secret key must be strong enough to avoid simple brute-force attacks (high entropy, sufficiently long, and randomly generated)
2. The secret key must NEVER be leaked

Unfortunately for the first point, hashcat received its public release in 2009. Since then, advances in GPU hardware and frameworks like NVIDIA CUDA have turned tools such as hashcat into increasingly effective brute-force engines, capable of breaking secrets that were once considered safe. Still, given a sufficiently long and random key, this is not a concern with HMAC-SHA256.

The second issue is a much larger problem for symmetric signing approaches like HMAC. Since the signing key is also used for validation, it must be provided to the server, not just the token issuer. And this secret key must never be accidentally exposed in deployment pipelines, configuration files, or logs. This increased attack surface is a massive risk!

What’s worse, a compromised signing key gives no indication to server owners. With no key rotation, a stolen key can be used to sign tokens with ANY level of access indefinitely. Attackers can use this key to quietly read or corrupt sensitive data without revealing their access. For any AIStore deployments that are not carefully gated in a protected environment, this could spell disaster.

With the 4.3 and subsequent 4.4 releases, AIStore AuthN now supports RSA signing keys and OIDC Issuer Discovery -- two essential features to mitigate the risk of this total security collapse.

Table of Contents

• RSA JWT Signing
• OIDC Issuer Discovery
  • Static Key Distribution
  • Trusted Issuers
  • OIDC in AuthN
  • Drawbacks and Limitations
• Complete Kubernetes Deployment
  • Running the Deployment
  • AuthN Config
  • AIS Config
• Conclusion and Future Work
  • Signing Key Rotation
  • Multi-replica Support
  • Service Account Authentication
• References

RSA JWT Signing

Previously, AIStore AuthN relied on HS256, which uses HMAC-SHA256 with a shared secret key. This is a symmetric algorithm, where the same secret is used for both signing JWTs and validating them.

This meant the signing key was distributed and could potentially exist in files, K8s secrets, K8s Pod specs, or environment variables in the actual AIS deployment.

We needed to be able to distribute a key publicly without exposing the ability to sign new tokens. That’s where asymmetric RSA signing key pairs come into the picture. With RSA, the private key never leaves the AuthN service. JWT signatures are validated only by a public key that cannot be used to sign new tokens.

AuthN also now supports encrypting the private key locally with a passphrase so it’s never unprotected on disk even within the service.

See RSA Signing in the AuthN docs for more details.

OIDC Issuer Discovery

Static Key Distribution

Even with the improved security of RSA keys, relying on static key distribution presents challenges.

First, this still doesn’t fully address the issue of compromised keys. Private key leaks are less likely, as they are never distributed, but we still risk silent exposure. Without key rotation, a compromised private key can be used to mint fraudulent JWTs indefinitely. And by using a static public key in AIS config, we can’t simply rotate the validation key in AIS without invalidating all existing tokens.

The static config also adds friction to deployment, since AuthN generates the key pair. Any AIS cluster deployment would need to inject the generated public key into its config.

Trusted Issuers

OIDC issuer lookup solves all of this by validating JWTs with a cached set of keys from trusted issuers. Instead of checking a JWT signature with a static public key, AIS uses the iss and kid claims from the JWT to look up the associated public key.

AIS itself has supported the concept of OIDC issuer discovery since version 4.1, but this was restricted to 3rd-party JWT issuers, which needed additional configuration to support the custom JWT format for AIS access.

This update brings that functionality to the native AIStore AuthN service, offering much better security and simplified deployment compared to the previous approach of symmetric, static signing keys.

OIDC in AuthN

AuthN does NOT fully implement the OIDC spec. It simply exposes the path /.well-known/openid-configuration, which responds with a “discovery document” containing jwks_uri. That jwks_uri path then returns the complete set of valid public JSON Web Keys (JWK). A JWK is a generic JSON container for different key types. In the case of AuthN, it represents an encoded RSA public key with some extra metadata.

This JWK set (JWKS) is then cached on the AIStore proxies, where the keys are used to validate JWT signatures.

Below is a diagram showing the full flow; see the AuthN docs for more implementation details.

Drawbacks and Limitations

One disadvantage is that previously, AIS had no dependency on the availability of the AuthN service. Now, AIS expects AuthN to be reachable for updating the local cache of key sets on a regular basis, increasing the requirement for AuthN reliability. Deploying in K8s simplifies this, but multi-replica support for AuthN requires ongoing work (see future work).

However, AIS will not need to query AuthN on every request, and in fact caches the key sets intelligently thanks to the JWX library.

Note: AIS currently only refreshes its cached key sets for a specific issuer on proxy restart. This is a known deficiency that limits the usability of key rotation and will be fixed in a future release. See the signing key rotation section below.

Complete Kubernetes Deployment

With RSA signing and OIDC discovery, the signing key is no longer shared, keys can be rotated without touching AIS config, and AIStore and AuthN can be deployed in any order without pre-distributing keys.

To demonstrate, we’ll show a local AIS cluster deployed in K8s alongside AuthN, runnable in K8s KinD via a single script.

See the full deployment scripts on the ais-k8s repo.

Running the Deployment

See the guide in ais-k8s for full details. First, you’ll need a few prerequisites:

• Docker or Podman
• Kubernetes in Docker
• kubectl
• Helm
• Helmfile

Next, to create the local deployment, clone ais-k8s and navigate to local.

Then run ./test-cluster.sh --auth.

That’s it! The script will bootstrap a local K8s cluster with all dependencies and an entire stack for AIS: K8s operator, AIS cluster, AIS AuthN, and an admin client deployment.

Once deployed, run the following to drop into a shell on the admin client pod inside the cluster:

kubectl exec -it -n ais deploy/ais-client -- /bin/bash

Initially, the AIS CLI won’t have access because AIS is enforcing authentication:

root@ais-client-7d869f99bf-dp76b:/# ais ls
Error: token required

This pod is pre-configured with environment variables for accessing the AuthN service. Run ais auth login $AIS_AUTHN_USERNAME -p $AIS_AUTHN_PASSWORD to fetch a token. Now the client in this pod has full admin access to the local cluster.

root@ais-client-7d869f99bf-dp76b:/# ais auth login $AIS_AUTHN_USERNAME -p $AIS_AUTHN_PASSWORD
Logged in (/root/.config/ais/cli/auth.token)
# Successful request
root@ais-client-7d869f99bf-dp76b:/# ais ls
No buckets in the cluster.

Below is a simplified diagram showing the entire setup:

AuthN Config

In recent versions of AuthN, RSA is the default signing method and will auto-generate a key pair on initial startup.

The relevant configuration for enabling OIDC lookup in the AuthN local helm environment is net.ExternalURL:

net:
  externalURL: "https://ais-authn.ais.svc.cluster.local:52001"

This tells the AuthN service what to use when building the jwks_uri in the openid-configuration response. The URL that clients can use to access the service depends on the deployment, so it must be configured in advance.

AIS Config

Because the AIS cluster runs in the same local deployment, we can use the K8s service DNS to access AuthN directly.

In the local-auth helm values for AIS, we set configToUpdate to update the AIS internal configuration to trust JWTs signed by the given allowed issuer.

The auth section configures how the operator and admin clients connect and provision an admin token by using credentials from a K8s secret.

# Configure AIS to trust JWTs issued by the local AuthN issuer
configToUpdate:
  auth:
    enabled: true
    # Instead of signature.key, we configure a list of issuers that we trust
    oidc:
      allowed_iss: ["https://ais-authn.ais.svc.cluster.local:52001"]

# Client AuthN API config used by operator and admin client
auth:
  serviceURL: "https://ais-authn.ais.svc.cluster.local:52001"
  # Currently, AuthN only supports username and password login to fetch tokens
  usernamePassword:
    secretName: ais-authn-su-creds

Conclusion and Future Work

Moving towards RSA signing and OIDC issuer lookup is important for AuthN, but there’s more we want to build:

Signing Key Rotation

Issuer lookup supports multiple active signing keys, which in theory allows for seamless rotation. Currently, AuthN keys can be rotated manually via ais auth rotate-key. However, AIStore version 4.4 won’t accept tokens signed by the new keys until the cached keyset is refreshed. This refresh is only triggered by the previous JWK’s expiry date (currently unset by AuthN) or by a proxy restart.

Once live rotation is fully supported on the AIStore side, automated signing key rotation for AuthN is a natural progression. Configurable intervals for automated rotation would eliminate the manual step and reduce the window of exposure if a key is compromised.

Multi-replica Support

Since AIS now actively queries AuthN for key sets, a single-replica AuthN becomes a potential availability bottleneck. Supporting multiple replicas would bring AuthN to production-grade availability.

This is a non-trivial development that requires more than a simple scale-up. AuthN currently uses BuntDB as its underlying storage. Support for distributed DBs will be required for multi-replica access. Signing keys must also be distributed and synchronized between instances to support consistency between multiple signers.

Service Account Authentication

A current limitation is that the AIS operator requires K8s secrets for admin credentials to manage AuthN-enabled clusters. One proposed alternative is to support AuthN token provisioning via a K8s projected service account token. This moves the access control used for the operator and admin client deployments to K8s RBAC and away from static credentials.

Follow our progress on the main AIStore repo or try out the local deployment yourself!

References

AIStore Authentication

• AuthN Documentation
• AIS Auth Validation
• AIS Auth CLI
• ais-k8s Local Deployment
• AuthN in K8s docs
• AuthN in K8s Helm

Standards and Specs

• OpenID Connect Core 1.0
• OpenID Connect Discovery 1.0
• HMAC (RFC 2104)
• JSON Web Token (RFC 7519)
• JSON Web Key (RFC 7517)
• JSON Web Algorithms (RFC 7518) -- RSA

Libraries and Tools

• AIStore JWK caching library: lestrrat-go/jwx
• Kubernetes in Docker (KinD)
• AuthN storage: BuntDB

General

• AIStore GitHub
• AIStore Blog

In our benchmarks, NBI delivers roughly 15x to 17x speedup for repeated listing of an s3:// bucket with 3.2 million objects, highlighting how effective a precomputed local snapshot can be for large datasets. In this post, we walk through the design of NBI, the internal create and list workflows, the benchmark results, and how to use it from the AIStore Python SDK and CLI.

Table of Contents

• Motivation
• Workflow
• Usage
• Benchmark
• Current Limitations
• When to Use NBI (and When Not To)
• Conclusion
• References

Motivation

Remote bucket listing becomes expensive when the bucket is both large and repeatedly accessed. A full listing requires AIS to retrieve and assemble a large volume of object metadata from the backend before it can return a complete result to the client. When that bucket is relatively stable and listed again and again, the system ends up redoing essentially the same work each time, even though the contents change very little between requests.

The core issue is that the object metadata returned by listing is often reusable, but the system keeps rebuilding it from scratch. The larger the bucket, the more bandwidth, latency, and backend API work AIS must spend to reconstruct information it has effectively already seen.

NBI addresses that mismatch by treating the bucket listing results as cacheable metadata. Instead of rebuilding the full listing on every request, AIS captures it once, stores it locally in a compact form, and reuses that snapshot for subsequent listings.

While our benchmarks use S3, NBI is backend-agnostic and works identically with any remote backend — AWS S3, Google Cloud Storage, Azure Blob, OCI Object Storage, and remote AIS clusters.

Workflow

NBI runs in two phases: creation and listing.

Creation

When the user requests inventory creation, the proxy distributes the job to all targets. Each target independently walks the remote backend, but only keeps the entries whose names hash to that target. The entries are sorted, grouped into chunks of ~20K names each, encoded as compressed msgpack, and written to the AIS system bucket ais://.sys-inventory. The resulting object path follows the pattern {provider}/@#/{bucket}/inv-{uuid}.

Listing

When a list request carries the inventory flag, the proxy broadcasts to all targets instead of sending the request to the remote backend. Each target reads its local inventory chunks, binary-searches for the continuation token, and returns a page of entries. The proxy merges per-target pages to assemble a globally sorted result. No S3 calls are made.

Usage

Python SDK

from aistore import Client

client = Client("http://ais-endpoint:51080")
bck = client.bucket("my-bucket", provider="s3")

# Create the inventory (one time)
job_id = bck.create_inventory(name="trainset-v1", prefix="images/")
client.job(job_id).wait()

# List via inventory — no S3 calls made
page = bck.list_objects(inventory_name="trainset-v1", prefix="images/train/")
for entry in page.entries:
    print(entry.name)

# Clean up when no longer needed
bck.destroy_inventory(name="trainset-v1")

CLI

# Create inventory
$ ais nbi create s3://my-bucket

# Monitor creation
$ ais show job create-inventory

# Show inventory metadata
$ ais nbi show s3://my-bucket

# List via inventory
$ ais ls s3://my-bucket --inventory
$ ais ls s3://my-bucket --inventory --prefix images/train/

# Destroy inventory
$ ais nbi rm s3://my-bucket

Benchmark

We measured listing latency across 15 scale points from 1K to 3.2M objects in an s3:// bucket, with 3 runs per point. The chart below shows p50 latency for AIS regular listing, S3 direct (boto3), NBI creation (one-time), and NBI listing.

==========================================================================================================================
NBI latency-vs-scale  (3 runs each)
==========================================================================================================================
Objects     Creation    Regular          NBI          S3 Direct    Speedup
                         p50      sd     p50     sd    p50      sd   (R/N)
--------------------------------------------------------------------------
   1K          449ms    405ms   144ms    21ms    2ms   676ms    2.9s  19.0x
   2K          616ms    596ms    21ms    34ms    4ms   480ms   251ms  17.6x
   5K          811ms     1.2s   161ms    51ms   33ms    1.4s   186ms  22.4x
  10K           1.8s     1.9s   177ms   147ms   35ms    2.7s   165ms  12.6x
  20K           3.5s     3.4s   106ms   248ms   20ms    6.2s   154ms  13.8x
  40K           7.2s     7.2s   358ms   550ms   95ms   10.8s   278ms  13.2x
  50K           8.6s     8.8s   689ms   631ms   30ms   12.9s   242ms  13.9x
  80K          13.9s    13.8s    1.1s    1.0s   22ms   21.4s   995ms  13.2x
 100K          17.0s    17.9s   387ms    1.4s   45ms   27.3s   822ms  12.6x
 200K          31.8s    37.6s    2.1s    2.9s  225ms   53.8s    6.6s  13.1x
 400K         1m  5s   1m 16s   364ms    6.3s  698ms  1m 50s    4.8s  12.3x
 600K         1m 43s    2m 0s    1.8s    9.1s  276ms  2m 39s    3.5s  13.3x
 800K         2m 10s   2m 46s    2.2s   12.5s  939ms  3m 43s   901ms  13.4x
   1M         2m 42s   3m 38s   920ms   16.4s  213ms  4m 42s   10.7s  13.3x
 3.2M         8m 34s  16m 55s    4.3s   1m 0s   2.8s 14m 46s   23.1s  16.9x
==========================================================================================================================

NBI listing latency still increases with object count because it scans locally stored inventory data, but its absolute latency remains far lower than regular listing. On the chart, the NBI curve appears almost flat compared to a regular AIS list or a direct S3 list. The speedup remains consistent across the entire 1K-3.2M range, reaching up to 16.9x at 3.2M objects.

Current Limitations

NBI is experimental in AIStore 4.3, and the current implementation keeps inventory management intentionally simple. At the moment, AIStore supports only one inventory per bucket, so concurrent inventories for the same bucket are not supported. Inventories are created manually via CLI or SDK and remain static until they are recreated or removed; if an inventory already exists and you want a new one, you can recreate it with --force, or simply remove it first with ais nbi rm and then create it again.

The current creation path is also optimized for correctness and simplicity rather than minimum backend work. During inventory creation, all targets walk the remote bucket in parallel and each keeps only its own portion of the results. Automatic refresh and more efficient creation strategies are planned for future releases.

Note: NBI replaces the older S3-specific --s3-inventory path, which depended on provider-generated CSV/Parquet inventory files. The new implementation is AIS-native, backend-agnostic, and does not require external tooling.

When to Use NBI (and When Not To)

Good fit:

• Large remote buckets (100K+ objects) that are listed repeatedly
• Training pipelines that enumerate a dataset before each epoch
• Data audits or dashboards that scan bucket contents periodically
• Any workflow where the bucket is relatively stable between listings

Not a good fit:

• Small buckets — creation cost exceeds the listing savings
• Rapidly changing buckets — the snapshot goes stale quickly, and frequent recreation negates the benefit
• ais:// buckets — metadata is already local (to each listing target), so NBI provides no speedup
• One-off listings — if you only list a bucket once, the creation overhead is pure cost

Conclusion

NBI delivers 15x better listing performance for large remote buckets, with measured speedups reaching 14-22x across the range we tested. That makes it a practical solution for repeated listing of multi-million-object s3://, gs://, and other remote buckets where rebuilding the full result from the backend on every request is too slow and too expensive.

References

• NBI documentation
• NBI benchmark script

Our benchmarks confirm the impact: fetching a 128 GiB object via parallel download is up to 9x faster than a standard single-stream GET request. When integrated with PyTorch DataLoader, parallel download reduces per-batch fetch latency by 11x compared to single-stream GET on a 10 TiB bucket.

This post describes parallel download’s design, internal workflow, and the trade-offs behind its performance improvements. It also summarizes the current benchmark results and shows how to use it from the AIStore Python SDK and PyTorch.

Table of Contents

• Motivation
• Architecture and Workflow
• Usage
• Benchmark
• Conclusion
• References

Motivation: Why Parallel Download Scales Better for Large Objects

The motivation for parallel download starts with a simple observation: once an object becomes large enough, reading it through one sequential GET leaves a lot of the cluster’s available bandwidth unused. Starting from AIStore 4.0, AIStore has supported chunked objects as a first-class storage representation: the cluster actively creates the chunks, places them across storage devices, and manages that layout internally. Once the object is stored that way, the natural next step is to build a read path that can exploit the layout instead of collapsing everything back into one serialized stream. That is the role of parallel download. It turns one logical object read into multiple coordinated chunk reads and, in practice, unlocks two distinct performance gains:

• Breaking the Single-Disk Limit: A single disk can only deliver so much read throughput, often well below the bandwidth of a modern data-center NIC. If a large object is fetched as one sequential stream, read throughput is effectively capped by the disk serving that stream. AIStore’s chunked object representation removes that bottleneck by distributing object chunks across the target’s available disks, allowing one logical object read to engage multiple disks in parallel.

• Taking Advantage of NVMe Parallelism: NVMe SSDs are built around deep queues and internal parallelism (NVMe Overview), so they perform best when multiple read requests are in flight at the same time. Parallel chunk reads give the device more work to schedule concurrently across its internal resources, which often raises effective read throughput well beyond what one long sequential request can sustain. This is exactly the behavior we will see later in the benchmark results.

Taken together, these two effects point to the same strategy: concurrent chunk fetching. The client needs to understand the object’s chunk boundaries and issue multiple range-read requests in parallel while preserving the correct chunk order at the destination. That is exactly what parallel download does. When the object is large enough, parallel download can improve single-object throughput both by engaging more of the cluster’s storage layout and by driving more of the underlying NVMe read parallelism.

Architecture and Workflow

Parallel download uses a coordinator-worker design, but it has two distinct execution patterns depending on whether the caller consumes the object as a stream or materializes the full object in memory.

1. Streaming Mode: Ring-Buffer Transfer

When the caller consumes the object incrementally, the parallel download API uses a bounded ring buffer to preserve ordered streaming semantics while multiple chunk fetches stay in flight.

At a high level, the workflow is:

1. The client issues a HEAD request to fetch the object’s metadata, including total object size and chunk size.
2. Parallel download allocates a shared buffer of size num_workers * chunk_size, giving each worker one slot in the ring, and spawns num_workers subprocesses.
3. Each subprocess worker issues range-read GET requests for its assigned chunk.
4. As chunks arrive, workers place them into their assigned buffer slots.
5. The main process consumes the slots in order, preserving the original byte order of the object as it copies data into the reader output stream.
6. Once a slot is fully consumed, the main process marks it reusable and signals the corresponding worker to fetch the next chunk.

This loop continues until the entire object has been streamed to the caller.

The ring-buffer design matters for two reasons:

• Bounded memory usage: the buffer stays fixed at num_workers * chunk_size no matter how large the object is.
• A full download pipeline: as soon as the consumer releases a slot, another range-read can begin, keeping the configured level of parallelism active until the final chunk is fetched.

2. Full-Object Mode: Direct Shared Memory

When the caller needs the full object materialized in memory, the parallel download API does not use the ring buffer. Instead, it allocates one shared-memory segment large enough to hold the full object and downloads directly into that destination.

At a high level, the workflow is:

1. The client issues a HEAD request to fetch the object’s metadata, including total object size and chunk size.
2. Parallel download allocates a shared-memory buffer to hold the full object.
3. Worker subprocesses issue parallel range-read GET requests for their assigned byte ranges.
4. Each worker writes directly into its exact offset inside the final shared-memory destination.
5. Once all workers finish, the caller receives a view over that full shared-memory segment.

This pattern avoids the extra copy from ring-buffer slots into a streaming output, but it trades that for a much larger memory reservation because the full object must fit in shared memory at once.

Usage

AIStore currently exposes parallel download through four interfaces: the Python SDK, PyTorch integration, the native Go API, and the CLI.

Python SDK

Use get_reader(num_workers=...) to enable parallel download for a single object read. The returned reader can be consumed as a streaming iterator:

from aistore import Client

client = Client("AIS_ENDPOINT")
bucket = client.bucket("my_bucket")

reader = bucket.object("large-object.bin").get_reader(num_workers=8)
for chunk in reader:
    # ...process the chunk

If your application needs the entire object materialized in memory, the same reader also supports read_all(). It returns a ParallelBuffer backed by shared memory. From there, you can either copy into a regular bytes object or access the underlying buffer directly and avoid the extra copy:

with bucket.object("large-object.bin").get_reader(num_workers=8).read_all() as buf:
    raw = buf.tobytes()  # option 1: copy into a new bytes object
    raw = buf.buf        # option 2: use the memoryview directly

Note: read_all() does not use the streaming ring buffer. It allocates a full-size shared-memory segment for the object and downloads the entire object into that buffer. On Linux, those shared-memory objects are normally created through POSIX shared memory and exposed via /dev/shm. As a result, very large objects can consume shared-memory capacity quickly and also contribute to overall memory pressure. If you use this path on Linux, monitor /dev/shm usage during testing, for example with df -h /dev/shm. Prefer the streaming iterator when the full object does not need to be materialized in memory at once.

Use Case: High-throughput reads for a single large object from an AIS cluster.

PyTorch Integration

AISParallelMapDataset plugs directly into the standard PyTorch DataLoader. Each __getitem__ call downloads one object using parallel range-reads and returns a ParallelBuffer:

from torch.utils.data import DataLoader
from aistore import Client
from aistore.pytorch import AISParallelMapDataset

bucket = Client("AIS_ENDPOINT").bucket("training-data")
dataset = AISParallelMapDataset(bucket, num_workers=8)

loader = DataLoader(dataset, batch_size=8, num_workers=2, collate_fn=lambda x: x)
for batch in loader:
    for name, buf in batch:
        tensor = torch.frombuffer(buf.buf, dtype=torch.float32)
        # ...train on tensor
        buf.close() # must be closed to avoid resource leak

Note: There are two different num_workers settings here, and they control different kinds of parallelism. AISParallelMapDataset(..., num_workers=N) controls the workers used inside each object download. DataLoader(..., num_workers=M) controls PyTorch subprocesses that prefetch samples across the batch pipeline. Setting both to high values multiplies total concurrency, which can oversubscribe CPU resources and make shared-memory buffer lifetime harder to manage. In practice, treat these as two knobs competing for the same client-side resources, not as independent speedups you can increase without limit.

Use Case: Loading large objects (video tensors, audio clips, high-resolution images) into a PyTorch training pipeline where per-sample download latency is the bottleneck.

Go API: Stream Mode

The Go streaming variant is api.MultipartDownloadStream(). It is the Go equivalent of the Python reader-based API: it returns an io.ReadCloser and performs concurrent range-reads behind the scenes while keeping only a bounded ring buffer in memory.

reader, attrs, err := api.MultipartDownloadStream(bp, bck, objName, &api.MpdStreamArgs{
    NumWorkers: 8,
    ChunkSize:  8 * cos.MiB,
})
if err != nil {
    return err
}
defer reader.Close()

_, err = io.Copy(dst, reader)

Use Case: The Go interface for the same reader-based parallel-download workflow.

CLI

The AIS CLI exposes parallel download through the --mpd option for large-object downloads. Under the hood, it uses the Go direct-write API api.MultipartDownload(), which writes each chunk directly into its final offset in the destination file.

Use Case: Downloading a large object directly into a local file or other seekable destination with minimal client-side buffering.

# Use `--mpd` option to download a single large object with parallel chunk fetching.
$ ais get ais://my-bucket/large-object.bin /tmp/large-object.bin --mpd --num-workers 8

Benchmark

The following measurements show how much performance parallel download can unlock in practice.

1. Single Large-Object Read

Results in this section were produced with the single-object benchmark script. We evaluated single large-object reads on two AIStore clusters. Both used the same overall configuration:

• Kubernetes Cluster: 3 bare-metal nodes, each hosting one AIS proxy (gateway) and one AIS target (storage server)
• Benchmark Client: 1 client machine
• Benchmark Object: one 128 GiB object
• Target CPU: 48 cores per node
• Target Memory: 995 GiB per node
• Client CPU: 48 cores
• Client Memory: 995 GiB
• Client Network Bandwidth: 100 Gbps

The two environments differed mainly in storage media and capacity:

NVMe-based Cluster: 16 × 5.8 TiB NVMe SSDs per Target

On the NVMe cluster, parallel download reached up to 9x speedup over a standard single-stream GET in the large-object benchmark. The chart includes both chunked and non-chunked cases: the monolithic label means the object was stored as a regular non-chunked object, while the other labels are AIS chunk sizes used to distribute the object across disks. Across that full sweep, once multiple read requests are in flight, throughput rises sharply across nearly all chunk sizes. The best results come from combining sufficiently large chunks with enough workers to keep the device busy, which is the NVMe parallelism discussed earlier.

HDD-based Cluster: 10 × 9.1 TiB Drives per Target

On the HDD cluster, parallel download still delivered up to 6.9x speedup, but the pattern is different. Here, the gain depends much more on the object being properly chunked across disks so that parallel download can read from multiple devices in parallel. Unlike NVMe, HDDs do not provide the same internal parallelism, so the improvement is more sensitive to chunk size and tapers off sooner for very large chunks.

Taken together, these two charts show that parallel download does not have a single best configuration that works everywhere. The optimal chunk size and worker count depend on your client-side resources, storage media, and object size distribution. For that reason, we encourage users to benchmark a small set of chunk-size and worker combinations on their own workload, find the sweet spot, and then use that setting for the full training or data-loading job. In our case, the best region was around 64-128 MiB chunks with 64 workers, and we will carry that tuning into the next benchmark.

2. Full Data-Loading Job via PyTorch

Results in this section were produced with the PyTorch data-loading benchmark script. To measure end-to-end impact, we ran that benchmark on the same NVMe-based cluster described above. The workload used a 10.61 TiB bucket containing 1,589 large training-sample objects ranging from 2.51 GiB to 17.32 GiB (average 6.84 GiB).

Based on the single-object benchmark, 64 MiB was the best chunk size on this cluster, so we rechunked the dataset before running the job:

$ ais bucket rechunk ais://mpd-bench --chunk-size 64MiB --objsize-limit 1

We then compared two end-to-end configurations over 64 batches with batch_size=8:

• GET: standard single-stream reads via AISMapDataset
• Parallel: per-object parallel downloads via AISParallelMapDataset with workers=48

As measured by the PyTorch data-loading benchmark script, the per-batch latency chart shows a clean separation between the two modes across the entire run. Standard GET stays in the 150-265 second range per batch, while the parallel mode stays near 14-23 seconds. The gap is not limited to a few outliers or warm-up effects; it persists across all 64 batches.

The same pattern is visible at the cluster level. During the benchmark run, total GET throughput stays near the single-stream baseline while the GET phase is running, then jumps sharply when the parallel phase begins:

$ AIS_ENDPOINT=<cluster-endpoint> AIS_BUCKET=mpd-bench BATCH_SIZE=8 NUM_BATCHES=64 AIS_WORKERS=48 python3 python/tests/perf/pytorch/parallel_download.py
Bucket: ais://mpd-bench
Objects: 1589  total=10865.5 GiB  avg=6.84 GiB  min=2.51 GiB  max=17.32 GiB
Config:  batch_size=8  num_batches=64  parallel_workers=48
...
                               GET    Parallel   Speedup
──────────────────────────────────────────────────────────
Throughput (GiB/s)            0.28        3.09     11.0x
Samples/sec                   0.04        0.46     11.0x
Total wall time (s)       12322.12     1117.81     11.0x
Batch latency mean (s)      192.53       17.47     11.0x
Batch latency med (s)       187.31       17.23     10.9x
Batch latency p95 (s)       231.40       20.74     11.2x
Time-to-first-batch (s)     161.36       15.79     10.2x

The same gap appears in the aggregate results. The parallel mode raises throughput from 0.28 GiB/s to 3.09 GiB/s, cuts mean batch latency from 192.53s to 17.47s, and reduces total wall time from 12,322s to 1,118s. Across the full benchmark, the improvement stays consistently around 10-11x.

Conclusion

Parallel download gives AIStore a parallel read path for large objects by turning one logical GET into multiple coordinated chunk fetches. In practice, that allows the client to take advantage of chunked object placement across disks and, on NVMe-based systems, to drive much more of the storage device’s internal read parallelism.

In our benchmarks, parallel download improved single-object throughput by up to 9x and reduced PyTorch per-batch latency by about 11x. Those gains carried through from synthetic single-object reads to a realistic end-to-end data-loading job, showing that parallel download can translate directly into shorter training input pipelines when large objects dominate the workload.

References

• AIStore 4.0 Release – Chunked Objects
• AIStore Python Object Reader
• AIStore PyTorch AISParallelMapDataset
• NVM Express: NVMe Overview

At NVIDIA, for example, SwiftStack (an S3-compatible object store) is widely used internally. But it is rarely present alone. Other S3-compatible systems appear more often than not: cloud accounts, regional replicas, compliance copies. It is a rule rather than the exception for several storage backends to quietly coexist in workloads run by any given team.

Hence, same-name datasets get copied, mutated, and passed across accounts, eventually finding their way back to us for concurrent use - e.g., s3://data in its many incarnations.

Same bucket name.
Different endpoints.
Different credentials.
Different contents.

Same Name, Many Buckets

In real deployments, what s3://data actually refers to often looks like this:

s3://data exists in:
├── SwiftStack (on-prem)
├── OCI (region ABC)
├── AWS S3 (us-east-1)
├── (and more)

From a human perspective, these buckets feel interchangeable. From a system’s perspective, they absolutely are not.

What’s in the Name

Traditional object storage APIs quietly assume that a bucket name uniquely identifies a dataset. That assumption breaks down the moment environments span multiple providers.

In AIS, a bucket is a triplet (see below) with properties:

          ┌────────── Bucket Identity ───────────┐
          │ ( provider, namespace, bucket name ) │
          └────────────────┬─────────────────────┘
                 ┌─────────┴─────────┐
                 │ bucket properties │
                 └───────────────────┘

Two buckets may share the same name and the same provider, yet belong to different namespaces - and therefore represent entirely different datasets. Credentials, policies, lifecycle rules, and contents remain isolated.

Bucket namespaces are not necessarily static (although they usually are). In AIS, namespace resolution itself can be a runtime decision that’d entail distributing updated bucket metadata - typically a split-second operation.

Dynamic Binding

Separately from namespace, AIS allows a logical bucket to be bound to another bucket as its backing data source.

Note: dynamic binding is not request forwarding or caching. It specifies where a dataset physically resides and how it is accessed remotely.

A logical bucket (e.g., ais://my-training-data) may source its contents from:

• an on-prem S3-compatible system,
• a public cloud bucket,
• a regional replica,
• or a derived dataset produced by a processing pipeline.

Consider two related datasets:

• Original: raw images, audio, or video with minimal labeling
• Processed: augmented, re-labeled, and reordered for efficient training

Both represent the same logical corpus. Training code references a single name: ais://my-training-data. At runtime, the platform decides which backing data to bind:

• training --> processed dataset
• validation --> raw dataset
• debugging --> local copy (or a subset thereof)
• compliance --> immutable regional mirror

          ┌─────────────────────────────────────────────┐
          │                Application                  │
          │            ais://my-training-data           │
          └───────────────────────┬─────────────────────┘
                                  │
          ┌───────────────────────┴─────────────────────┐
          │               Bucket Identity               │
          │   (provider + namespace + bucket name)      │
          └───────────────────────┬─────────────────────┘
                                  │
          ┌───────────────────────┴─────────────────────┐
          │               Backend Binding               │
          │                 (at runtime)                │
          └───────────────────────┬─────────────────────┘
                                  - (current binding)
          ┌──────────────┬────────┴─────┬───────────────┐
          │  SwiftStack  │    AWS S3    │     OCI       │
          │   s3://data  │   s3://data  │  s3://data    │
          └──────────────┴──────────────┴───────────────┘

Recap

Bucket names are not identities.
Dataset selection is a configuration and/or runtime decision, not an application concern.
Infrastructure must absorb the complexity.

References

• AIStore: scalable storage for AI applications
• Bucket Properties
• Backend Providers

PS. I’ve changed SwiftStack, OCI and AWS specifics in this post; the underlying problem and the solution - are real.

Our benchmarks confirm the impact: fetching a 4GiB remote object via blob downloader is now 4x faster than a standard cold-GET. When integrated with the prefetch job, this approach delivers a 2.28x performance gain compared to monolithic fetch operations on a 1.56TiB S3 bucket.

This post describes the blob downloader’s design, internal workflow, and the optimizations that drive its performance improvements. It also outlines the benchmark setup, compares blob downloader against regular monolithic cold GETs, and shows how to use the blob downloader API from the supported clients.

Table of Contents

• Motivation
• Architecture and Workflow
• Usage
• Benchmark
• Conclusion
• References

Motivation: Why Blob Downloader Scales Better for Large Object?

Splitting large objects into smaller, manageable chunks for parallel downloading is a proven strategy to increase throughput and resilience. In fact, cloud providers like AWS and GCP explicitly recommend concurrent range-read requests for optimal performance. The core advantages include:

• Isolating Failures and Reducing Retries: With a single sequential stream, a network hiccup can force a restart or large rollback. With range-reads, failures are isolated to individual chunks, so only the affected chunk needs to be retried.

• Leveraging Distributed Server Throughput: Cloud objects are typically spread across many disks and nodes. Concurrent range-reads allow the client to pull data from multiple storage nodes in parallel. This aligns with the provider’s internal architecture and bypasses the single-node or per-disk I/O limits.

Beyond these standard benefits, AIStore leverages the concurrent range-read pattern to unlock an architectural advantage: chunked object representation. Introduced in AIStore 4.0, this capability allows objects to be stored as separate chunk files, which are automatically distributed across all available disks on a target. This enables the blob downloader to stream each range-read payload directly to a local chunk file, achieving zero-copy efficiency and aggregating the full write bandwidth of all underlying disks.

Architecture and Workflow

The blob downloader uses a coordinator-worker pattern to execute the download process. When a request is initiated, the main coordinator thread fetch the remote object’s metadata to determine its total size and logically segments it into smaller chunks.

This is the same general pattern often referred to as a worker pool, a work-queue with a pool of workers, or a producer–consumer pipeline.

Once the segmentation is complete, the coordinator initializes a pool of worker threads and begins dispatching work. It assigns specific byte ranges to available workers, who then independently issue concurrent “Range Read” requests to the remote storage backend.

As workers receive data, they write each chunk directly to separate local files and report back to the coordinator to receive its next assignment. This continuous loop proceeds until every segment of the object has been successfully persisted.

Load-Aware Runtime Adaptation

Blob downloader is wired into AIStore’s load system, which continuously grades node pressure (memory, CPU, goroutines, disk) and returns throttling advice.

At a high level, blob downloader:

• checks load once before starting a job and may reject or briefly delay it when the node is already under heavy memory pressure,
• derives a safe chunk size from current memory conditions instead of blindly honoring the user’s request, and
• lets workers occasionally back off (sleep) when disks are too busy while downloads are in progress.

The result is that blob downloads run at full speed when the cluster has headroom, but automatically slow down instead of pushing the node into memory or disk overload.

Usage

AIStore exposes blob download functionality through three distinct interfaces, each suited to different use cases.

1. Single Object Blob Download Job

Start a blob download job for one or more specific objects.

Use Case: Direct control over blob downloads, monitoring individual jobs.

AIS CLI Example:

# Download single large object
$ ais blob-download s3://my-bucket/large-model.bin --chunk-size 4MiB --num-workers 8 --progress
blob-download[X-def456]: downloading s3://my-bucket/large-model.bin
Progress: [████████████████████] 100% | 50.00 GiB/50.00 GiB | 2m30s

# Download multiple objects
$ ais blob-download s3://my-bucket --list "obj1.tar,obj2.bin,obj3.dat" --num-workers 4

2. Prefetch + Blob Downloader

The prefetch operation is integrated with blob downloader via a configurable blob-threshold parameter. When this threshold is set (by default, it is disabled), prefetch routes objects whose size meets or exceeds the value to an internal blob-download job, while smaller objects continue to use standard cold GET.

Use Case: Batch prefetching of remote buckets where some objects are very large, letting the job automatically decide when to engage blob downloader behind the scenes.

AIS CLI Example:

# List remote bucket
$ ais ls s3://my-bucket
NAME             SIZE            CACHED
model.ckpt       12.50GiB        no
dataset.tar      8.30GiB         no
config.json      4.20KiB         no

# Prefetch with 1 GiB threshold:
# - objects ≥ threshold use blob downloader (parallel chunks)
# - objects < threshold use standard cold GET
$ ais prefetch s3://my-bucket --blob-threshold 1GiB --blob-chunk-size 8MiB
prefetch-objects[E-abc123]: prefetch entire bucket s3://my-bucket

3. Streaming GET

The blob downloader splits the object into chunks, downloads them concurrently into the cluster, and simultaneously streams the assembled result to the client as it arrives.

Use Case: Stream a large object directly to the client while simultaneously caching it in the cluster.

Python SDK Example:

from aistore import Client
from aistore.sdk.blob_download_config import BlobDownloadConfig

# Set up AIS client and bucket
client = Client("AIS_ENDPOINT")
bucket = client.bucket(name="my_bucket", provider="aws")

# Configure blob downloader (4MiB chunks, 16 workers)
blob_config = BlobDownloadConfig(chunk_size="4MiB", num_workers="16")

# Stream large object using blob downloader settings
reader = bucket.object("my_large_object").get_reader(blob_download_config=blob_config)
print(reader.readall())

Benchmark

The benchmark was run on an AIStore cluster using the following system configuration:

• Kubernetes Cluster: 3 bare-metal nodes, each hosting one AIS proxy (gateway) and one AIS target (storage server)
• Storage: 16 × 5.8 TiB NVMe SSDs per target
• CPU: 48 cores per node
• Memory: 995 GiB per node
• Network: dual 100 GbE (100000 Mb/s) NICs per node

1. Single Blob Download Request

The chart above compares the time to fetch a single remote object using blob download versus a standard cold GET across a range of object sizes (16 MiB to 8 GiB).

For smaller objects, cold GET performs slightly better due to the coordination overhead inherent in blob download. However, once objects exceed 256 MiB, blob download begins to show significant advantages. The speedup grows significantly with object size.

These results validate the architectural benefits discussed earlier: concurrent range-read requests combined with distributed chunk writes deliver substantial gains for large objects.

2. Prefetch with Blob Download Threshold

In the prefetch benchmark, we created an S3 bucket containing 4,443 remote objects, ranging from 10.68 MiB to 3.53 GiB in size, for a total remote footprint of 1.56 TiB.

$ ais bucket summary s3://ais-tonyche/blob-bench
NAME                     OBJECTS (cached, remote)        OBJECT SIZES (min, avg, max)            TOTAL OBJECT SIZE (cached, remote)
s3://ais-tonyche         0    4443                       10.68MiB   305.77MiB  3.53GiB           0         1.56TiB

The chart above compares different --blob-threshold values for this mixed-size workload and reports both total prefetch duration and aggregate disk write throughput. In our environment, a threshold around 256 MiB strikes the best balance by routing large objects through blob download while letting smaller objects use regular cold GET.

• If the threshold is set too high: blob downloader is underutilized because more parallelizable large objects fall back to monolithic GETs.
• If the threshold is set too low: blob downloader is overused on small objects, flooding the system with chunked downloads and adding coordination overhead without improving throughput.

Across all thresholds, the key pattern is that assigning a reasonable share of large objects to blob downloader raises aggregate disk write throughput, which in turn shortens total prefetch time. When the threshold is tuned so that genuinely large objects are handled via blob download, the cluster is able to drive the highest parallel writes across targets. In our setup, a threshold of about 256 MiB achieved this balance, delivering a 2.28× shorter prefetch duration than a pure monolithic cold GET of the same bucket.

Conclusion

The key takeaway is simple: on real workloads with multi‑GiB objects, blob downloader reduces time to fetch large remote objects by up to 4× in our benchmarks. It achieves this by driving much higher aggregate disk throughput than a single cold GET can sustain.

Benchmarks also show that performance is highly sensitive to the --blob-threshold setting: in our 1.56 TiB S3 bucket, a threshold around 256 MiB maximized disk write throughput during the prefetch job. The ideal value in your deployment will depend on cluster configuration, network conditions, backend provider, and object size distribution, but there will almost always be a sweet spot where blob downloader is neither underutilized nor overused.

In practice, the guidance is simple: use a small benchmark to pick a reasonable threshold for your environment, and let blob downloader plus load advice handle the rest. Today, that choice is exposed as the --blob-threshold knob on prefetch jobs, while the load system ensures that even an aggressive setting won’t push targets into memory or disk overload. Longer term, the goal is to make this decision mostly internal — using observed object sizes and node load to engage blob downloader automatically — so most users can rely on sane defaults and only reach for explicit tuning when they really need it.

References

• AWS S3 performance guidelines – byte-range / parallel downloads
• GCP Cloud Storage – improving throughput with client libraries
• HTTP Range Requests (RFC 7233)
• AIStore 4.0 release – chunked objects
• AIStore Blob Downloader documentation

The API returns a single ordered archive - TAR by default - containing the requested objects and/or sharded files.

A given GetBatch may specify any number of items and span any number of buckets.

From the caller’s perspective, each request behaves like a regular synchronous GET, but you can read multiple batches in parallel.

Inputs may mix plain objects with any of the four supported shard formats (.tar, .tgz/.tar.gz, .tar.lz4, .zip), and outputs can use the same formats (default: TAR).

Ordering is strict: ask for data items named A, B, C - and the resulting batch will contain A, then B, then C.

Items A, B, C, etc. can reference plain objects or sharded files, stored locally or in remote cloud buckets.

Two delivery modes are available. The streaming path starts sending as the resulting payload is assembled. The multipart path returns two parts: a small JSON header (apc.MossOut) with per-item status and sizes, followed by the archive payload.

Get-Batch provides the largest gains for small-to-medium object sizes, where it effectively amortizes TCP and connection-setup overheads across multiple requests. For larger objects, overall performance improvement tapers off because the data transfer time dominates total latency, making the per-request network overhead negligible in comparison.

Fig. 1. Up to 25x single-worker speed-up in early benchmarks.

The graph plots speed-up factor (Y-axis) against object size (X-axis), showing how batch size (100, 1K, 10K objects per batch) and object size affect performance. Each test used 10k objects on a 3-node AIStore cluster (48 CPUs, 187 GiB RAM, 10×9.1 TiB disks per node). The gains come from reducing per-request TCP overhead and parallelizing object fetches.

PS. Cluster-wide multi-worker benchmarks are in progress and will be shared soon.

Maintaining accurate and up-to-date HTTP API documentation is critical for the developer experience when building and debugging SDKs. Clear HTTP documentation saves developers from digging through AIStore source code to understand expected endpoints, actions, query parameters, and request formats—whether implementing new features or troubleshooting issues in the SDK. With REST API endpoints spanning object management, cluster operations, ETL workflows, and administrative functions, manually maintaining this documentation quickly becomes a bottleneck that leads to inconsistencies and outdated information.

This is where GenDocs comes in—a powerful tool that automatically generates comprehensive OpenAPI/Swagger documentation directly from AIStore’s Go source code using descriptive annotation-based parsing.

GenDocs streamlines AIStore’s documentation workflow, eliminates manual maintenance overhead, and ensures that API documentation stays perfectly synchronized with the codebase as it evolves.

The Challenge: Scale and Consistency

AIStore’s REST API surface is extensive, covering everything from basic object operations to complex multi-cloud data management and ETL transformations. Each endpoint requires documentation that includes:

• HTTP methods and paths with parameter definitions
• Request/response schemas and examples
• Action-based operations with multiple model variants
• Interactive code samples and curl commands
• Proper categorization and cross-references

Maintaining this manually across a rapidly evolving codebase presents several challenges:

• Synchronization Drift: Documentation inevitably falls behind code changes
• Human Error: Manual updates are prone to inconsistencies and omissions
• Developer Overhead: Engineers spend valuable time on documentation maintenance
• Scalability: As the API grows, manual processes become increasingly unsustainable

The GenDocs Solution

GenDocs solves these problems through annotation-driven documentation generation. Instead of maintaining separate documentation files, developers add lightweight annotations directly in the Go source code alongside their API handlers. GenDocs then parses these annotations to automatically generate comprehensive OpenAPI specifications which are rendered into a formatted website that developers can easily reference.

Core Design Principles

1. Developer-Friendly: Minimal annotation syntax that doesn’t clutter code
2. Source of Truth: Documentation lives alongside implementation code
3. Automatic Generation: Zero manual steps to update documentation
4. Universal format: Generates standard OpenAPI spec (YAML/JSON)

Annotation Syntax

GenDocs uses a simple but powerful annotation format. Here’s how developers document an API endpoint:

// +gen:endpoint GET /v1/buckets/{bucket-name}/objects/{object-name} [provider=string]
// Retrieves an object from the specified bucket.
// Supports streaming for large objects and conditional requests.
func GetObject(w http.ResponseWriter, r *http.Request) {
    // implementation...
}

This single annotation automatically generates:

• OpenAPI endpoint definition
• Parameter documentation
• HTTP examples with proper curl commands

Advanced Features

Action-Based Endpoints

Many AIStore endpoints support multiple operations through action parameters. In AIStore, an “action” is a JSON message in the request body that at minimum includes an {"action":"..."} string; some actions also carry a structured value field. The action constants (e.g. apc.ActCopyBck) map to the action string used in the body, and the associated model defines the value.

// +gen:endpoint PUT /v1/buckets/{bucket-name} action=[apc.ActCopyBck=apc.TCBMsg|apc.ActETLBck=apc.TCBMsg]
// +gen:payload apc.ActCopyBck={"action": "copy-bck", "value": {"dry_run": false}}
// +gen:payload apc.ActETLBck={"action": "etl-bck", "value": {"id": "ETL_NAME"}}
// Administrative bucket operations including copy and ETL transformations.
func BucketHandler(w http.ResponseWriter, r *http.Request) {
    // implementation...
}

This generates comprehensive documentation showing:

• All supported actions and their models
• Complete JSON payload examples

Automatic Model Discovery

GenDocs automatically discovers Go structs marked with // swagger:model and incorporates them into the API documentation:

// swagger:model
type Transform struct {
    Name     string       `json:"id,omitempty"`
    Pipeline []string     `json:"pipeline,omitempty"`
    Timeout  cos.Duration `json:"request_timeout,omitempty" swaggertype:"primitive,integer"`
}

Note: swaggertype is only needed by swagger when mapping custom Go types (e.g., cos.Duration) to primitive types (e.g. integer) in the generated OpenAPI spec. Primitive fields like string, int, bool, etc. do not require it.

Intelligent Payload Generation

For simple actions that only require an action name, GenDocs automatically generates basic payloads, reducing annotation overhead:

// +gen:endpoint PUT /v1/cluster action=[apc.ActResetConfig=apc.ActMsg|apc.ActRotateLogs=apc.ActMsg]
// These simple actions auto-generate: {"action": "reset-config"} and {"action": "rotate-logs"}

Integration with AIStore’s Workflow

GenDocs is seamlessly integrated into AIStore’s development workflow and CI pipeline:

Documentation Website Deployment Workflow

1. Code Changes: Developers add/modify API endpoints with annotations
2. Local Testing: make api-docs-website generates documentation locally
3. CI Pipeline: GitHub Actions automatically regenerates docs on merge
4. Website Deployment: Updated documentation is deployed to the AIStore website

Build Process

Figure: GenDocs multi-phase pipeline transforming source code annotations into comprehensive API documentation

The documentation generation process is a multi-stage pipeline that transforms source code annotations into an OpenAPI specification and markdown.

(1) The process begins when GenDocs scans the entire AIStore codebase, discovering every +gen:endpoint annotation and building a complete inventory of API endpoints, parameters, and data models. (2) During this discovery phase, the tool also collects +gen:payload definitions and (3) action mappings that will be used to generate realistic examples.

(4) Once the scanning is complete, GenDocs transforms these annotations into standard Swagger comments that can be processed by the OpenAPI toolchain. This transformation includes generating operation IDs, parameter documentation, and request/response schemas for each endpoint.

(5) The OpenAPI specification is then generated using the Swagger tooling, producing both YAML and JSON formats that contain the complete API definition. However, the standard OpenAPI specification lacks some of the rich metadata that makes AIStore’s documentation particularly useful.

(6) This is where GenDocs’ vendor extension system comes into play. The tool injects AIStore-specific extensions into the OpenAPI specification, including action-to-model mappings and complete HTTP examples with curl commands. These extensions are what enable the interactive features and comprehensive examples in the final documentation.

(7) The final step involves converting the enhanced OpenAPI specification into markdown format using the OpenAPI Generator CLI with custom templates. (8) This produces the website-ready documentation that is integrated into AIStore’s Jekyll-based documentation site.

In practice, the CI pipeline runs this workflow automatically—developers only need to provide the GenDocs annotation syntax.

User Experience Enhancements

The auto-generated documentation provides users with:

# Working curl examples for every endpoint
curl -i -L -X PUT \
  -H 'Content-Type: application/json' \
  -d '{"action": "copy-bck", "value": {"dry_run": false}}' \
  'AIS_ENDPOINT/v1/buckets/source-bucket'

Technical Architecture and Annotations

Parsing Engine

Maintaining accurate API documentation is difficult when complex model structs are spread across the codebase. Manually discovering these structs and keeping cross-references between endpoints, actions, and data models in sync is time-consuming and error-prone.

To solve this, the Abstract Syntax Tree (AST) parsing approach is used to analyze the codebase directly. It automatically discovers model structs, builds a complete inventory of API models and their relationships, and maintains precise links between +gen:endpoint annotations and their corresponding handler functions.

A second challenge is flexibility: developers often want to place annotations close to the logic they describe, even if that means spreading them across multiple files. For example, a payload definition might live near a helper function rather than in the main endpoint file.

By using a file walker that recursively scans the codebase, GenDocs is collecting every +gen:payload annotation. It then parses endpoints file-by-file, ensuring that payload definitions are correctly applied to their endpoints regardless of where they are declared.

To further reduce drift, we started with go generate as the primary goal was to keep documentation annotations in line with the code. Annotations live next to handlers and regeneration runs with builds, so the docs track the exact code state—no separate “docs repo,” less drift, and less context‑switching for developers.

To prevent annotations from becoming too verbose, we auto‑generate simple { "action":"..." } payloads where possible. When an action takes a structured value or a name, we add a +gen:payload. S3‑compatible endpoints are the exception—they expect XML. For those, we point to an XML body via payload=, and the generator switches the Content‑Type automatically.

On the spec side, Swaggo scans Go code and inline annotations and emits an OpenAPI document that feeds straight into the website pipeline. For custom wrappers (for example, cos.Duration), the swaggertype tag tells the generator how the field should appear in the spec, keeping models faithful to the API’s serialization.

Descriptive Comments

Right after a +gen:endpoint line, GenDocs reads the plain comment lines and encapsulates them into the endpoint’s summary. Those few sentences become the description on the website detailing what it does, why a developer would call it, and any guardrails (auth, permissions, size limits).

Separately, model struct fields can include Go comments alongside their JSON which become per‑field descriptions in the generated schema (e.g., allowed values, units, defaults). Keeping these comments close to the code ensures the final API docs reflect the intended behavior and field semantics without manual editing. In addition, the vendor extension framework enables injection of AIStore-specific metadata while maintaining full OpenAPI specification compliance.

Case Study: Isolating a client issue with a direct API call

A bucket deletion operation failed when invoked via CLI. To determine whether the issue was in the client or the AIStore cluster, the operation was executed directly using the documented HTTP example generated by GenDocs. The direct API call succeeded, confirming server behavior was correct and narrowing the problem to the CLI implementation. This illustrates how canonical HTTP examples enable developers to easily isolate client‑versus‑server issues, reduce time to root cause, and focus fixes on the right component.

curl -i -L -X DELETE \
  -H 'Content-Type: application/json' \
  -d '{"action":"destroy-bck"}' \
  'AIS_ENDPOINT/v1/buckets/BUCKET_NAME'

Conclusion

GenDocs is a shift in AIStore’s approach to API documentation—from manual maintenance to automated generation that scales with the codebase. By embedding documentation directly in source code through lightweight annotations, the tool eliminates synchronization issues while significantly improving documentation quality and the developer experience.

In practice, this yielded measurable benefits: comprehensive endpoint coverage, immediate updates with code changes, and consistent formatting across the API surface. Developers can focus on feature development rather than documentation maintenance, while users receive accurate, up‑to‑date documentation with working examples.

References

• GenDocs Tool Documentation
• AIStore HTTP API Documentation
• OpenAPI Specification
• AIStore Repository
• Swagger UI Documentation

Machine learning teams increasingly rely on large datasets from HuggingFace to power their models. But traditional download tools struggle with terabyte-scale datasets containing thousands of files, creating bottlenecks that slow development cycles.

This post introduces AIStore’s new HuggingFace download integration, which enables efficient downloads of large datasets with parallel batch jobs.

Table of contents

1. Background
2. CLI Integration: Simplified Workflows
3. Download Optimizations
4. Complete Walkthrough: NonverbalTTS Dataset
5. Next Steps
6. Conclusion

Background

Sequential downloads create significant bottlenecks when dealing with complex datasets that have hundreds of thousands of files distributed across multiple directories.

AIStore addresses this by parallelizing downloads within each target using multiple workers (one per mountpath), batching jobs based on file size, and collecting file metadata in parallel. This approach leverages the network throughput from each individual target to the HuggingFace servers.

CLI Integration: Simplified Workflows

Prerequisites

The following examples assume an active AIStore cluster. If the destination buckets (e.g., ais://datasets, ais://models) don’t exist, they will be created automatically with default properties.

AIStore’s CLI includes HuggingFace-specific flags for the ais download command that handle distributed operations behind the scenes.

Basic Download Commands

# Download entire dataset
$ ais download --hf-dataset squad ais://datasets/squad/

# Download entire model  
$ ais download --hf-model bert-base-uncased ais://models/bert/

# Download specific file
$ ais download --hf-dataset squad --hf-file train/0.parquet ais://datasets/squad/

Authentication and Configuration

# Export your HuggingFace token and use for private/gated content
$ export HF_TOKEN=your_hf_token_here
$ ais download --hf-dataset private-dataset --hf-auth $HF_TOKEN ais://private-data/

# Control batching with blob threshold
$ ais download --hf-dataset large-dataset --blob-threshold 200MB ais://datasets/large/

Progress Monitoring

# Real-time progress tracking
$ ais show job --refresh 2s

# Detailed job information
$ ais show job download --verbose

Download Optimizations

The system uses some key techniques to improve download performance:

Job Batching: Size-Based Distribution

Job batching categorizes files based on configurable size thresholds:

# Configure blob threshold for job batching
$ ais download --hf-dataset squad --blob-threshold 100MB ais://ml-datasets/

Files are categorized into two groups:

• Large files (above blob threshold): Get individual download jobs for maximum parallelism
• Small files (below threshold): Batched together to reduce overhead

Figure: How AIStore batches files based on size threshold (100MB in this example)

Concurrent Metadata Collection

Before downloading files, AIStore makes parallel HEAD requests to the HuggingFace API to collect file metadata (like file sizes) concurrently rather than sequentially. This reduces setup time for datasets with many files.

Complete Walkthrough: NonverbalTTS Dataset

Let’s walk through an example downloading a machine learning dataset and processing it with ETL operations:

Walkthrough Prerequisites

For this walkthrough, we’ll create and use three buckets:

• ais://deepvs - for the initial dataset download
• ais://ml-dataset - for ETL-processed files
• ais://ml-dataset-parsed - for the final parsed dataset

If these buckets don’t exist, they will be created automatically with default properties.

Step 1: Download Dataset with Configurable Job Batching

# Download deepvk/NonverbalTTS dataset with job batching
$ ais download --hf-dataset deepvk/NonverbalTTS ais://deepvs --blob-threshold 500MB --max-conns 5
Found 11 parquet files in dataset 'deepvk/NonverbalTTS'
Created 7 individual jobs for files >= 500MiB
Started download job dnl-B-oOHruKH9
To monitor the progress, run 'ais show job dnl-B-oOHruKH9 --progress'

Step 2: Monitor Distributed Job Execution

# Watch configurable job distribution across cluster targets
$ ais show job
download jobs  
JOB ID           XACTION         STATUS          ERRORS  DESCRIPTION
dnl-B-oOHruKH9   D6JOGa7PH9      1 pending       0       multi-download -> ais://deepvs
dnl-zoOHr7PG3    D6JOGa7PH9      1 pending       0       https://huggingface.co/api/datasets/deepvk/NonverbalTTS/parquet/default/other/0.parquet -> ais://deepvs/0.parquet
dnl-oJOHruKG3    D6JOGa7PH9      1 pending       0       https://huggingface.co/api/datasets/deepvk/NonverbalTTS/parquet/default/train/1.parquet -> ais://deepvs/1.parquet
dnl-F_ogHauKH9   D6JOGa7PH9      1 pending       0       https://huggingface.co/api/datasets/deepvk/NonverbalTTS/parquet/default/train/2.parquet -> ais://deepvs/2.parquet
dnl-PoOHr7KG9    D6JOGa7PH9      1 pending       0       https://huggingface.co/api/datasets/deepvk/NonverbalTTS/parquet/default/train/3.parquet -> ais://deepvs/3.parquet
....

Step 3: Verify Download Completion

# Check bucket summary after download
$ ais ls ais://deepvs --summary
NAME             PRESENT         OBJECTS         SIZE (apparent, objects, remote)        USAGE(%)
ais://deepvs     yes             6 0             2.76GiB 2.76GiB 0B                      0%

Options for Using Downloaded Data

At this point, you have several options:

1. Use directly: Work with the downloaded files as-is if they meet your requirements
2. Transform with ETL: Apply preprocessing for format conversion, file organization, or data standardization
3. Custom processing: Use your own tools for data preparation

Why transform? HuggingFace datasets often have complex paths or formats that benefit from standardization. This walkthrough demonstrates ETL transformations for file organization (consistent naming) and format conversion (Parquet → JSON for framework compatibility).

Step 4: Initialize ETL Transformers

Note: ETL operations require AIStore to be deployed on Kubernetes. See ETL documentation for deployment requirements and setup instructions.

Before applying transformations, initialize the required ETL containers:

# Initialize batch-rename ETL transformer for file organization
$ ais etl init -f https://raw.githubusercontent.com/NVIDIA/ais-etl/main/transformers/batch_rename/etl_spec.yaml

# Initialize parquet-parser ETL transformer for data parsing
$ ais etl init -f https://raw.githubusercontent.com/NVIDIA/ais-etl/main/transformers/parquet-parser/etl_spec.yaml

# Verify ETL transformers are running
$ ais etl show

Step 5: Preprocessing using ETL

# Organize and rename files using batch rename ETL
$ ais etl bucket batch-rename-etl ais://deepvs ais://ml-dataset
etl-bucket[BatchRename] ais://deepvs => ais://ml-dataset

# Verify renamed files with structured naming
$ ais ls ais://ml-dataset/
NAME                        SIZE            
train_0.parquet             485MiB          
train_1.parquet             492MiB          
train_2.parquet             511MiB          
...

# Convert parquet files to JSON format for easier ML framework integration
$ ais etl bucket parquet-parser-etl ais://ml-dataset ais://ml-dataset-parsed
etl-bucket[xO_sVT3Im] ais://ml-dataset => ais://ml-dataset-parsed

# Verify processed dataset ready for ML training
$ ais ls ais://ml-dataset-parsed --summary
NAME                         PRESENT         OBJECTS         SIZE (apparent, objects, remote)        USAGE(%)
ais://ml-dataset-parsed      yes             7 0             8.68GiB 8.68GiB 0B                      1%

Step 6: ML Pipeline Integration

AIStore integrates seamlessly with popular ML frameworks. Here’s how to use the processed dataset in your training pipeline:

Option A: Direct SDK Usage (Simple)

from aistore.sdk import Client
import json

client = Client("http://localhost:51080")
bucket = client.bucket("ml-dataset-parsed")

# Load processed training data
for obj in bucket.list_objects():
    if obj.name.startswith("train_"):
        data = json.loads(obj.get_reader().read_all())
        # Process individual training samples
        for sample in data:
            # Your training logic here
            pass

Option B: PyTorch Integration (Recommended for ML Training)

from aistore.sdk import Client
from aistore.pytorch import AISIterDataset
from torch.utils.data import DataLoader
import json

# Create dataset that reads directly from the cluster
client = Client("http://localhost:51080")
dataset = AISIterDataset(ais_source_list=client.bucket("ml-dataset-parsed"))

# Configure DataLoader with multiprocessing
loader = DataLoader(
    dataset,
    batch_size=32,
    num_workers=4,  # Parallel data loading across multiple cores
)

# Training loop
for batch_names, batch_data in loader:
    # Parse JSON data
    parsed_samples = [json.loads(data) for data in batch_data]
    
    # Convert to tensors and train your model
    # model.train_step(parsed_samples)
    pass

Next Steps

The HuggingFace integration opens up some practical areas for expansion:

Download and Transform API: AIStore supports combining download and ETL transformation in a single API call, eliminating the two-step process shown in the walkthrough. This allows downloading HuggingFace datasets with immediate transformation (e.g., Parquet → JSON) in one operation. CLI integration for this functionality is in development.

Additional Dataset Formats: Beyond the current Parquet support, HuggingFace datasets are available in multiple formats that teams commonly need:

• JSON format - Direct JSON downloads for frameworks requiring this format
• CSV format - For traditional data processing workflows
• WebDataset format - For large-scale ML pipelines using WebDataset

Conclusion

AIStore’s HuggingFace integration addresses common dataset download bottlenecks in machine learning workflows. Job batching and concurrent metadata collection enable efficient, parallel downloads of terabyte-scale datasets that would otherwise overwhelm traditional tools. Once stored in AIStore, teams can leverage local ETL operations to transform and prepare data without additional network transfers. This approach provides a streamlined path from raw downloads to training-ready datasets, eliminating the typical download-wait-process cycle that slows ML development.

References:

AIStore Core Documentation

• AIStore GitHub
• AIStore Blog
• AIStore Downloader Documentation
• AIStore Python SDK
• AIStore PyTorch Integration - High-performance data loading for ML training

ETL (Extract, Transform, Load) Resources

• ETL Documentation - Comprehensive guide to AIStore ETL capabilities and Kubernetes deployment
• ETL CLI Reference - Command-line interface for ETL operations
• Batch-Rename Transformer - File organization and renaming
• Parquet Parser Transformer - Parquet to JSON conversion
• AIStore Kubernetes Deployment - Production Kubernetes deployment tools and documentation

External Resources

• HuggingFace Documentation
• HuggingFace Datasets API Reference
• Apache Parquet Format Specification