Logical sharding layer for blob storage

The problem

A cloud storage account is not an infinite resource. It’s a bucket with a ceiling, and there are three separate ceilings:

• Request Througput per Storage Account: around 20,000 requests/sec.
• Bandwidth Limit per Storage Account: 7.5 GB/s for ingress and 25 GB/s for egress.
• Storage Capacity per Storage Account: 5 PiB.

The problem is when a single account doesn’t have enough capacity for your workload. You cannot scale a storage account up — there’s no larger instance size to move to. You get throttled, and throttling on the storage layer surfaces as latency and errors in every service that reads or writes blobs.

The usual workaround is to spread data across several accounts. That works, and it immediately creates a worse problem: now the application has to know which account holds which blob. Account identity leaks into application code, and every consumer needs the same placement logic. Rebalancing means changing all of them at once.

There’s a second, quieter cost. LIST and HEAD against object storage are slow and individually metered. Listing a prefix means asking the storage provider to walk its own namespace and charging you per operation — and once data is spread across N accounts, a single logical LIST becomes N provider calls that you then have to merge.

Solution

A gateway that owns placement, plus a metadata store that owns the namespace.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

           Clients
              │
              ▼
┌──────────────────────────────┐
│   Bucket Gateway (FastAPI)   │
│  Authorization & Validation  │
│  Placement Algorithm         │
│  Circuit Breaker             │
└───────────────┬──────────────┘
      ┌─────────┴─────────┐
      ▼                   ▼
┌───────────┐       ┌───────────┐
│ Metadata  │       │  Storage  │
│ Postgres  │       │  Shard 1  │
│ + Redis   │       │  Shard 2  │
└───────────┘       │  Shard 3  │
                    └───────────┘

The key design decision is that Postgres, not object storage, is the source of truth for the namespace. Object storage holds bytes at opaque keys (blobs/{uuid}); Postgres holds the mapping from (bucket, key) to (shard, version, size, etag, content-type).

Around that sit four pieces: a placement algorithm that picks a shard for each write, a circuit breaker per storage account, a Redis cache in front of the metadata, and a background cleanup worker.

Solving the Problem

Remove per-account limit

With N accounts behind the gateway, capacity, QPS, and bandwidth are all N times a single account’s limit. Managing multiple accounts is straightforward: adding a new shard simply requires adding an entry to STORAGE_SHARDS and restarting the gateway—with zero impact on existing read workloads.

Placement logic stays inside the gateway, so clients never learn about shards

On write:

1. Authorize the request against the bucket.
2. Check preconditions such as If-None-Match.
3. Select a shard — filter out accounts over 90% capacity or 80% QPS, exclude any with an open circuit breaker, prefer a regional account if one is requested, then choose randomly among whatever is left.
4. Upload under a versioned key, blobs/{uuid}.
5. On failure, retry against a different shard.
6. Commit metadata to Postgres atomically.
7. Update the Redis cache.

The algorithm abstracts away how to handle unreliable, close-to-limit accounts in step 3 and 5. Clients are almost guaranteed to find a shard to write to in a pool of shards.

Reduce LIST and HEAD costs

LIST is a Postgres query with a filter. HEAD is a look up that usually hits Redis. Both operations stop being O(number of shards) provider calls and become one indexed query — which removes the per-operation storage charge and the fan-out latency at the same time.

Benchmarks

Circuit breaker prevent degraded account from causing outage

Three failures within 60 seconds opens the breaker for five minutes; the account drops out of the eligible set for placement, then gets probed in half-open state and closes automatically on success. Failures route around, rather than pile up.

Implementation

Source code

— Sep 5, 2026