Segmented listing
Split the key space into independent segments and scan them concurrently instead of walking one long chain.
Rust CLI / Object Storage
List huge S3-compatible buckets without waiting on one slow chain.
s3-turbo-list is a high-performance CLI for operations teams and data engineers: concurrent bucket inventory, provider diagnostics, traceable runs, diffing, checkpoint/resume, and Parquet outputs for analysis.
$ s3-turbo-list --dry-run --agent \
--output-dir out --delimiter '' \
list --region us-east-2 --bucket my-bucket
plan.segments 64
plan.output out/*.parquet
trace.enabled true
checkpoint.resume ready
$ s3-turbo-list manifest-summary run.json --check
artifacts ok
parquet.rows verified
exit_code 0Features
Trace JSONL
Record every S3 API call as structured JSONL so provider behavior is observable after the run.
Checkpoint / resume
Resume interrupted scans from saved segment progress with identity checks to avoid mismatched runs.
Parquet export
Write inventory and diff results directly into compressed Parquet for DuckDB, pandas, and audit workflows.
Why it exists
Sequential listing works until the bucket is large enough to become an operation.
Standard listing tools are often bound to a single ListObjectsV2 pagination chain. That is fine for small buckets, but painful when object counts move into millions or hundreds of millions.
s3-turbo-list slices the key space, runs listing work in parallel, and keeps the run auditable through manifests, traces, checkpoints, and Parquet artifacts.
Workflow
Probe
Run compat-probe or doctor first to validate endpoint behavior before a large listing.
Plan
Use dry-run and agent-friendly JSON to inspect output paths, hints, and warnings.
Scan
List with parallel segments, bounded memory, run manifests, and trace output.
Analyze
Load Parquet in DuckDB or pandas, or inspect trace JSONL for provider-level behavior.
Compatibility
Built for real S3-compatible endpoints, not only the happy path.
AWS S3MinIOCloudflare R2BOSOSSB2
Endpoint presets, compat-probe, trace output, and validation reports make it easier to understand how each provider behaves before scaling up a scan.