s3-turbo-list · Rust CLI
A CLI that discovers parallel work for large S3-compatible list / diff runs.
v0.21.0 field benchmark
2 vCPU host · default list QPS · OSS Beijing. Observed under constraint, not a throughput ceiling.
c=8 is the Parquet reference run. c=24 is a scheduling outlier, not a general limit.
What it does
One CLI for the two questions that matter before a migration or investigation: what is here, and what changed?
Turn a large S3-compatible bucket into an analysis-ready object list without hand-tuning a scanner first.
Compare source and target object metadata, then work from one ordered difference dataset.
Not a sync engine s3-turbo-list does not copy objects or replace a storage browser. It produces a reliable basis for the next decision.
Why it exists
A large-bucket scan is rarely the end of the task. You need an object list for analysis, or a difference set to decide whether a migration is complete.
s3-turbo-list makes those answers its default output. It keeps the command surface small and exposes planning, recovery, and evidence only when the run needs them.
How list adapts
The first list reads the bucket's structure, opens parallel work where that structure exists, then only splits the segments that remain long.
Probe real prefix boundaries before recursive work begins.
Use those boundaries to open parallel list work without a prepared hints file.
Only fan out segments that remain long while useful capacity is available.
Finish with an analysis-ready object list instead of terminal-only output.
What remains
Both paths write results for downstream work. They are deliberately not the same operation.
LIST
Checkpoint and resume keep a long list recoverable; optional trace and manifest files preserve how it ran.
DIFF
Both sides are listed in parallel and merged in key order, so downstream tools can filter what is equal, missing, or changed.
Compatibility
AWS S3, MinIO, and Baidu BOS are covered by the project’s verified compatibility path. R2, OSS, and B2 have presets; run `compat-probe` before trusting a long job on any endpoint.
Questions & how-to
For a quick, small inspection, `aws s3 ls` is often enough. s3-turbo-list is for large inventories and repeatable comparisons where output and recovery matter.
No. List mode probes real `CommonPrefixes` boundaries on the first run. Hints files remain an optional control for repeated inventories.
Startup discovery pre-partitions flat namespaces at list startup with parallel single-key probes, so the first run can open parallel work earlier instead of relying only on runtime splits.