tt manages content addressed file pools that are physically moved between hosts on an offline carrier HDD. Each pool is described by a list file that records, for every file, a hash, the exact byte size, and a path relative to the pool root. Matching between pools is always by hash, so the two hosts and the carrier can each mount the pool at a different base path. The base path is passed on the command line and never stored.

The list format is JSONL (one JSON object per line):

{"hash_type":"md5","hash":"9dcedb82e91028c79ee67a2eb1977268","size":1200000,"path":"Action/film1.mkv"}

hash_type travels with every record, so a pool can migrate from md5 to sha256 without a format break. size is the exact logical size (metadata().len()), not on-disk block usage, since that varies by filesystem.

Build

With Nix (a dev shell with the Rust toolchain is provided):

nix develop        # drops you into a shell with cargo, rust-analyzer, just
cargo build --release

Or with a normal Rust toolchain:

cargo build --release

The binary is target/release/tt. Linux only.

Typical usage

The workflow moves everything in a source pool that a destination pool does not yet have, via a carrier HDD, verifying the hash at every hop.

# 1. hash both pools (on their respective hosts)
tt scan --root /srv/pool-a --out source.jsonl   # source host
tt scan --root /srv/pool-b --out dest.jsonl     # dest host

# 2. work out what the destination is missing (by hash)
tt diff --source source.jsonl --dest dest.jsonl --out 2cp.jsonl

# how much data is that?
tt sum --list 2cp.jsonl

# 3+4. copy the missing files onto the carrier HDD.
#      each file is hashed while it streams and verified before it counts.
tt copy --list 2cp.jsonl \
        --src-root /srv/pool-a \
        --dest-root /mnt/carrier \
        --done carry.done.jsonl --conflict carry.conflict.jsonl

# 5. physically move the carrier HDD to the other location.

# 6. copy from the carrier into the destination pool (different base path,
#    same relative layout), again verified by hash.
tt copy --list carry.done.jsonl \
        --src-root /mnt/carrier \
        --dest-root /srv/pool-b \
        --done dest.done.jsonl

Keeping a pool list current over time uses the incremental scan, which only hashes new and changed files instead of rehashing terabytes every run:

tt scan --root /srv/pool-a --out source.new.jsonl \
        --update source.jsonl \
        --log-deleted removed.jsonl \
        --log-changed replaced.jsonl

To verify a pool has not rotted, recheck it. Fast checks path and exact size; slow also rehashes:

tt recheck --list source.jsonl --root /srv/pool-a          # fast
tt recheck --list source.jsonl --root /srv/pool-a --slow   # rehash

Have an old md5sum style list from before tt? Import it into the new format without rehashing the whole pool:

tt import --old legacy-list.md5 --root /srv/pool-a --out source.jsonl

Need to go the other way and feed a tool that still wants the old format? Export back to md5sum style (see export below).

Commands

All matching is by hash. Every command that touches files takes an explicit root so different mount points are handled by the caller.

`scan` (alias `calc-md5`)

Walk a pool root, hash every file, write a list.

option	description
`--root <ROOT>`	pool root to walk
`--out <OUT>`	list file to write
`--hash <HASH>`	hash algorithm, `md5` (default) or `sha256`
`--exclude <GLOB>`	glob matched against each path component; matching files and whole directory subtrees are skipped. Repeatable. Passing any `--exclude` replaces the default `.Trash-*`
`--update <BASE>`	incremental mode: reuse `BASE` as a starting point. Files whose size is unchanged keep their recorded hash and are not rehashed; only new and size-changed files are hashed; vanished files are dropped. Same-size corruption is not caught here, use `recheck --slow` for that
`--log-deleted <FILE>`	with `--update`, write records whose file vanished from disk here so they can be archived later
`--log-changed <FILE>`	with `--update`, write the superseded old record of each size-changed file here (the rehashed record goes to `--out`)

`recheck`

Verify a pool against its list.

option	description
`--list <LIST>`	list to verify
`--root <ROOT>`	pool root
`--slow`	also rehash every file, detecting silent corruption and bit rot
`--update <FILE>`	write a corrected list here, rehashing only what changed
`--log-changed <FILE>`	write the superseded old record of each size-changed or hash-mismatched file here, preserving the previously recorded (known-good) hash instead of letting `--update` overwrite it
`--log-deleted <FILE>`	write records whose file is missing on disk here (they are dropped from the corrected list), mirroring `scan --log-deleted`

Exits non-zero if any file is missing, size-changed, or hash-mismatched.

`diff`

Emit the records present in source but missing (by hash) from dest: the list of what still needs copying.

option	description
`--source <SOURCE>`	source list
`--dest <DEST>`	destination list
`--out <OUT>`	to-copy list to write

`copy` (alias `diff-cp`)

Copy every file in a list from one root to another, preserving the relative path. Each file is hashed as it streams and verified against the recorded hash before it is accepted.

option	description
`--list <LIST>`	list of files to copy
`--src-root <SRC_ROOT>`	root the files are read from
`--dest-root <DEST_ROOT>`	root the files are written under
`--min-free <BYTES>`	keep at least this many free bytes on the destination filesystem (default 1 GiB)
`--done <FILE>`	write successfully copied records here
`--conflict <FILE>`	write records that could not be copied (already present with different content, or hash mismatch) here

Files already present with matching content are skipped. On a hash mismatch the partial file is removed. Exits non-zero if there were any conflicts or errors.

`sum` (alias `show2cp`)

Sum the exact byte size of every record in a list and print a human readable total. No du, the size is already in the list.

option	description
`--list <LIST>`	list to total

`dups` (alias `show-eq-md5`)

Report hashes that appear on more than one path (duplicate content).

option	description
`--list <LIST>`	list to scan for duplicates

`import`

Convert a legacy md5sum style list (absolute paths, no size) into the new format without rehashing. Paths outside --root and files missing on disk are skipped with a warning, and any skipped entry makes the run exit non-zero so a partial import cannot pass unnoticed.

option	description
`--old <OLD>`	legacy list file
`--root <ROOT>`	pool root the legacy absolute paths live under
`--out <OUT>`	new-format list to write
`--hash <HASH>`	hash algorithm the legacy digests were produced with (default `md5`)

`export`

Convert a new-format list back to the legacy md5sum style format (<md5> <path>), for tooling that still expects the old layout. Only md5 records are emitted; other hash types are skipped with a warning and make the run exit non-zero, since the old column layout assumes a 32-char md5.

option	description
`--list <LIST>`	new-format list to read
`--out <OUT>`	legacy-format file to write
`--root <PREFIX>`	prefix prepended to each relative path to rebuild the absolute paths the old lists used; omit to emit relative paths

Development

Source layout: src/main.rs holds only the CLI definition and dispatch; each subcommand lives under src/commands/. Shared building blocks sit in format (JSONL list I/O), hash, legacy (old md5sum-list parsing), progress and util.

Run the checks CI runs:

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test

The integration tests in tests/cli.rs drive the real binary against throwaway pools and encode every bug class fixed during the 1.99 hardening pass, so regressions fail loudly.

README.md

TerabyteTerminator (tt)

Build

Typical usage

Commands

scan (alias calc-md5)

recheck

diff

copy (alias diff-cp)

sum (alias show2cp)

dups (alias show-eq-md5)

import

export