Send a Folder and Pipe a Stream

Two small open-source commands for moving bytes between two machines. nfltr-send sends files and folders, verified and resumable. nfltr-pipe joins one machine's output to another's input. Each prints an address; whoever holds it receives.


Goal

Send the folder photos/ from one machine to another, then copy a project directory as a tar stream, with neither machine opening a port.

Prerequisites

1. Send a folder

On the sending machine, serve the folder and write its address to a file:

sender$ nfltr-send send photos/ > address
serving 12 files, 48213504 bytes; give the receiver this address (a secret: whoever holds it can receive):

The address is a secret: whoever holds it can receive. Copy the file address to the other machine over a channel you trust, then receive:

receiver$ nfltr-send receive --to ~/Downloads < address
received 12 files, 48213504 bytes in 0.6 s (80 MB/s), path direct

~/Downloads/photos appears only once every chunk of every file has arrived and verified. A cut transfer resumes: run the same receive again and only what is missing is sent. A name that already exists in ~/Downloads refuses the transfer; nothing is replaced. The sender keeps serving until you stop it (Ctrl+C).

2. Pipe a stream

On the source machine, stream a directory into nfltr-pipe listen. It prints an address for the other end:

src$ tar -C ~/project -cf - . | nfltr-pipe listen
listening; give the other end this address (a secret: whoever holds it can connect):
nf1...

On the destination, paste the address into a file only you can read, so it never shows on a command line, then dial:

dst$ umask 077; cat > addr    # paste the address, Enter, then Ctrl-D
dst$ nfltr-pipe dial --address-file addr | tar -C ~/project -xf -

dial exits 0 only after the sender's last byte and its end arrived. A transfer that fails part way exits non-zero on both ends, and tar sees a truncated archive, never a complete one. Any command that writes a stream works on the left, and any that reads one on the right: a database dump into a restore, a log into grep.

Next