Line sorter and deduplicator

A mailing list or a log with the same line twice. We compare whole lines. ‘Jon / Jonathan’ is two rows.

A volunteer list exported twice, a robots.txt allow list with copies, a log you want unique IPs from. This page is exact line unique plus optional sort. It is not a CRM merge and not fuzzy matching.

Whitespace-only differences survive unless you clean whitespace first. Editing one haystack is find and replace. Comparing two versions is text diff.

Mailing lists and error storms

The same address pasted twice, or a log line that repeats. We key on the whole line. Full-width vs half-width “same” emails can survive. Leading spaces hide duplicates until you run the whitespace cleaner. Not fuzzy name matching.

How to use it

1. Paste one item per line.
2. Deduplicate. Sort if you need a stable eyeball pass.
3. Copy. Spot-check the first and last lines.

Honest limits

Not Unicode locale sort for every language. Not a 2 GB Hadoop job. Memory is the ceiling.