Deduplicate and sort a list
Tags, email domains, package names, hostnames: any list that has been added to by several people over time has duplicates in different cases, stray spaces and empty lines. This recipe returns one clean, sorted copy.
The steps
- 1
Clean hidden charactersvia Invisible Character Detector
Removes zero-width characters and normalises the odd spaces that make two identical-looking lines compare as different.
- 2
Trim linesvia Line Sorter & Deduplicator
Strips leading and trailing whitespace from each line so "React " and "React" are the same entry.
- 3
Sort linesvia Line Sorter & Deduplicator
Alphabetical, case ignored for both ordering and duplicates, blank lines dropped.
Why three steps and not one
Deduplication only works on lines that are actually identical, and most near-duplicates in a real list are not: one has a trailing space, one has a different case, one has an invisible character pasted in from a web page. The first two steps make lines comparable; the third does the comparing. Running the sort alone on a raw list keeps most of the duplicates it was meant to remove.
Which copy of a duplicate survives
The first one in the input, in its original case. If the list has "react" before "React", the lower-case form is kept. If you want a consistent case across the whole list, run the result through the case converter afterwards; it is one of the tools each step links to.
| Option | What it does |
|---|---|
| Ignore case | Sorts and compares as if everything were lower-case. |
| Remove empty lines | Drops lines that are blank after trimming. |
| Remove duplicates | Keeps the first occurrence of each line. |
| Order | A to Z here; the tool also offers natural order for lists with numbers. |
Lists this works well on
- Allow-lists and block-lists of domains before they go into a config.
- Tags or labels collected from several documents.
- Dependency names gathered from more than one project to find the union.
- Attendee handles for a mailing list, once the case is settled.
Questions
- Can I keep the original order and only remove duplicates?
- Yes: open the line sorter from the last step, turn off sorting and keep the duplicate removal, or use the dedupe operation in a recipe of your own.
- Numbers sort oddly.
- Plain alphabetical order puts 10 before 2. Choose natural order in the line sorter for lists that contain numbers.