Duplicate Line Remover

About the Duplicate Line Remover

This strips repeated lines out of a list, keeping only the first appearance of each. It's the quick fix for de-duplicating an email or address list, cleaning up a merged export where rows appear twice, tidying a keyword or tag list, or reducing a log to its unique lines. The options let you treat lines as duplicates even when they differ only by capitalisation or by leading and trailing spaces.

How it works

It splits your text into lines and walks through them in order, remembering which it has seen. The first time a line appears it's kept; any later identical line is dropped, and a running count tells you how many duplicates were removed. What counts as "identical" depends on the toggles:

• Trim whitespace — leading/trailing spaces are ignored when comparing (and trimmed in the output).
• Ignore case — Apple and apple are treated as the same line.
• Remove blank lines — empty lines are dropped entirely rather than kept.

Assumptions and behaviour

• The first occurrence wins and the original order of the remaining lines is preserved — it doesn't sort.
• Comparisons apply trim and case-folding only if you enable them; by default, lines must match exactly (including spacing and case) to count as duplicates.
• With trim on, kept lines are output trimmed, not just compared trimmed.

Limitations

• It removes whole-line duplicates only — it can't find duplicated words within a line, or near-duplicates that differ by a character or punctuation.
• "Ignore case" uses simple lowercasing, which is reliable for English but may not fold every accented or non-Latin character the way a locale-aware comparison would.
• It keeps the first of each duplicate; there's no option to keep the last instead.
• Very large lists are processed in memory in your browser, so extremely big files depend on available RAM.

Privacy

De-duplication runs entirely in your browser. Your list is never uploaded or stored.