List Deduplicator

What This Tool Does

The List Deduplicator takes line-delimited text input and removes duplicate occurrences, returning a unique list. By default it keeps the first occurrence of each value and discards every subsequent match, preserving the original input order — the same semantics as Python's list(dict.fromkeys(items)) idiom and JavaScript's [...new Set(items)] expression. Switching to Sort A–Z, Sort Z–A, or Sort by Length deduplicates first, then sorts; Reverse simply flips the input order without touching duplicates.

Four toggleable options control how the comparison happens: Case-insensitive comparison normalizes each line with .toLowerCase() before the membership check (so Apple and apple collapse into one entry); Trim whitespace calls .trim() on each line first (removing trailing spaces and stray \r from Windows CRLF endings); Remove empty lines drops blank lines from the output; and Number lines prefixes each output line with its 1-based index for use as a numbered reference. The status bar at the bottom reports the original line count, the count after cleanup, the resulting unique count, and the number of duplicates removed — so you always have an audit trail of what happened.

Everything runs client-side in vanilla JavaScript. Your data never leaves your browser tab, which matters when the list is a customer email export, an API key dump, or any other sensitive content where uploading to a third-party server would create compliance or confidentiality risk.

How to Use It

1. Paste your list into the input textarea on the left (or top on mobile). One value per line.
2. Select an operation chip: Remove Duplicates (default), Sort A–Z, Sort Z–A, Sort by Length, or Reverse.
3. Toggle any of the four options as needed — case-insensitive comparison, trim whitespace, remove empty lines, number lines.
4. The deduplicated output renders instantly in the right pane as you type. The status bar shows the before/after counts.
5. Click Copy to put the result on your clipboard, or Download to save it as deduped-list.txt.
6. Use Try Example at any time to load a 10-line fruit list (with three intentional duplicates) so you can see how the modes interact before pasting your real data.

Tip: pressing Tab inside the textarea inserts two spaces instead of jumping focus, which is convenient when you're pasting indented data and want to keep editing in place.

Worked Example: 1,000-Line CSV Column → Deduplicated With Top-10 Chart

Suppose you have a 1,000-row customer-feedback CSV export and you want to count how many distinct categories your support team tagged tickets with. You extracted the category column into a flat text file — 1,000 lines, one category per line, with the obvious duplication you'd expect when only ~30 categories are in active use.

Pasting the column into the deduplicator with default settings (Remove Duplicates mode, Trim whitespace on) produces a clean list of 28 unique categories, with the status bar reporting 1000 items → 28 unique (972 duplicates removed). The 28 categories represent 100% of the distinct values; the 972 removed lines are the long tail of repeat occurrences across the same labels.

To see the distribution — which categories dominate vs. which are tail entries — we ran the same input through the frequency-count snippet from the FAQ and plotted the top 10 most-frequent values. The bar chart below shows the result.

Interpreting the result: the top 3 categories alone explain 47.3% of all tickets — a clear signal for where to focus self-service documentation or chatbot intent coverage. The long tail of 18 remaining categories (each appearing fewer than 41 times) likely contains a mix of edge cases worth keeping and obsolete labels worth retiring. Either decision starts from a clean unique list, which is what the deduplicator delivers in one step.

Common Use Cases

The List Deduplicator covers a surprisingly broad set of day-to-day cleanup tasks. The five contexts below are the most-reported reasons users hit this tool.

Email list cleanup

Marketing exports, CRM dumps, and form submissions accumulate duplicate email addresses from re-submissions, multi-list memberships, and copy-paste errors. Sending the same campaign twice to the same address damages sender reputation with Gmail and Outlook deliverability filters. Paste the list, turn on Case-insensitive comparison (since JOHN@example.com and john@example.com route to the same mailbox per RFC 5321), and the deduper collapses them. Pair with the Email Normalizer first if you also need to strip Gmail dot-aliases or +tag suffixes.

Log file analysis

Grepping a server log for unique error messages, IP addresses, or user-agent strings is a one-liner with grep | sort -u at the command line — but if you're already in a browser tab reviewing a downloaded log, pasting the column into this tool is faster than spinning up a terminal. Toggle off Remove empty lines if blank-line spacing carries information; leave on if not.

Generating unique IDs

Bulk-importing records into a system that requires unique slugs, SKU codes, or human-readable identifiers? Generate a candidate list (often programmatically), paste into the deduper to catch collisions, and pair the output with the URL Slug Generator to normalize formatting. The status bar's duplicate count tells you whether your generation algorithm has a collision problem worth fixing upstream.

Removing duplicate URLs

SEO sitemap audits, crawl result cleanup, and link-list curation all involve URL deduplication. Be careful: https://example.com/path and https://example.com/path/ (trailing slash) are different strings and will both survive — pre-normalize trailing slashes before pasting, or accept that trailing-slash variants are genuinely different URLs from a crawler's perspective. The URL Parser can help you canonicalize URLs first.

SQL SELECT DISTINCT alternative for ad-hoc data

When the data isn't in a database — it's in a CSV, a chat log, or a spreadsheet column — you don't have SELECT DISTINCT col FROM table available. The List Deduplicator is the in-browser equivalent: paste the column, get the distinct list. No connection string, no import step, no temporary table. For multi-column distinct (where the unique key spans columns A and B together), you still need Excel or pandas; this tool is single-column / line-oriented by design.

Edge Cases

Most edge cases come from invisible characters or encoding subtleties — the values look identical to a human eye but are not equal to a strict string-equality check. The five below catch the majority of real-world surprises.

Unicode normalization (NFC vs NFD)

Unicode encodes some accented characters two different ways. The string café can be one of:

• NFC (Normalization Form Canonical Composition): the precomposed code point U+00E9 for é, four bytes total in UTF-8.
• NFD (Normalization Form Canonical Decomposition): base letter e (U+0065) plus a combining acute accent U+0301, five bytes total.

To the eye they are identical; to JavaScript's === operator they are different strings. The deduplicator will keep both entries because the underlying Set uses strict equality. macOS file APIs return file names in NFD by default; most other systems (Windows, Linux, web forms) use NFC. If your input mixes data from both sources, pre-normalize with input.split('\n').map(l => l.normalize('NFC')).join('\n') in the browser console before pasting. A native Unicode normalization toggle is on the roadmap for a future version.

Trailing-whitespace duplicates

CSV exports, log scrapes, and clipboard-from-PDF input commonly carry trailing spaces, tabs, or stray \r characters from Windows CRLF line endings being split on \n only. Two values that look identical to a spreadsheet user can be strict-unequal in JavaScript: "john@example.com" and "john@example.com " differ by one trailing space. The Trim whitespace checkbox (on by default) calls .trim() on each line before the Set membership check, which fixes this entire class of duplicate. Leave it on unless leading whitespace is semantically meaningful (e.g., YAML indentation or significant whitespace in poetry).

BOM-prefixed first line

UTF-8 files saved by Windows Notepad or older Excel exports often begin with a byte-order mark (BOM, U+FEFF) on the very first line. When that file is split into lines and the first value is, say, email, JavaScript actually sees "email" — invisible to your eye, but distinct from a plain "email" appearing later. The deduper will keep both. .trim() does not remove the BOM in older browser engines. Workaround: pre-strip with input.value.replace(/^/, '') in the console, or in a text editor save the file as UTF-8 without BOM before exporting.

Sort stability when keeping first vs last

The deduper keeps the first occurrence of each value and discards subsequent ones. Sometimes you want the opposite — the most recent entry to win, with earlier duplicates discarded (common in log timelines or version pinning). Workaround until a "keep last" toggle ships: reverse the input first (use Reverse mode, copy the output, paste back), then dedupe — first-occurrence-on-reversed-input equals last-occurrence-on-original-input. JavaScript's Array.prototype.sort is guaranteed stable since ECMAScript 2019, so subsequent Sort A–Z or Sort by Length operations preserve the relative order of equal keys established by the dedup pass.

Memory limits in browser for >1M lines

The browser-side dedup approach holds three structures in memory simultaneously: the original line array, the Set's internal hash table, and the output array. Empirically, 1 million lines of short (≤60-character) strings comfortably fits in a 4 GB Chrome tab; 5 million starts hitting heap limits and triggering long garbage-collection pauses; 10 million reliably throws RangeError: Maximum call stack size exceeded or browser tab crashes. For input larger than 1M lines, drop into Node.js with the same algorithm: const seen = new Set(); require('readline').createInterface({input: fs.createReadStream(file)}).on('line', l => { if (!seen.has(l)) { seen.add(l); console.log(l); } }); — same O(n) time, no heap ceiling, and you can pipe input/output through standard Unix utilities.

Behind the Scenes

The deduplication algorithm is one of the most-discussed examples in introductory data structures and one of the smallest "useful" programs you can write. Below is what's actually happening when you hit Remove Duplicates.

Set vs Map in JavaScript for dedup

JavaScript offers two natural data structures for tracking "have I seen this?" queries: Set (stores values directly, no associated payload) and Map (key → value pairs). For pure deduplication, Set is the right choice — there is no associated payload to store. Set.prototype.has(x) returns true/false; Set.prototype.add(x) inserts if absent. Both operations are O(1) amortized expected time, backed by a hash table in V8 (Chrome's JS engine), JavaScriptCore (Safari), and SpiderMonkey (Firefox).

If you also want occurrence counts in the same pass, swap Set for Map: const counts = new Map(); for (const line of lines) counts.set(line, (counts.get(line) || 0) + 1); — one pass, all unique values as keys, all counts as values. The List Deduplicator uses Set for the membership check and reports a derived duplicate-count via lines.length - unique.length rather than a per-value count, which is sufficient for the headline status-bar number.

Why [...new Set(arr)] is O(n)

The idiomatic one-liner [...new Set(arr)] deduplicates an array in linear time. Internally: the Set constructor iterates arr once (n insertions, each O(1) amortized), then the spread operator iterates the Set once to build the output array (another n operations). Total: 2n operations, which is O(n). It's tempting to assume hash-table operations are "really" O(log n) or worse — they are not; well-designed hash tables hit O(1) amortized for both insert and lookup, with rare O(n) rehash steps amortized across many insertions. The naive double-loop alternative (arr.filter((v,i) => arr.indexOf(v) === i)) is O(n²) because indexOf rescans the array on every element, and it becomes painfully slow above ~10,000 entries. Always prefer the Set approach.

Hash collisions

A hash table works by mapping each string to a numeric bucket via a hash function. Two different strings can hash to the same bucket — a collision — which the table handles by chaining (a linked list per bucket) or by open addressing (probing nearby buckets). With a well-designed hash function (V8 uses a variant of MurmurHash for strings), collisions are rare enough that average-case operations stay O(1). Worst-case adversarial input — where someone deliberately crafts strings that all collide to the same bucket — can degrade performance to O(n), and historically this was a DoS vector against web frameworks. Modern engines mitigate this with seeded hash functions (the seed is randomized per process), so realistic input never hits the worst case. For the deduper's use case — natural-language strings, identifiers, URLs — collisions are negligible.

Unicode normalization API

The browser's built-in String.prototype.normalize(form) method converts a string between the four Unicode normalization forms: NFC (canonical composition), NFD (canonical decomposition), NFKC (compatibility composition), and NFKD (compatibility decomposition). Of these, NFC is the most common for storage and display, NFD is what macOS file APIs return, and NFKC is useful when you want fi (the fi ligature, U+FB01) to collapse to plain fi (U+0066 + U+0069). The internal implementation uses pre-computed Unicode character database tables shipped with the JavaScript engine — fast, no network call. The deduper does not currently call .normalize() automatically; see the Unicode edge case above for the workaround.

Comparison: This Tool vs sort -u vs uniq vs Excel vs Python

Several other tools do something similar. The table below shows where each shines and where it falls down.

The rule of thumb: this tool for pasted in-browser cleanup where order matters and the data is sensitive; sort -u for command-line streaming where alphabetical output is fine; uniq -c when you specifically need occurrence counts at the shell; Excel when the dedup key spans multiple columns; Python when the dedup is one step in a larger scripted workflow.