# Remove Duplicate Lines Deduplicate lines in a list or text block. Keep the first or last occurrence, with options for case sensitivity, whitespace trimming, and dropping empty lines. > Canonical page: https://elysiatools.com/en/tools/remove-duplicate-lines - **Category:** Text Processing - **Keywords:** remove duplicate lines, deduplicate lines, dedupe lines, unique lines, remove duplicate rows, clean list ## Overview Remove duplicate lines from a list or text block and report how many were dropped. Useful for cleaning up CSVs, log files, URL lists, keyword dumps, and any place repeats sneak in. **Keep mode:** - **First occurrence** — when a line appears more than once, the *first* copy is kept and later copies removed. This is the usual choice: it preserves the original top-to-bottom order you pasted. - **Last occurrence** — the *last* copy is kept instead. Handy when the most recent entry is the one you trust (e.g. a log where the latest line supersedes earlier ones). **Comparison options:** - **Case sensitive** (default on) — treats `Apple` and `apple` as different lines. Turn it off to treat them as duplicates. - **Trim whitespace before comparing** — strips leading/trailing spaces and tabs on each line before deciding if two lines are equal, so ` hello` and `hello` count as the same line. (Trimming is used only for comparison; the original spacing in kept lines is preserved in the output.) - **Ignore empty lines** — drops empty lines (lines with no characters or only whitespace) entirely, rather than collapsing them to one. **Good to know:** - Whole-line comparison only — this tool does not deduplicate individual words or fields within a line. - Counts in the metadata tell you the original line count, the kept line count, and exactly how many duplicates were removed, so you can confirm the cleanup. ## Inputs - **Text** (textarea): Paste a list or text block with duplicate lines... - **Keep** (select) - **Case sensitive** (checkbox) - **Trim whitespace before comparing** (checkbox) - **Ignore empty lines** (checkbox) ## When to use - Cleaning up raw log files where duplicate entries clutter the analysis. - Deduplicating lists of URLs, email addresses, or keywords for marketing campaigns. - Sanitizing raw CSV rows or configuration lists before importing them into databases. ## How it works - Paste your list or text block into the input text area. - Configure your comparison preferences, such as keeping the first or last occurrence, case sensitivity, and whitespace trimming. - Process the text to instantly generate a clean list and view the count of removed duplicates. ## Use cases - Removing duplicate URLs from a web scraping export to prepare a clean sitemap. - Filtering out repeated error messages in server logs while keeping the latest entry. - Cleaning up a mailing list by removing duplicate email addresses regardless of trailing spaces or casing. ## Frequently asked questions ### Can I deduplicate individual words within a single line? No, this tool only performs whole-line comparisons and deduplicates entire lines, not individual words or fields. ### What is the difference between keeping the first or last occurrence? Keeping the first occurrence preserves the original top-to-bottom order of your list, while keeping the last occurrence retains the most recent duplicate entry. ### How does the whitespace trimming option work? It strips leading and trailing spaces or tabs before comparing lines, but preserves the original spacing of the kept lines in the final output. ### Will this tool remove empty lines? Yes, if you enable the 'Ignore empty lines' option, all empty or whitespace-only lines will be completely removed. ### Is the deduplication case-sensitive? By default, yes. You can disable the 'Case sensitive' option to treat uppercase and lowercase variations of the same text as duplicates. ## Related tools - [Whitespace Normalizer](https://elysiatools.com/en/tools/whitespace-normalizer): Replace consecutive spaces/tabs with single space and trim line start/end whitespace - [Empty Line Remover](https://elysiatools.com/en/tools/empty-line-remover): Remove all empty lines from text (lines with no content) - [HTML Tag Stripper](https://elysiatools.com/en/tools/html-tag-stripper): Remove HTML tags and extract clean text content - [Line Number Remover](https://elysiatools.com/en/tools/line-number-remover): Remove line numbers from text or code with intelligent pattern recognition - [Non-Alphanumeric Cleaner](https://elysiatools.com/en/tools/non-alphanumeric-cleaner): Remove all non-alphanumeric special characters from text with flexible preservation options - [Number Format Cleaner](https://elysiatools.com/en/tools/number-format-cleaner): Normalize and clean various number formats in text for consistency - [Prefix Remover](https://elysiatools.com/en/tools/prefix-remover): Remove common prefixes from each line with intelligent detection and customizable options - [Suffix Remover](https://elysiatools.com/en/tools/suffix-remover): Remove common suffixes from each line with intelligent detection and customizable options ## Samples - [Duplicate Line Samples](https://elysiatools.com/en/samples/text-duplicate-line-samples): Sample files with various types of duplicate lines for testing duplicate removal tools - [Special Characters Samples](https://elysiatools.com/en/samples/text-special-characters-samples): Sample text files with various special characters, punctuation, and symbols for testing non-alphanumeric character removal - [Code Comment Samples](https://elysiatools.com/en/samples/code-comment-samples): Sample code files with various comment styles for testing comment removal - [Android File Operations Java Samples](https://elysiatools.com/en/samples/android-file-operations-java): Android Java file operations examples including text file read/write, file copy/move, directory traversal, and file validation ## Related content - [Text Case, Encoding, and Normalization Conversion Tools](https://elysiatools.com/en/hubs/text-convert): Compare text case conversion, character-width conversion, encoding conversion, quoted-printable handling, and inline text normalization tools in one hub. - [Text Tools](https://elysiatools.com/en/hubs/text-utility): Explore 33 text tools for utility workflows and compare closely related utilities quickly. - [Text Analysis, Readability, and Content Inspection Tools](https://elysiatools.com/en/hubs/text-analyze): Compare text statistics, language detection, readability scoring, sentiment analysis, moderation review, and pattern analysis tools in one hub. - [Text Redaction, Highlighting, and Presentation Formatting Tools](https://elysiatools.com/en/hubs/text-format): Compare tools for masking sensitive text, finding PII, normalizing phone numbers, highlighting phrases, centering text, and formatting diffs in one hub.