Cleaning Up Your Bitwarden Vault: Introducing the Bitwarden Deduplicator Tool

Introduction

If you’re serious about online security, you’re probably using a password manager like Bitwarden to store your credentials. Bitwarden has gained popularity for being open-source, feature-rich, and available across all major platforms. However, if you’ve been using it for a while, you might have noticed an annoying issue: duplicate entries cluttering your vault.

These duplicates can accumulate over time due to various reasons—importing from multiple sources, syncing issues, or even manual entry errors. Not only do they create visual clutter, but they also make your vault harder to navigate and increase the size of your exports.

After encountering this problem myself, I decided to create a solution: the Bitwarden Deduplicator, a simple yet powerful tool to clean up your Bitwarden vault by removing duplicate entries.

The Problem: Duplicate Entries in Bitwarden

Duplicate entries in password managers are more common than you might think. They typically occur in these scenarios:

• Multiple imports: If you’ve migrated from other password managers or imported CSV files multiple times
• Syncing issues: When conflicts occur during synchronization across devices
• Browser extensions: Some browser extensions might create duplicates when saving credentials
• Manual entries: Creating entries manually without checking if they already exist

In my case, I had accumulated nearly 1,800 items in my Bitwarden vault over several years. After running an analysis, I was shocked to discover that almost 43% of these were duplicates! This not only made finding the right credentials more difficult but also significantly increased the size of my JSON exports.

The Solution: Bitwarden Deduplicator

To address this problem, I created the Bitwarden Deduplicator, a Python utility that processes your Bitwarden JSON export file, identifies and removes duplicates, and creates a clean file that you can import back into Bitwarden.

Key Features

• Removes duplicate folders and items from Bitwarden JSON exports
• Preserves the first occurrence of each unique entry
• Maintains references between items and folders
• Generates detailed summary reports of the deduplication process
• Command-line interface with customizable options
• No external dependencies (uses only Python standard library)

Technical Approach

The tool identifies duplicates using these criteria:

• For folders: Folder names are used as unique identifiers
• For items: A composite key is created from the item’s name, type, username, and URIs

This approach ensures that true duplicates are removed while preserving entries that might appear similar but contain different critical information.

How It Works

The deduplication process follows these steps:

1. Export your vault: Export your Bitwarden vault as an unencrypted JSON file
2. Run the deduplicator: Process the JSON file with the deduplication tool
3. Review the summary: Check the deduplication statistics
4. Import the clean file: Import the deduplicated JSON file back into Bitwarden

Let’s look at the core functionality that handles the deduplication:

# Process folders - remove duplicates
unique_folders = {}
for folder in data.get('folders', []):
    folder_name = folder.get('name', '')
    if folder_name and folder_name not in unique_folders:
        unique_folders[folder_name] = folder

# Process items - remove duplicates
unique_items = {}
for item in data.get('items', []):
    # Create a composite key for deduplication
    key_parts = [
        item.get('name', ''),
        str(item.get('type', ''))
    ]
    
    login = item.get('login', {})
    if login:
        username = login.get('username', '')
        if username:
            key_parts.append(username)
        
        uris = login.get('uris', [])
        for uri_obj in uris:
            uri = uri_obj.get('uri', '')
            if uri:
                key_parts.append(uri)
    
    key = '|'.join(key_parts)
    
    if key and key not in unique_items:
        unique_items[key] = item

This code creates dictionaries to store unique folders and items, using carefully chosen keys to identify duplicates. The first occurrence of each unique entry is preserved, while subsequent duplicates are discarded.

Real-World Results

In my personal vault, the results were impressive:

• Original file: 1.38 MB with 88 folders and 1,794 items
• Deduplicated file: 0.80 MB with 46 folders and 1,015 items
• Results: Removed 42 duplicate folders (47.7% reduction) and 779 duplicate items (43.4% reduction)
• Overall size reduction: 42.1%

Here’s a visual representation of the before and after:

Before Deduplication:
Folders: [████████████████████████████████████████████████] 88
Items:   [████████████████████████████████████████████████] 1,794
Size:    [████████████████████████████████████████████████] 1.38 MB

After Deduplication:
Folders: [██████████████████████████                      ] 46 (47.7% reduction)
Items:   [██████████████████████████                      ] 1,015 (43.4% reduction)
Size:    [██████████████████████████                      ] 0.80 MB (42.1% reduction)

Using the Tool

Installation

No installation is required. Simply download the script from the GitHub repository and run it with Python 3.6 or higher.

# Clone the repository
git clone https://github.com/biplobice/bitwarden-deduplicator.git

# Navigate to the directory
cd bitwarden-deduplicator

# Make the script executable (Unix/Linux/macOS)
chmod +x deduplicate_bitwarden.py

Basic Usage

python deduplicate_bitwarden.py your_bitwarden_export.json

This will create a deduplicated file named your_bitwarden_export_deduplicated.json in the same directory.

Advanced Options

python deduplicate_bitwarden.py your_bitwarden_export.json -o custom_output.json -s summary.md -q

Command-line Arguments

• input_file: Path to the Bitwarden JSON export file (required)
• -o, --output: Path to save the deduplicated JSON file
• -s, --summary: Path to save the deduplication summary in Markdown format
• -q, --quiet: Suppress progress output

Exporting from and Importing to Bitwarden

Exporting from Bitwarden

1. Log in to your Bitwarden vault
2. Go to “Tools” > “Export Vault”
3. Choose “JSON (Unencrypted)” as the file format
4. Enter your master password and click “Export Vault”
5. Save the JSON file to your computer

Importing back to Bitwarden

1. Log in to your Bitwarden vault
2. Go to “Tools” > “Import Data”
3. Select “Bitwarden (json)” as the file format
4. Choose your deduplicated JSON file
5. Click “Import Data”

Security Considerations

When working with password manager exports, security should be your top priority:

• The script processes unencrypted Bitwarden JSON exports, which contain sensitive information
• Always handle these files securely and delete them after use
• Run the script on a trusted computer
• Consider using the -q (quiet) option to prevent sensitive information from appearing in terminal output

Technical Implementation Details

Design Decisions

Several key design decisions guided the development of this tool:

1. No external dependencies: The script uses only the Python standard library to maximize compatibility and minimize security risks.
2. Command-line interface: A simple CLI provides flexibility while maintaining ease of use.
3. Composite keys for deduplication: Using multiple properties to identify duplicates ensures accuracy.
4. Preservation of folder references: The tool maintains the relationship between items and their folders.

Challenges and Solutions

One of the main challenges was determining what constitutes a “duplicate” in Bitwarden. After analysis, I found that using a composite key of name, type, username, and URIs provided the best balance between removing true duplicates and preserving unique entries.

Another challenge was handling folder references. When folders are deduplicated, the items that reference them need to be updated accordingly. The solution was to create a mapping between old and new folder IDs and update the references during processing.

Future Improvements

Some planned enhancements for future versions:

1. Interactive mode: Allow users to review and confirm each duplicate before removal
2. Custom deduplication rules: Let users define their own criteria for identifying duplicates
3. Direct API integration: Connect directly to the Bitwarden API (with proper authentication)
4. GUI interface: Create a simple graphical interface for users who prefer not to use the command line

Conclusion

The Bitwarden Deduplicator is a simple yet effective tool for cleaning up your password vault. By removing duplicates, you can:

• Make your vault more organized and easier to navigate
• Reduce the size of your exports and backups
• Improve the overall performance of your Bitwarden experience

I created this tool to solve my own problem, but I’m sharing it with the community in hopes that others will find it useful too. The code is open-source under the MIT license, so feel free to use it, modify it, or contribute to its development.

Give it a try and let me know how it works for you! You can find the tool on GitHub at https://github.com/biplobice/bitwarden-deduplicator.

Resources

• Bitwarden Deduplicator on GitHub
• Bitwarden Help Center
• Bitwarden Community Forums

---

Have questions or suggestions? Feel free to open an issue on GitHub or reach out to me directly.

This tool is not affiliated with or endorsed by Bitwarden Inc.