Suggestion/Request: Item Deduplication Script

Updated 11302022-061136

• GitHub Issue
• Created: 10042022-163150
• WTF
• Things

I apologize for hopping right in and quite informally, but I'm actually very confident many 1Password users other than myself have been seeking a method of deduplicating (imported, mostly) items in their vaults because I have spent significant amount of time searching the web - including 1Password's own official community platforms - for a way to reliably achieve this post-the tools deprecation/removal(?) of the native deduplication tool from the macOS client in recent versions.

If it'd be helpful, I can go back through that browsing history and actually cite some Stack Overflow/forum links.

I know that I, personally, am seeking the most discriminating sort of deduplication - as in, it'd still be very helpful if provided methods only worked with precisely identical matches.

Thank you! If there's a more appropriate place for me to share this request, please do let me know. @scottisloud

• 10042022-184627

@extratone thank you so much for your early enthusiasm for this very much in-progress repo of examples!

1Password has definitely heard the requests for deduplication support, so I assure you those requests are on the respective teams' radars, though I don't have anything to share on that front.

A big challenge with de-duplication is assessing what is a duplicate. Even if you identify identical attributes, there's the matter of determining whether it is an accidental duplicate to be deleted, whether it is a deliberate duplicate to be preserved, and what may be a false positive caused by unique items with similar or identical attributes to each other (e.g., domain, username, item name). Even "exact duplicate" is difficult to actually pin down and it could be easy to miss real duplicates that aren't exact matches (e.g., two items that differ only by a UTM suffix on the website value). And finally, once suspected duplicates are identified, there's then the matter of choosing which duplicate is the canonical item. A script is just not going to offer sufficient ability to review suspected duplicates before acting.

Personally I think this is something better handled by the client applications than a CLI script. The client applications have considerable advantages in terms of performance and UX that the CLI could never match, particularly since evaluating duplicates is subjective and usually involves input from the user on a per-item basis.

For many of the reasons above, a general-purpose or broadly-applicable de-duplication script is going to be practically impossible to put together, even for exact matches. At best, it may be possible to use the CLI to address a specific incident that resulted in duplicates with very predictable attributes.

I do not want to dismiss the possibility we may explore an example script at some point, but at this time there are other use-cases that the CLI is more well-suited to address that we will be prioritizing for future examples. But please don't let that stop you from experimenting with approaches, involving the CLI or otherwise, for de-duping.

This is kind of a non-answer, but I want to leave you with something to go on at least. I'm sure you've encountered these before, but some suggestions:

• Use watchtower to find instances of reused passwords. In theory items with identical passwords are highly likely to be duplicates.
• Use a 1Password desktop application to sort items by title. There's a high likelihood duplicates will have the same or similar titles. You can quickly compare the contents to two seemingly identical items.
• Deal with duplicates as they present themselves. In theory, duplicates will have the same domain, so they'll all show up when you go to log into that website. If you notice this, you can deal with those duplicates in that moment, and over time, deal with duplicates for your most commonly-used items.

Thanks for your interest in our new examples repo, and although I can't make any promises about a de-duplication script any time soon, I hope you'll continue to experiment with the 1Password CLI! @tannaurus

• 10052022-002418

👋 hey there @extratone! I'm hoping to grab a small bit of clarification 👀

Let me introduce myself first! My name is Tanner, I'm the engineer behind many of our importers. Your mention of the importer caught my eye 👀

Would you be able to elaborate more on how the imported items are a frequent pain point here? Is this the result of importing an export from another password manager more than once? Or perhaps the export itself had duplicates within it?

I'd be interested in giving our importers a closer look to see if there's something we could do here. Can't make any promises, though. I do love me them importers @scottisloud

• 11232022-105821

I'm going to close this issue for now, since it seems there is nothing immediately actionable but please always feel free to open new issues any time.