How it works
By Pedro Albaladejo · ClearCopies
Updated August 12, 2026 · 7 min read
How Metadata-Based Duplicate Detection Works
Follow the pipeline from read-only inventory through exact grouping, review, and plan export.
Metadata detection is more than sorting a CSV. A trustworthy result depends on complete enumeration, explicit scan state, consistent hash parsing, deterministic grouping, and honest treatment of missing data.
Build the pipeline as distinct phases: discover, normalize, candidate bucket, exact match, safety classification, review, and export. Preserve the evidence for each group so a reviewer can understand why it appeared.
A practical checklist
- 1Enumerate the drive by reading metadata.
- 2Normalize stable IDs, byte sizes, locations, and hash values.
- 3Bucket equal sizes and compare quickXorHash.
- 4Attach context and require at least one keeper.
- 5Export the reviewed plan before any file changes.
Understand the situation
Keep detection evidence separate from retention judgment. Technical signals can establish that bytes match, while path, permissions, history, and ownership determine whether one file item is operationally redundant.
Use a controlled workflow
A robust pipeline handles pagination, missing values, throttling, and changes during the scan. Incomplete data should lower confidence or remain unresolved instead of being converted into a clean-looking but unsafe result.
Make the final decision
Treat exactness and removability as separate outputs. The engine can say that bytes match; the user decides which item identity survives based on path, owner, sharing, history, and policy.
ClearCopies has a deliberately narrow promise: scan supported cloud metadata, confirm exact candidates by byte size plus a matching provider fingerprint, let the user review, and export the plan without downloading original content.
Limits and risks to check
- — Partial enumeration makes group counts incomplete.
- — Missing hashes must not inherit confidence from names.
- — Stale item state can invalidate a later action.
- — Metadata handling needs the same security discipline as other sensitive data.
Official references
Frequently asked questions
Does the scanner open documents?
Not in ClearCopies’ MVP exact flow. It reads metadata returned by OneDrive.
Can metadata detect similar documents?
Not semantic or visual similarity. It identifies exact byte relationships when supported fingerprints are available.
Why export a plan?
It makes keep decisions, evidence, and exceptions reviewable before any file is changed.
Scan first. Decide with evidence.
ClearCopies reads supported cloud-drive metadata and groups exact copies by byte size plus a provider-supplied content fingerprint. Original file bodies are not downloaded for the scan. You review the result and export a plan before any file moves.