How it works
Updated August 12, 2026 · min read
How Metadata-Based Duplicate Detection Works
Follow the pipeline from read-only inventory through exact grouping, review, and plan export.
Metadata detection is more than sorting a CSV. A trustworthy result depends on complete enumeration, explicit scan state, consistent hash parsing, deterministic grouping, and honest treatment of missing data.
Build the pipeline as distinct phases: discover, normalize, candidate bucket, exact match, safety classification, review, and export. Preserve the evidence for each group so a reviewer can understand why it appeared.
A practical checklist
- 1Enumerate the drive under read-only OAuth.
- 2Normalize stable IDs, byte sizes, locations, and hash values.
- 3Bucket equal sizes and compare quickXorHash.
- 4Attach context and require at least one keeper.
- 5Export the reviewed plan without requiring write access.
Understand the situation
Metadata detection is more than sorting a CSV. A trustworthy result depends on complete enumeration, explicit scan state, consistent hash parsing, deterministic grouping, and honest treatment of missing data.
Keep detection evidence separate from retention judgment. Technical signals can establish that bytes match, while path, permissions, history, and ownership determine whether one file item is operationally redundant.
Use a controlled workflow
Build the pipeline as distinct phases: discover, normalize, candidate bucket, exact match, safety classification, review, and export. Preserve the evidence for each group so a reviewer can understand why it appeared.
A robust pipeline handles pagination, missing values, throttling, and changes during the scan. Incomplete data should lower confidence or remain unresolved instead of being converted into a clean-looking but unsafe result.
- Normalize stable IDs, byte sizes, locations, and hash values.
- Bucket equal sizes and compare quickXorHash.
- Attach context and require at least one keeper.
Make the final decision
Treat exactness and removability as separate outputs. The engine can say that bytes match; the user decides which item identity survives based on path, owner, sharing, history, and policy.
ClearCopies has a deliberately narrow promise: scan supported cloud metadata, confirm exact candidates by byte size plus a matching provider fingerprint, let the user review, and export the plan without downloading original content.
Limits and risks to check
- — Partial enumeration makes group counts incomplete.
- — Missing hashes must not inherit confidence from names.
- — Stale item state can invalidate a later action.
- — Metadata handling needs the same security discipline as other sensitive data.
Official references
Frequently asked questions
Does the scanner open documents?
Not in ClearCopies’ MVP exact flow. It reads metadata returned by OneDrive.
Can metadata detect similar documents?
Not semantic or visual similarity. It identifies exact byte relationships when supported fingerprints are available.
Why export a plan?
It makes keep decisions, evidence, and exceptions reviewable before any write permission or action.
Scan first. Decide with evidence.
ClearCopies reads supported cloud-drive metadata and groups exact copies by byte size plus a provider-supplied content fingerprint. Original file bodies are not downloaded for the scan. You review the result and export a plan before any separate write step.