Deduplication File Handling

Core Problem

Nhiều user upload cùng 1 file (same content)
→ hệ thống lưu nhiều bản giống nhau
→ tốn storage + bandwidth

Goal:

Same file → chỉ lưu 1 lần

Core Idea

Dùng content-based deduplication (hash) để xác định file giống nhau

Basic Flow

Upload file
  ↓
Compute hash (SHA256)
  ↓
Check DB:
  ├── Exist → reuse
  └── Not exist → store file

Data Model

files

id
hash (UNIQUE)
storage_path
size

user_files

user_id
file_id

Nhiều user → cùng 1 file

🔄 Handling Flow (Recommended)

1. Upload file (temp storage / S3)
2. Compute hash
3. Try insert (UNIQUE hash)
   ├── Success → new file
   └── Fail → file đã tồn tại
4. Link user → file

Concurrency Problem

Race condition

User A upload
User B upload cùng lúc
→ cả 2 chưa thấy file
→ duplicate insert ❌

✅ Solution

1/ Unique constraint (MUST)

UNIQUE(hash)

⇒ DB đảm bảo chỉ có 1 bản

2/ Optimistic handling

Try insert
→ nếu fail (duplicate)
→ fetch existing file

3/ Redis lock (optional)

lock:file_hash

⇒ Giảm race nhưng không thay thế DB constraint

Optimization

1) Reference counting

files:
  ref_count

• +1 khi dùng
• -1 khi delete → ref_count = 0 → xóa file

2) Client-side hash (optional)

• Client tính hash trước
• Check API:

Nếu tồn tại → skip upload

3) Chunk upload (large file)

File → chunk1 + chunk2 + chunk3

⇒ Resume upload + optimize bandwidth

4) Chunk-level dedupe (advanced)

• Deduplicate từng chunk
• Dùng trong:

Pitfalls

❌ Check trước rồi upload

→ race condition

❌ Không unique constraint

→ duplicate data

Upload file lớn nhiều lần

→ waste bandwidth

Real-world

Google Drive / Dropbox

• File-level + chunk-level dedupe
• Global storage optimization

Fintech

• Statement / report trùng → reuse file

Cheat Sheet

Do

• Hash file (SHA256)
• Unique constraint
• Async upload + processing
• Reference mapping

Don’t

• Trust pre-check
• Store duplicate files
• Ignore race condition