office_translator/_bmad-output/implementation-artifacts/2-15-job-cleanup-fichiers-ttl-60-min.md

Story 2.15: Job Cleanup Fichiers (TTL 60 min)

Status: done

Story

As a system, I want to automatically delete temporary files after 60 minutes, so that disk space is managed and user data is not retained (RGPD compliance).

Acceptance Criteria

1. TTL Metadata: Files are stored with TTL metadata (already in place from Story 2.14)
2. Cleanup Interval: The cleanup job runs every 5 minutes (configurable via CLEANUP_INTERVAL_MINUTES)
3. Age-Based Deletion: All files older than 60 minutes are hard-deleted from:
   • config.UPLOAD_DIR (inputs)
   • config.OUTPUT_DIR (outputs)
   • config.TEMP_DIR (temporary files)
4. Orphaned File Detection: Files with no corresponding Redis/DB record are also deleted
5. Structured Logging: Cleanup is logged with: files_deleted, bytes_freed_mb, cleanup_run_timestamp
6. Resilience: Job continues even if individual file deletion fails (no crash on error)
7. Zero Retention: This ensures zero data retention (NFR15, NFR17)

Tasks / Subtasks

• Task 1: Update Cleanup Interval Configuration (AC: #2)

  • 1.1 Change default CLEANUP_INTERVAL_MINUTES from 15 to 5 in config.py
  • 1.2 Ensure environment variable override works: CLEANUP_INTERVAL_MINUTES=5

• Task 2: Add Orphaned File Detection (AC: #4)

  • 2.1 In FileCleanupManager.cleanup(), add logic to detect orphaned files
  • 2.2 An orphaned file = file exists on disk but no Redis key matches it
  • 2.3 Use StorageTracker or scan Redis keys with pattern translation:file:*
  • 2.4 If file path not found in any tracked metadata, mark as orphan
  • 2.5 Delete orphaned files regardless of age (they're already lost)

• Task 3: Enhance Structured Logging (AC: #5)

  • 3.1 Use structlog for cleanup logs (check if already integrated)
  • 3.2 Log format: {"event": "cleanup_completed", "files_deleted": N, "bytes_freed_mb": X.XX, "orphaned_deleted": N, "cleanup_run_timestamp": ISO8601}
  • 3.3 Log individual file deletion errors at warning level, not error (to avoid alert noise)

• Task 4: Verify TTL Configuration (AC: #3, #7)

  • 4.1 Ensure FILE_TTL_MINUTES = 60 is correctly used
  • 4.2 Verify max_file_age_seconds calculation uses config value

• Task 5: Add Unit & Integration Tests

  • 5.1 Unit test: orphaned file detection logic
  • 5.2 Integration test: files older than TTL are deleted
  • 5.3 Integration test: cleanup continues after individual failure
  • 5.4 Test: logging output format

Dev Notes

🔥 CRITICAL: Existing Implementation

IMPORTANT: A cleanup system already exists in middleware/cleanup.py. Your job is to ENHANCE it, not rebuild it.

Current State (FileCleanupManager):

• ✅ Periodic cleanup loop (_cleanup_loop) - runs every cleanup_interval
• ✅ Age-based deletion in cleanup() method
• ✅ TTL tracking via track_file() method
• ✅ Resilience: try/except around each file deletion
• ✅ Disk usage enforcement (_enforce_size_limit)
• ⚠️ Default interval is 10-15 minutes (needs to be 5)
• ⚠️ No explicit orphaned file detection

Previous Story Intelligence (2.14)

Story 2.14 implemented:

• StorageTracker service in services/storage_tracker.py - tracks files in Redis
• utils/file_handler.py - has cleanup_file() method
• routes/translate_routes.py - triggers metadata logging after file save
• Redis key pattern: translation:file:{job_id} with 60 min TTL

Architecture Compliance

Naming Conventions:

• Files: snake_case (e.g., cleanup.py)
• Classes: PascalCase (e.g., FileCleanupManager)
• Variables: snake_case (e.g., files_deleted, bytes_freed)

Logging Pattern:

# With structlog (preferred)
logger.info(
    "cleanup_completed",
    files_deleted=5,
    bytes_freed_mb=12.5,
    orphaned_deleted=2
)

# Without structlog (fallback)
logger.info(f"Cleanup completed: files_deleted=5, bytes_freed_mb=12.5")

Error Handling:

# NEVER crash on individual file error
try:
    filepath.unlink()
except Exception as e:
    logger.warning(f"Failed to delete {filepath}: {e}")
    # Continue to next file

Project Structure Notes

Files to modify:

• config.py - Update CLEANUP_INTERVAL_MINUTES default
• middleware/cleanup.py - Add orphan detection, enhance logging
• services/storage_tracker.py - May need method to list all tracked files

Test location:

• tests/test_cleanup.py (if exists) or create new

Implementation Approach

1. Start with config change - simplest, high impact
2. Add orphan detection - requires Redis integration
3. Enhance logging - easy win
4. Add tests - verify everything works

Orphaned File Detection Strategy

async def cleanup(self) -> dict:
    # ... existing age-based cleanup ...
    
    # NEW: Orphaned file detection
    redis_client = _get_async_redis()
    if redis_client:
        # Get all tracked file paths from Redis
        tracked_paths = set()
        keys = await redis_client.keys("translation:file:*")
        for key in keys:
            data = await redis_client.get(key)
            if data:
                metadata = json.loads(data)
                if "file_path" in metadata:
                    tracked_paths.add(metadata["file_path"])
    
    # Delete files not in Redis (orphans)
    for directory in [self.upload_dir, self.output_dir, self.temp_dir]:
        for filepath in directory.iterdir():
            if str(filepath) not in tracked_paths:
                # This is an orphan - delete it
                filepath.unlink()

References

• [Source: _bmad-output/planning-artifacts/epics.md#Story 2.15]
• [Source: _bmad-output/planning-artifacts/prd.md#NFR15, NFR17]
• [Source: _bmad-output/planning-artifacts/architecture.md#File Management]
• [Source: config.py#CLEANUP_ENABLED, CLEANUP_INTERVAL_MINUTES, FILE_TTL_MINUTES]
• [Source: middleware/cleanup.py#FileCleanupManager]

Dev Agent Record

Agent Model Used

{{agent_model_name_version}}

Completion Notes List

• ✅ Updated CLEANUP_INTERVAL_MINUTES to 5 in config.py and middleware/cleanup.py
• ✅ Implemented orphaned file detection using Redis scan in FileCleanupManager.cleanup()
• ✅ Added structured logging with structlog for cleanup runs
• ✅ Refactored FileCleanupManager to use max_file_age_minutes instead of hours for consistency
• ✅ Fixed a NameError in services/storage_tracker.py related to DEFAULT_TTL
• ✅ Verified with 5 tests in tests/test_cleanup.py (TTL, orphans, resilience, logging, config)

Code Review Fixes Applied

• 🔧 Fixed test imports to avoid middleware/init.py dependency (FastAPI)
• 🔧 Rewrote test_logging_format with actual assertions (was fake test)
• 🔧 Rewrote test_cleanup_resilience with proper resilience verification
• 🔧 Added test_cleanup_interval_env_override for AC#2
• 🔧 Added test_redis_unavailable_graceful for fallback behavior
• 🔧 Added MAX_TOTAL_SIZE_GB to config.py (was missing)
• 🔧 Added warning log when Redis unavailable for orphan detection
• 🔧 Fixed race condition in cleanup by collecting files before iteration

File List

• config.py
• middleware/cleanup.py
• services/storage_tracker.py
• tests/test_cleanup.py