124 lines
6.5 KiB
Markdown
124 lines
6.5 KiB
Markdown
---
|
|
stepsCompleted: [1, 2, 3, 4, 5]
|
|
inputDocuments: ['_bmad-output/planning-artifacts/prd.md', '_bmad-output/planning-artifacts/ux-design-specification.md']
|
|
workflowType: 'architecture'
|
|
project_name: 'Data_analysis'
|
|
user_name: 'Sepehr'
|
|
date: '2026-01-10'
|
|
---
|
|
|
|
# Architecture Decision Document
|
|
|
|
_This document builds collaboratively through step-by-step discovery. Sections are appended as we work through each architectural decision together._
|
|
|
|
## Project Context Analysis
|
|
|
|
### Requirements Overview
|
|
|
|
**Functional Requirements:**
|
|
The system requires a robust data processing pipeline capable of ingesting diverse file formats (Excel/CSV), performing automated statistical analysis (Outlier Detection, RFE), and rendering interactive visualizations. The frontend must support a high-performance, editable grid ("Smart Grid") that mimics spreadsheet behavior.
|
|
|
|
**Non-Functional Requirements:**
|
|
* **Performance:** Sub-second response times for grid interactions on datasets up to 50k rows.
|
|
* **Stateless Architecture:** Phase 1 requires no persistent user data storage; sessions are ephemeral.
|
|
* **Scientific Rigor:** Reproducibility of results is paramount, requiring strict versioning of libraries and random seeds.
|
|
* **Security:** Secure file handling and transport (TLS 1.3) are mandatory.
|
|
|
|
**Scale & Complexity:**
|
|
* **Primary Domain:** Scientific Web Application (Full-stack).
|
|
* **Complexity Level:** Medium. The complexity lies in the bridge between the interactive frontend and the computational backend, ensuring synchronization and performance.
|
|
* **Estimated Architectural Components:** ~5 Core Components (Frontend Shell, Data Grid, Visualization Engine, API Gateway, Computational Worker).
|
|
|
|
### Technical Constraints & Dependencies
|
|
|
|
* **Backend:** Python is mandatory for the scientific stack (Pandas, Scikit-learn, Statsmodels).
|
|
* **Frontend:** Next.js 16 with React Server Components (for shell) and Client Components (for grid).
|
|
* **UI Library:** Shadcn UI + TanStack Table (headless) + Recharts.
|
|
* **Deployment:** Must support containerized deployment (Docker) for reproducibility.
|
|
|
|
### Cross-Cutting Concerns Identified
|
|
|
|
* **Data Serialization:** Efficient transfer of large datasets (JSON/Arrow) between Python backend and React frontend.
|
|
* **State Management:** Synchronizing the client-side grid state with the server-side analysis context.
|
|
* **Error Handling:** Unifying error reporting from the Python backend to the React UI (e.g., "Singular Matrix" error).
|
|
|
|
## Starter Template Evaluation
|
|
|
|
### Primary Technology Domain
|
|
Scientific Data Application (Full-stack Next.js + FastAPI) optimized for self-hosting.
|
|
|
|
### Selected Starter: Custom FastAPI-Next.js-Docker Boilerplate
|
|
**Rationale for Selection:**
|
|
Explicitly chosen to support a "Two-Service" deployment model on a Homelab infrastructure. This ensures process isolation between the analytical Python engine and the React UI.
|
|
|
|
**Architectural Decisions Provided by Starter:**
|
|
* **Language & Runtime:** Python 3.12 (Backend managed by **uv**) and Node.js 20 (Frontend).
|
|
* **Styling Solution:** Tailwind CSS with Shadcn UI.
|
|
* **Testing:** Pytest (Backend) and Vitest (Frontend).
|
|
* **Code Organization:** Clean Monorepo with separated service directories.
|
|
|
|
**Deployment Strategy (Homelab):**
|
|
* **Frontend Service:** Next.js in Standalone mode (Docker).
|
|
* **Backend Service:** FastAPI with Uvicorn (Docker).
|
|
* **Communication:** Internal Docker network for API requests to minimize latency.
|
|
|
|
## Core Architectural Decisions
|
|
|
|
### Decision Priority Analysis
|
|
**Critical Decisions (Block Implementation):**
|
|
* **Data Serialization Protocol:** Apache Arrow (IPC Stream) is mandatory for performance.
|
|
* **State Management Strategy:** Hybrid (TanStack Query for Async + Zustand for UI State).
|
|
* **Container Strategy:** Docker Compose with isolated networks for Homelab deployment.
|
|
|
|
### Data Architecture
|
|
* **Format:** Apache Arrow (IPC Stream) for grid data; JSON for control plane.
|
|
* **Validation:** Pydantic (v2) for all JSON payloads.
|
|
* **Persistence:** None (Stateless) for Phase 1. `tempfile` module in Python for transient storage during analysis.
|
|
|
|
### API & Communication Patterns
|
|
* **Protocol:** REST API (FastAPI) with `StreamingResponse` for data export.
|
|
* **Serialization:** `pyarrow.ipc.new_stream` on backend -> `tableFromIPC` on frontend.
|
|
* **CORS:** Strictly configured to allow only the Homelab domain (e.g., `data.home`).
|
|
|
|
### Frontend Architecture
|
|
* **State Manager:**
|
|
* **Zustand (v5):** For high-frequency grid state (selection, edits).
|
|
* **TanStack Query (v5):** For analytical job status and data fetching.
|
|
* **Component Architecture:** "Smart Grid" pattern where the Grid component subscribes directly to the Zustand store to avoid re-rendering the entire page.
|
|
|
|
### Infrastructure & Deployment
|
|
* **Containerization:** Multi-stage Docker builds to keep images light (distroless/python and node-alpine).
|
|
* **Orchestration:** Docker Compose file defining `frontend`, `backend`, and a shared `network`.
|
|
|
|
## Implementation Patterns & Consistency Rules
|
|
|
|
### Pattern Categories Defined
|
|
**Critical Conflict Points Identified:** 5 major areas where AI agents must align to prevent implementation divergence.
|
|
|
|
### Naming Patterns
|
|
* **Backend (Python):** Strict `snake_case` for modules, functions, and variables (PEP 8).
|
|
* **Frontend (TSX):** `PascalCase` for Components (`SmartGrid.tsx`), `camelCase` for hooks and utilities.
|
|
* **API / JSON:** `snake_case` for all keys to maintain 1:1 mapping with Pandas DataFrame columns and Pydantic models.
|
|
|
|
### Structure Patterns
|
|
* **Project Organization:** Co-located logic. Features are grouped in folders: `/features/data-grid`, `/features/analysis-engine`.
|
|
* **Test Location:** Centralized `/tests` directory at the service root (e.g., `backend/tests/`, `frontend/tests/`) to simplify Docker test runs.
|
|
|
|
### Format Patterns
|
|
* **API Response Wrapper:**
|
|
* Success: `{ "status": "success", "data": ..., "metadata": {...} }`.
|
|
* Error: `{ "status": "error", "message": "User-friendly message", "code": "TECHNICAL_ERROR_CODE" }`.
|
|
* **Date Format:** ISO 8601 strings (`YYYY-MM-DDTHH:mm:ssZ`) in UTC.
|
|
|
|
### Process Patterns
|
|
* **Loading States:** Standardized `isLoading` and `isProcessing` flags in Zustand/TanStack Query.
|
|
* **Validation:**
|
|
* Backend: Pydantic v2.
|
|
* Frontend: Zod (synchronized with Pydantic via OpenAPI generator).
|
|
|
|
### Enforcement Guidelines
|
|
**All AI Agents MUST:**
|
|
1. Check for existing Pydantic models before creating new ones.
|
|
2. Use the `logger` utility instead of `print()` or `console.log`.
|
|
3. Add JSDoc/Docstrings to every exported function.
|