sepehr/data_analysis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
sepehr
2026-01-11 22:04:05 +01:00
commit 87a8b6b844
549 changed files with 96211 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
_bmad-output/planning-artifacts/architecture.md Normal file
View File

@@ -0,0 +1,123 @@
---
stepsCompleted: [1, 2, 3, 4, 5]
inputDocuments: ['_bmad-output/planning-artifacts/prd.md', '_bmad-output/planning-artifacts/ux-design-specification.md']
workflowType: 'architecture'
project_name: 'Data_analysis'
user_name: 'Sepehr'
date: '2026-01-10'
---
# Architecture Decision Document
_This document builds collaboratively through step-by-step discovery. Sections are appended as we work through each architectural decision together._
## Project Context Analysis
### Requirements Overview
**Functional Requirements:**
The system requires a robust data processing pipeline capable of ingesting diverse file formats (Excel/CSV), performing automated statistical analysis (Outlier Detection, RFE), and rendering interactive visualizations. The frontend must support a high-performance, editable grid ("Smart Grid") that mimics spreadsheet behavior.
**Non-Functional Requirements:**
* **Performance:** Sub-second response times for grid interactions on datasets up to 50k rows.
* **Stateless Architecture:** Phase 1 requires no persistent user data storage; sessions are ephemeral.
* **Scientific Rigor:** Reproducibility of results is paramount, requiring strict versioning of libraries and random seeds.
* **Security:** Secure file handling and transport (TLS 1.3) are mandatory.
**Scale & Complexity:**
* **Primary Domain:** Scientific Web Application (Full-stack).
* **Complexity Level:** Medium. The complexity lies in the bridge between the interactive frontend and the computational backend, ensuring synchronization and performance.
* **Estimated Architectural Components:** ~5 Core Components (Frontend Shell, Data Grid, Visualization Engine, API Gateway, Computational Worker).
### Technical Constraints & Dependencies
* **Backend:** Python is mandatory for the scientific stack (Pandas, Scikit-learn, Statsmodels).
* **Frontend:** Next.js 16 with React Server Components (for shell) and Client Components (for grid).
* **UI Library:** Shadcn UI + TanStack Table (headless) + Recharts.
* **Deployment:** Must support containerized deployment (Docker) for reproducibility.
### Cross-Cutting Concerns Identified
* **Data Serialization:** Efficient transfer of large datasets (JSON/Arrow) between Python backend and React frontend.
* **State Management:** Synchronizing the client-side grid state with the server-side analysis context.
* **Error Handling:** Unifying error reporting from the Python backend to the React UI (e.g., "Singular Matrix" error).
## Starter Template Evaluation
### Primary Technology Domain
Scientific Data Application (Full-stack Next.js + FastAPI) optimized for self-hosting.
### Selected Starter: Custom FastAPI-Next.js-Docker Boilerplate
**Rationale for Selection:**
Explicitly chosen to support a "Two-Service" deployment model on a Homelab infrastructure. This ensures process isolation between the analytical Python engine and the React UI.
**Architectural Decisions Provided by Starter:**
* **Language & Runtime:** Python 3.12 (Backend managed by **uv**) and Node.js 20 (Frontend).
* **Styling Solution:** Tailwind CSS with Shadcn UI.
* **Testing:** Pytest (Backend) and Vitest (Frontend).
* **Code Organization:** Clean Monorepo with separated service directories.
**Deployment Strategy (Homelab):**
* **Frontend Service:** Next.js in Standalone mode (Docker).
* **Backend Service:** FastAPI with Uvicorn (Docker).
* **Communication:** Internal Docker network for API requests to minimize latency.
## Core Architectural Decisions
### Decision Priority Analysis
**Critical Decisions (Block Implementation):**
* **Data Serialization Protocol:** Apache Arrow (IPC Stream) is mandatory for performance.
* **State Management Strategy:** Hybrid (TanStack Query for Async + Zustand for UI State).
* **Container Strategy:** Docker Compose with isolated networks for Homelab deployment.
### Data Architecture
* **Format:** Apache Arrow (IPC Stream) for grid data; JSON for control plane.
* **Validation:** Pydantic (v2) for all JSON payloads.
* **Persistence:** None (Stateless) for Phase 1. `tempfile` module in Python for transient storage during analysis.
### API & Communication Patterns
* **Protocol:** REST API (FastAPI) with `StreamingResponse` for data export.
* **Serialization:** `pyarrow.ipc.new_stream` on backend -> `tableFromIPC` on frontend.
* **CORS:** Strictly configured to allow only the Homelab domain (e.g., `data.home`).
### Frontend Architecture
* **State Manager:**
* **Zustand (v5):** For high-frequency grid state (selection, edits).
* **TanStack Query (v5):** For analytical job status and data fetching.
* **Component Architecture:** "Smart Grid" pattern where the Grid component subscribes directly to the Zustand store to avoid re-rendering the entire page.
### Infrastructure & Deployment
* **Containerization:** Multi-stage Docker builds to keep images light (distroless/python and node-alpine).
* **Orchestration:** Docker Compose file defining `frontend`, `backend`, and a shared `network`.
## Implementation Patterns & Consistency Rules
### Pattern Categories Defined
**Critical Conflict Points Identified:** 5 major areas where AI agents must align to prevent implementation divergence.
### Naming Patterns
* **Backend (Python):** Strict `snake_case` for modules, functions, and variables (PEP 8).
* **Frontend (TSX):** `PascalCase` for Components (`SmartGrid.tsx`), `camelCase` for hooks and utilities.
* **API / JSON:** `snake_case` for all keys to maintain 1:1 mapping with Pandas DataFrame columns and Pydantic models.
### Structure Patterns
* **Project Organization:** Co-located logic. Features are grouped in folders: `/features/data-grid`, `/features/analysis-engine`.
* **Test Location:** Centralized `/tests` directory at the service root (e.g., `backend/tests/`, `frontend/tests/`) to simplify Docker test runs.
### Format Patterns
* **API Response Wrapper:**
* Success: `{ "status": "success", "data": ..., "metadata": {...} }`.
* Error: `{ "status": "error", "message": "User-friendly message", "code": "TECHNICAL_ERROR_CODE" }`.
* **Date Format:** ISO 8601 strings (`YYYY-MM-DDTHH:mm:ssZ`) in UTC.
### Process Patterns
* **Loading States:** Standardized `isLoading` and `isProcessing` flags in Zustand/TanStack Query.
* **Validation:**
* Backend: Pydantic v2.
* Frontend: Zod (synchronized with Pydantic via OpenAPI generator).
### Enforcement Guidelines
**All AI Agents MUST:**
1. Check for existing Pydantic models before creating new ones.
2. Use the `logger` utility instead of `print()` or `console.log`.
3. Add JSDoc/Docstrings to every exported function.

35
_bmad-output/planning-artifacts/bmm-workflow-status.yaml Normal file
View File

@@ -0,0 +1,35 @@
generated: "2026-01-10"
project: "Data_analysis"
project_type: "software"
selected_track: "method"
field_type: "greenfield"
workflow_path: "_bmad/bmm/workflows/workflow-status/paths/method-greenfield.yaml"
workflow_status:
- id: "prd"
status: "_bmad-output/planning-artifacts/prd.md"
agent: "pm"
command: "/bmad:bmm:workflows:create-prd"
- id: "create-ux-design"
status: "_bmad-output/planning-artifacts/ux-design-specification.md"
agent: "ux-designer"
command: "/bmad:bmm:workflows:create-ux-design"
- id: "create-architecture"
status: "_bmad-output/planning-artifacts/architecture.md"
agent: "architect"
command: "/bmad:bmm:workflows:create-architecture"
- id: "create-epics-and-stories"
status: "_bmad-output/planning-artifacts/epics.md"
agent: "pm"
command: "/bmad:bmm:workflows:create-epics-and-stories"
- id: "test-design"
status: "optional"
agent: "tea"
command: "/bmad:bmm:workflows:test-design"
- id: "implementation-readiness"
status: "_bmad-output/planning-artifacts/implementation-readiness-report-2026-01-10.md"
agent: "architect"
command: "/bmad:bmm:workflows:implementation-readiness"
- id: "sprint-planning"
status: "required"
agent: "sm"
command: "/bmad:bmm:workflows:sprint-planning"

312
_bmad-output/planning-artifacts/epics.md Normal file
View File

@@ -0,0 +1,312 @@
---
stepsCompleted: [1, 2, 3]
inputDocuments: ['_bmad-output/planning-artifacts/prd.md', '_bmad-output/planning-artifacts/architecture.md', '_bmad-output/planning-artifacts/ux-design-specification.md']
---
# Data_analysis - Epic Breakdown
## Overview
This document provides the complete epic and story breakdown for Data_analysis, decomposing the requirements from the PRD, UX Design if it exists, and Architecture requirements into implementable stories.
## Requirements Inventory
### Functional Requirements
- **FR1:** Users can upload datasets in .xlsx, .xls, and .csv formats via drag-and-drop or file selection.
- **FR2:** System automatically detects column data types (numeric, categorical, datetime) upon ingest.
- **FR3:** Users can manually override detected data types if the inference is incorrect.
- **FR4:** Users can rename columns directly in the interface to sanitize inputs.
- **FR5:** Users can view loaded data in a paginated, virtualized grid capable of displaying 50,000+ rows.
- **FR6:** Users can edit cell values directly (double-click to edit) with inputs validated against the column type.
- **FR7:** Users can sort columns (asc/desc) and filter rows based on values/conditions (e.g., "> 100").
- **FR8:** Users can perform Undo/Redo operations (Ctrl+Z/Ctrl+Y) on data edits within the current session.
- **FR9:** Users can exclude specific rows from analysis without deleting them (soft delete/toggle).
- **FR10:** System automatically identifies univariate outliers using IQR/Z-score and visualizes them in the grid/plots.
- **FR11:** System automatically identifies multivariate outliers using Isolation Forest upon user request.
- **FR12:** Users can accept or reject outlier exclusion proposals individually or in bulk.
- **FR13:** Users can select a Target Variable (Y) to trigger an automated Feature Importance analysis.
- **FR14:** System recommends the Top-N predictive features based on RFE (Recursive Feature Elimination) or Random Forest importance.
- **FR15:** Users can configure a Linear Regression (Simple/Multiple) by selecting Dependent (Y) and Independent (X) variables.
- **FR16:** Users can configure a Binary Logistic Regression for categorical target variables.
- **FR17:** System generates a "Model Summary" including R-squared, Adjusted R-squared, F-statistic, and P-values for coefficients.
- **FR18:** System generates standard diagnostic plots: Residuals vs Fitted, Q-Q Plot, and Scale-Location.
- **FR19:** Users can view a Correlation Matrix (Heatmap) for selected numeric variables.
- **FR20:** Users can view an interactive "Analysis Report" dashboard summarizing data health, methodology, and model results.
- **FR21:** Users can export the full report as a branded PDF document.
- **FR22:** System appends an "Audit Trail" to the report listing library versions, random seeds, and data exclusion steps for reproducibility.
### NonFunctional Requirements
- **Performance:** Grid latency < 200ms for 50k rows. Analysis throughput < 15s. Upload speed < 3s for 5MB.
- **Security:** Data ephemerality (purge after 1h). TLS 1.3 encryption. Input sanitization for files.
- **Reliability:** Graceful degradation for bad data. Support 50 concurrent requests via async task queue.
- **Accessibility:** Keyboard navigation for "Smart Grid". Screen reader support. WCAG 2.1 Level AA compliance.
### Additional Requirements
**Architecture:**
- **Starter Template:** Custom FastAPI-Next.js-Docker Boilerplate.
- **Data Serialization:** Apache Arrow (IPC Stream) required for grid data.
- **State Management:** Hybrid approach (TanStack Query for Server State + Zustand for Grid UI State).
- **Deployment:** "Two-Service" model on Homelab via Docker Compose.
- **Naming Conventions:** `snake_case` for Python/API, `PascalCase` for React components.
- **Testing:** Pytest (Backend) and Vitest (Frontend).
**UX Design:**
- **Visual Style:** "Lab & Tech" (Slate/Indigo/Mono) with Shadcn UI.
- **Responsive:** Desktop Only (1366px+).
- **Core Interaction:** "Guided Data Hygiene Loop" (Insight Panel).
- **Design System:** TanStack Table for virtualization + Recharts for visualization.
- **Mode:** Native Dark Mode support.
### FR Coverage Map
- **FR1:** Epic 1 - Data Ingestion
- **FR2:** Epic 1 - Type Auto-detection
- **FR3:** Epic 1 - Manual Type Override
- **FR4:** Epic 1 - Column Renaming
- **FR5:** Epic 1 - High-Performance Grid View
- **FR6:** Epic 2 - Grid Cell Editing
- **FR7:** Epic 1 - Grid Sort/Filter
- **FR8:** Epic 2 - Edit Undo/Redo
- **FR9:** Epic 2 - Row Exclusion Logic
- **FR10:** Epic 2 - Univariate Outlier Detection
- **FR11:** Epic 2 - Multivariate Outlier Detection
- **FR12:** Epic 2 - Outlier Review UI (Insight Panel)
- **FR13:** Epic 3 - Feature Importance Engine
- **FR14:** Epic 3 - Smart Feature Recommendation
- **FR15:** Epic 4 - Linear Regression Configuration
- **FR16:** Epic 4 - Logistic Regression Configuration
- **FR17:** Epic 4 - Model Summary & Metrics
- **FR18:** Epic 4 - Diagnostic Plots
- **FR19:** Epic 3 - Correlation Matrix Visualization
- **FR20:** Epic 4 - Interactive Analysis Dashboard
- **FR21:** Epic 4 - PDF Export
- **FR22:** Epic 4 - Reproducibility Audit Trail
## Epic List
### Epic 1: Fondation & Ingestion de Données
"Je peux uploader mon fichier Excel et voir mes données brutes dans une grille fluide."
**FRs covered:** FR1, FR2, FR3, FR4, FR5, FR7.
### Epic 2: Nettoyage Interactif (Hygiene Loop)
"Je peux nettoyer mes données en supprimant les lignes erronées ou les outliers."
**FRs covered:** FR6, FR8, FR9, FR10, FR11, FR12.
### Epic 3: Intelligence & Sélection (Smart Prep)
"Le système me dit quelles variables sont importantes pour ma cible."
**FRs covered:** FR13, FR14, FR19.
### Epic 4: Modélisation & Reporting
"Je génère mon modèle de régression et j'exporte le rapport PDF."
**FRs covered:** FR15, FR16, FR17, FR18, FR20, FR21, FR22.
---
## Epic 1: Fondation & Ingestion de Données
"Je peux uploader mon fichier Excel et voir mes données brutes dans une grille fluide."
### Story 1.1: Initialisation du Monorepo & Docker
As a developer,
I want to initialize the project structure (Next.js + FastAPI + Docker),
So that I have a functional and consistent development environment.
**Acceptance Criteria:**
**Given** A fresh project directory.
**When** I run `docker-compose up`.
**Then** Both the Next.js frontend and FastAPI backend are reachable on their respective ports.
**And** The shared Docker network allows communication between services.
### Story 1.2: Ingestion de Fichiers Excel/CSV (Backend)
As a Julien (Analyst),
I want to upload an Excel or CSV file,
So that the system can read my production data.
**Acceptance Criteria:**
**Given** A valid `.xlsx` file with multiple columns and 5,000 rows.
**When** I POST the file to the `/upload` endpoint.
**Then** The backend returns a 200 OK with column metadata (names, detected types).
**And** The data is prepared as an Apache Arrow stream for high-performance delivery.
### Story 1.3: Visualisation dans la Smart Grid (Frontend)
As a Julien (Analyst),
I want to see my uploaded data in an interactive high-speed grid,
So that I can explore the raw data effortlessly.
**Acceptance Criteria:**
**Given** A dataset successfully loaded in the backend.
**When** I view the workspace page.
**Then** The TanStack Table renders the data using virtualization.
**And** Scrolling through 50,000 rows remains fluid (< 200ms latency).
### Story 1.4: Gestion des Types & Renommage (Data Hygiene)
As a Julien (Analyst),
I want to rename columns and correct data types,
So that the data matches my business context before analysis.
**Acceptance Criteria:**
**Given** A column "Press_01" detected as 'text'.
**When** I click the column header to rename it to "Pressure" and change type to 'numeric'.
**Then** The grid updates the visual formatting immediately.
**And** The backend validates that all values in the column can be cast to numeric.
### Story 1.5: Tri & Filtrage de Base
As a Julien (Analyst),
I want to sort and filter my data in the grid,
So that I can identify extreme values or specific subsets.
**Acceptance Criteria:**
**Given** A column "Temperature".
**When** I click 'Sort Descending'.
**Then** The highest temperature values appear at the top of the grid instantly.
---
## Epic 2: Nettoyage Interactif (Hygiene Loop)
"Je peux nettoyer mes données en supprimant les lignes erronées ou les outliers."
### Story 2.1: Édition de Cellule & Validation
As a Julien (Analyst),
I want to edit cell values directly in the grid,
So that I can manually correct obvious data entry errors.
**Acceptance Criteria:**
**Given** A data cell in the grid.
**When** I double-click the cell and enter a new value.
**Then** The value is updated in the local UI state (Zustand).
**And** The system validates the input against the column's data type (e.g., no text in numeric columns).
### Story 2.2: Undo/Redo des Modifications
As a Julien (Analyst),
I want to undo my last data edits,
So that I can explore changes without fear of losing the original data.
**Acceptance Criteria:**
**Given** A cell value was modified.
**When** I press `Ctrl+Z` (or click Undo).
**Then** The cell reverts to its previous value.
**And** `Ctrl+Y` (Redo) restores the edit.
### Story 2.3: Détection Automatique des Outliers (Backend)
As a system,
I want to identify statistical outliers in the background,
So that I can alert the user to potential data quality issues.
**Acceptance Criteria:**
**Given** A dataset is loaded.
**When** The analysis engine runs.
**Then** It uses Isolation Forest (multivariate) and IQR (univariate) to tag suspicious rows.
**And** Outlier coordinates are returned to the frontend.
### Story 2.4: Panel d'Insights & Revue des Outliers (Frontend)
As a Julien (Analyst),
I want to review detected outliers in a side panel,
So that I can understand why they are flagged before excluding them.
**Acceptance Criteria:**
**Given** Flagged outliers exist.
**When** I click the warning icon in a column header.
**Then** The `InsightPanel` opens with a boxplot visualization and a "Why?" explanation.
**And** A button "Exclude all 34 outliers" is prominently displayed.
### Story 2.5: Exclusion Non-Destructive de Données
As a Julien (Analyst),
I want to toggle the inclusion of specific rows in the analysis,
So that I can test different scenarios without deleting data.
**Acceptance Criteria:**
**Given** A flagged outlier row.
**When** I click "Exclude".
**Then** The row appears with 40% opacity in the grid.
**And** The row is ignored by all subsequent statistical calculations (R², Regression).
---
## Epic 3: Intelligence & Sélection (Smart Prep)
"Le système me dit quelles variables sont importantes pour ma cible."
### Story 3.1: Matrice de Corrélation Interactive
As a Julien (Analyst),
I want to see a visual correlation map of my numeric variables,
So that I can quickly identify which factors are related.
**Acceptance Criteria:**
**Given** A dataset with multiple numeric columns.
**When** I navigate to the "Correlations" tab.
**Then** A heatmap is displayed using Pearson correlation coefficients.
**And** Hovering over a cell shows the precise correlation value.
### Story 3.2: Calcul de l'Importance des Features (Backend)
As a system,
I want to compute the predictive power of features against a target variable,
So that I can provide scientific recommendations to the user.
**Acceptance Criteria:**
**Given** A dataset and a selected Target Variable (Y).
**When** The RFE (Recursive Feature Elimination) algorithm runs.
**Then** The backend returns an ordered list of features with their importance scores.
### Story 3.3: Recommandation Intelligente de Variables (Frontend)
As a Julien (Analyst),
I want the system to suggest which variables to include in my model,
So that I don't pollute my analysis with irrelevant data ("noise").
**Acceptance Criteria:**
**Given** Feature importance scores are calculated.
**When** I open the Model Configuration panel.
**Then** The top 5 predictive variables are pre-selected by default.
**And** An explanation "Why?" is available for each recommendation.
---
## Epic 4: Modélisation & Reporting
"Je génère mon modèle de régression et j'exporte le rapport PDF."
### Story 4.1: Configuration de la Régression
As a Julien (Analyst),
I want to configure the parameters of my regression model,
So that I can tailor the analysis to my specific hypothesis.
**Acceptance Criteria:**
**Given** A cleaned dataset.
**When** I select "Linear Regression" and confirm X/Y variables.
**Then** The system validates that the target variable (Y) is suitable for the chosen model type.
### Story 4.2: Exécution du Modèle (Backend)
As a system,
I want to execute the statistical model computation,
So that I can provide accurate regression results.
**Acceptance Criteria:**
**Given** Model parameters (X, Y, Algorithm).
**When** The "Run" action is triggered.
**Then** The backend computes R², Adjusted R², P-values, and coefficients using `statsmodels`.
**And** All results are returned as a JSON summary.
### Story 4.3: Dashboard de Résultats Interactif
As a Julien (Analyst),
I want to see the model results through interactive charts,
So that I can easily diagnose the performance of my regression.
**Acceptance Criteria:**
**Given** Computed model results.
**When** I view the "Results" page.
**Then** I see a "Real vs Predicted" scatter plot and a "Residuals" plot.
**And** Key metrics (R², P-value) are displayed with colored status indicators (Success/Warning).
### Story 4.4: Génération du Rapport PDF (Audit Trail)
As a Julien (Analyst),
I want to export my findings as a professional PDF report,
So that I can share and archive my validated analysis.
**Acceptance Criteria:**
**Given** A completed analysis session.
**When** I click "Export PDF".
**Then** A PDF is generated containing all charts, metrics, and a reproducibility section (lib versions, seeds).
**And** The report lists all rows that were excluded during the session.

154
_bmad-output/planning-artifacts/implementation-readiness-report-2026-01-10.md Normal file
View File

@@ -0,0 +1,154 @@
# Implementation Readiness Assessment Report
**Date:** 2026-01-10
**Project:** Data_analysis
## PRD Analysis
### Functional Requirements
FR1: Users can upload datasets in .xlsx, .xls, and .csv formats via drag-and-drop or file selection.
FR2: System automatically detects column data types (numeric, categorical, datetime) upon ingest.
FR3: Users can manually override detected data types if the inference is incorrect.
FR4: Users can rename columns directly in the interface to sanitize inputs.
FR5: Users can view loaded data in a paginated, virtualized grid capable of displaying 50,000+ rows.
FR6: Users can edit cell values directly (double-click to edit) with inputs validated against the column type.
FR7: Users can sort columns (asc/desc) and filter rows based on values/conditions (e.g., "> 100").
FR8: Users can perform Undo/Redo operations (Ctrl+Z/Ctrl+Y) on data edits within the current session.
FR9: Users can exclude specific rows from analysis without deleting them (soft delete/toggle).
FR10: System automatically identifies univariate outliers using IQR/Z-score and visualizes them in the grid/plots.
FR11: System automatically identifies multivariate outliers using Isolation Forest upon user request.
FR12: Users can accept or reject outlier exclusion proposals individually or in bulk.
FR13: Users can select a Target Variable (Y) to trigger an automated Feature Importance analysis.
FR14: System recommends the Top-N predictive features based on RFE (Recursive Feature Elimination) or Random Forest importance.
FR15: Users can configure a Linear Regression (Simple/Multiple) by selecting Dependent (Y) and Independent (X) variables.
FR16: Users can configure a Binary Logistic Regression for categorical target variables.
FR17: System generates a "Model Summary" including R-squared, Adjusted R-squared, F-statistic, and P-values for coefficients.
FR18: System generates standard diagnostic plots: Residuals vs Fitted, Q-Q Plot, and Scale-Location.
FR19: Users can view a Correlation Matrix (Heatmap) for selected numeric variables.
FR20: Users can view an interactive "Analysis Report" dashboard summarizing data health, methodology, and model results.
FR21: Users can export the full report as a branded PDF document.
FR22: System appends an "Audit Trail" to the report listing library versions, random seeds, and data exclusion steps for reproducibility.
Total FRs: 22
### Non-Functional Requirements
NFR1: Grid Latency: render 50,000 rows with filtering/sorting response times under 200ms.
NFR2: Analysis Throughput: Automated analysis on standard datasets (<10MB) must complete in under 15 seconds.
NFR3: Upload Speed: Parsing and validation of a 5MB Excel file should complete in under 3 seconds.
NFR4: Data Ephemerality: All user datasets purged after 1 hour of inactivity or session termination.
NFR5: Transport Security: Data transmission must be encrypted via TLS 1.3.
NFR6: Input Sanitization: File parser must validate MIME types and signatures to prevent macro execution.
NFR7: Graceful Degradation: Handle NaNs/infinite values with clear error messages instead of crashing.
NFR8: Concurrency: Support at least 50 concurrent analysis requests using an asynchronous task queue.
NFR9: Keyboard Navigation: Data grid must be fully navigable via keyboard.
Total NFRs: 9
### Additional Requirements
- **Stateless Architecture:** Phase 1 requires no persistent user data storage.
- **Scientific Rigor:** Reproducibility of results is paramount (Trace d'Analyse).
- **Desktop Only:** Strictly optimized for high-resolution desktop displays.
### PRD Completeness Assessment
The PRD is exceptionally comprehensive, providing numbered, testable requirements (FR1-FR22) and specific, measurable quality attributes (NFR1-NFR9). The "Experience MVP" strategy is clearly defined, and the project context (Scientific Greenfield) is well-articulated. No major gaps were identified during extraction.
## Epic Coverage Validation
### FR Coverage Analysis
| FR Number | PRD Requirement | Epic Coverage | Status |
| :--- | :--- | :--- | :--- |
| FR1 | Upload datasets (.xlsx, .xls, .csv) | Epic 1 Story 1.2 | ✓ Covered |
| FR2 | Auto-detect column data types | Epic 1 Story 1.2 | ✓ Covered |
| FR3 | Manual type override | Epic 1 Story 1.4 | ✓ Covered |
| FR4 | Rename columns | Epic 1 Story 1.4 | ✓ Covered |
| FR5 | High-performance grid (50k+ rows) | Epic 1 Story 1.3 | ✓ Covered |
| FR6 | Edit cell values directly | Epic 2 Story 2.1 | ✓ Covered |
| FR7 | Sort and filter rows | Epic 1 Story 1.5 | ✓ Covered |
| FR8 | Undo/Redo operations | Epic 2 Story 2.2 | ✓ Covered |
| FR9 | Exclude rows (soft delete) | Epic 2 Story 2.5 | ✓ Covered |
| FR10 | Univariate outlier detection (IQR) | Epic 2 Story 2.3 | ✓ Covered |
| FR11 | Multivariate outlier detection (Isolation Forest) | Epic 2 Story 2.3 | ✓ Covered |
| FR12 | Outlier review UI (Insight Panel) | Epic 2 Story 2.4 | ✓ Covered |
| FR13 | Feature Importance analysis | Epic 3 Story 3.2 | ✓ Covered |
| FR14 | Top-N predictive feature recommendations | Epic 3 Story 3.3 | ✓ Covered |
| FR15 | Linear Regression configuration | Epic 4 Story 4.1 | ✓ Covered |
| FR16 | Logistic Regression configuration | Epic 4 Story 4.1 | ✓ Covered |
| FR17 | Model Summary (R², P-values, etc.) | Epic 4 Story 4.2 | ✓ Covered |
| FR18 | Diagnostic plots | Epic 4 Story 4.3 | ✓ Covered |
| FR19 | Correlation Matrix (Heatmap) | Epic 3 Story 3.1 | ✓ Covered |
| FR20 | Analysis Report dashboard | Epic 4 Story 4.3 | ✓ Covered |
| FR21 | Export branded PDF | Epic 4 Story 4.4 | ✓ Covered |
| FR22 | Reproducibility Audit Trail | Epic 4 Story 4.4 | ✓ Covered |
### Missing Requirements
None. All 22 Functional Requirements from the PRD are mapped to specific stories in the epics document.
### Coverage Statistics
- Total PRD FRs: 22
- FRs covered in epics: 22
- Coverage percentage: 100%
## UX Alignment Assessment
### UX Document Status
* **Found:** `_bmad-output/planning-artifacts/ux-design-specification.md`
### Alignment Analysis
**UX ↔ PRD Alignment:**
* ✅ **User Journeys:** Optimized for identified personas (Julien & Marc).
* ✅ **Feature Coverage:** 100% of FRs have defined interaction patterns.
* ✅ **Workflow:** Assisted analysis loop matches the PRD vision.
**UX ↔ Architecture Alignment:**
* ✅ **Performance:** High-density grid requirements supported by Apache Arrow stack.
* ✅ **State Management:** Zustand choice supports high-frequency UI updates.
* ✅ **Responsive Strategy:** Consistent "Desktop Only" approach across all plans.
### Warnings
* None.
## Epic Quality Review
### Epic Structure Validation
* ✅ **Epic 1: Ingestion** - Focused on user value.
* ✅ **Epic 2: Hygiene** - Standalone value, no forward dependencies.
* ✅ **Epic 3: Smart Prep** - Incremental enhancement.
* ✅ **Epic 4: Modélisation** - Final completion of journey.
### Story Quality & Sizing
* ✅ **Story 1.1:** Correctly initializes project from Architecture boilerplate.
* ✅ **Acceptance Criteria:** All stories follow Given/When/Then format.
* ✅ **Story Sizing:** Optimized for single agent dev sessions.
### Dependency Analysis
* ✅ **No Forward Dependencies:** No story depends on work from a future epic.
* ✅ **Database Timing:** Stateless logic introduced exactly when required.
### Quality Assessment Documentation
* 🔴 **Critical Violations:** None.
* 🟠 **Major Issues:** None.
* 🟡 **Minor Concerns:** None.
## Summary and Recommendations
### Overall Readiness Status
**READY** ✅
### Critical Issues Requiring Immediate Action
* **None.**
### Recommended Next Steps
1. **Initialize Project:** Run `docker-compose up` to verify the monorepo skeleton (Epic 1 Story 1.1).
2. **Performance Spike:** Validate Apache Arrow streaming with a 50k row dataset early in development.
3. **UI Setup:** Configure the Shadcn UI ThemeProvider for native Dark Mode support from the start.
### Final Note
This assessment identifies 0 issues. The project planning is complete, coherent, and highly robust. You may proceed immediately to implementation.

303
_bmad-output/planning-artifacts/prd.md Normal file
View File

@@ -0,0 +1,303 @@
---
stepsCompleted: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
inputDocuments: []
workflowType: 'prd'
lastStep: 11
project_name: Data_analysis
user_name: Sepehr
date: 2026-01-10
briefCount: 0
researchCount: 0
brainstormingCount: 0
projectDocsCount: 0
---
# Product Requirements Document - Data_analysis
**Author:** Sepehr
**Date:** 2026-01-10
## Executive Summary
**Data_analysis** aims to democratize advanced statistical analysis by combining the robustness of Python's scientific ecosystem with the accessibility of a modern web interface. It serves as a web-based, modern alternative to Minitab, specifically optimized for regression analysis workflows. The platform empowers users—from data novices to analysts—to upload datasets (Excel/CSV), intuitively clean and manipulate data in an Excel-like grid, and perform sophisticated regression modeling with automated guidance.
### What Makes This Special
* **Guided Analytical Workflow:** Unlike traditional tools that present a toolbox, Data_analysis guides the user through a best-practice workflow: Data Upload -> Auto-Outlier Detection -> Smart Feature Selection -> Regression Modeling -> Explainable Results.
* **Hybrid Interface:** Merges the familiarity of spreadsheet editing (direct cell manipulation, copy-paste) with the power of a computational notebook, eliminating the need to switch between Excel and statistical software.
* **Modern Tech Stack:** Built on a robust Python backend (FastAPI/Django) for heavy statistical lifting (Pandas, Scikit-learn, Statsmodels) and a high-performance Next.js 16 frontend with Tailwind CSS and Shadcn UI, ensuring a fast, responsive, and visually appealing experience.
* **Automated Insights:** Proactively identifies data quality issues (outliers) and relevant predictors (feature selection) using advanced algorithms (e.g., Isolation Forest, Recursive Feature Elimination), visualizing *why* certain features are selected before the user commits to a model.
### Key Workflows & Scenarios
1. **"Auto-Optimisation" (Scénario Cœur):**
* Upload de fichier -> Détection automatique des Outliers (Isolation Forest) avec proposition de correction.
* Sélection automatique de Features (RFE/Lasso) pour identifier les variables clés.
* Régression Robuste finale pour un modèle moins sensible au bruit.
2. **Modern Minitab Classics:**
* **Régression Linéaire Simple & Multiple:** Interface interactive pour visualiser la droite de régression, les résidus et les métriques (R²) avec alertes automatiques sur les hypothèses.
* **Régression Logistique Binaire:** Pour les prédictions Oui/Non avec matrices de confusion et courbes ROC interactives.
3. **Comparaison de Modèles (Benchmark):**
* Lancement parallèle de plusieurs algorithmes (Régression Linéaire, Random Forest, XGBoost) pour recommander le plus performant.
## Project Classification
**Technical Type:** web_app
**Domain:** scientific
**Complexity:** medium
**Project Context:** Greenfield - new project
## Success Criteria
### User Success
* **Efficiency:** Complete a full regression cycle (Upload -> Cleaning -> Modeling) in under 3 minutes for typical datasets.
* **Cognitive Load Reduction:** Users feel guided and confident without needing deep statistical expertise; "Explainable AI" visuals clarify outlier removal and feature selection.
* **Data Mastery:** Users gain comprehensive insights into their data quality and variable importance through automated profiling.
### Business Success
* **Operational Speed:** Significant reduction in man-hours spent on manual data cleaning and repetitive modeling tasks.
* **Platform Adoption:** High user retention rate by providing a "least head-scratching" experience compared to Minitab or raw Excel.
### Technical Success
* **High-Performance Grid:** Excel-like interface handles 50k+ rows with sub-second latency for filtering and sorting.
* **Algorithmic Integrity:** Reliable Python backend providing accurate statistical outputs consistent with industry-standard libraries (Scikit-learn, Statsmodels).
## Product Scope
### MVP - Minimum Viable Product
* **Direct Data Import:** Robust Excel/CSV upload.
* **Smart Data Grid:** Seamless cell editing, filtering, and sorting in a modern web UI.
* **Automated Data Preparation:** Integrated Outlier Detection (visual) and Feature Selection algorithms.
* **Regression Core:** High-quality Linear (Simple/Multiple) regression output with clear diagnostics.
### Growth Features (Post-MVP)
* **Binary Logistic Regression:** Support for classification-based predictive modeling.
* **Model Benchmark:** Automated "tournament" mode to find the best performing algorithm.
* **Advanced Reporting:** Exportable dashboard summaries for stakeholder presentations.
### Vision (Future)
* **Time Series Forecasting:** Expansion into temporal data prediction.
* **Native Integration:** Two-way sync with Excel/Cloud Storage providers.
## User Journeys
**Journey 1: Julien, l'Ingénieur Qualité - La Course contre la Montre**
Julien est sous pression. Une ligne de production vient de signaler une dérive anormale sur des composants électroniques. Il est 11h, et il doit présenter une analyse de régression à son directeur à 14h pour décider s'il faut arrêter la production. Il a exporté un fichier Excel complexe avec 40 variables (température, humidité, pression, etc.). Habituellement, il perdrait une heure rien qu'à nettoyer les données et à essayer de comprendre quelles variables sont pertinentes.
Il ouvre **Data_analysis**. Il glisse son fichier Excel dans l'interface. Immédiatement, il voit ses données dans une grille familière. Le système fait clignoter un message : *"34 outliers détectés dans la colonne 'Pression_Zone_B'"*. Julien clique, visualise les points rouges sur un graphique et décide de les exclure en un clic. Ensuite, il lance la "Smart Feature Selection". Le système lui explique : *"Les variables Température_C et Vitesse_Tapis expliquent 85% de la dérive"*. À 11h15, Julien a déjà son modèle de régression validé et une visualisation claire. Il se sent serein pour sa réunion : il ne présente pas juste des chiffres, il présente une solution validée.
**Journey 2: Sarah, l'Administratrice IT - La Gestion sans Stress**
Sarah doit s'assurer que les outils utilisés par les ingénieurs sont sécurisés et ne saturent pas les ressources du serveur. Elle se connecte à son tableau de bord **Data_analysis**. Elle peut voir en un coup d'œil le nombre d'analyses en cours et la mémoire consommée par le backend Python. Elle crée un nouvel accès pour un stagiaire en quelques secondes. Pour elle, le succès, c'est que personne ne l'appelle pour dire "le logiciel a planté" parce qu'un fichier était trop gros.
**Journey 3: Marc, le Directeur de Production - La Décision Rapide**
Marc reçoit un lien de la part de Julien. Il n'est pas statisticien. Il ouvre le lien sur sa tablette. Il ne voit pas des lignes de code, mais un rapport interactif. Il voit le graphique de régression, lit l'explication simplifiée ("La vitesse du tapis est le facteur clé") et clique sur le PDF pour l'archiver. Il a pu prendre la décision d'ajuster la vitesse du tapis en 2 minutes, sauvant ainsi la production de l'après-midi.
### Journey Requirements Summary
**For Julien (Analyst):**
* **Fast Data Ingestion:** Drag-and-drop Excel/CSV support.
* **Visual Data Cleaning:** Automated outlier detection with interactive exclusion.
* **Explainable ML:** Feature selection that explains "why" (percentage of variance explained).
* **Validation:** Clear regression metrics (R², P-values) presented simply.
**For Sarah (Admin):**
* **System Health:** Dashboard for monitoring backend resources (Python server load).
* **Access Control:** Simple user management (RBAC).
* **Stability:** Robust error handling for large files to prevent system crashes.
**For Marc (Consumer):**
* **Accessibility:** Mobile/Tablet responsive view for reports.
* **Simplicity:** "Read-only" mode with simplified insights (no code/formulas).
* **Portability:** One-click PDF export for archiving/sharing.
## Domain-Specific Requirements
### Scientific Validation & Reproducibility
**Data_analysis** must adhere to strict scientific rigor to be a credible Minitab alternative. Users rely on these results for quality control and critical decision-making.
### Key Domain Concerns
* **Reproducibility:** Ensuring identical inputs yield identical outputs, regardless of when or where the analysis is run.
* **Methodological Transparency:** Avoiding "black box" algorithms; users must understand *how* an outlier was detected.
* **Computational Integrity:** Handling floating-point precision and large matrix operations without degradation.
### Compliance Requirements
* **Audit Trail:** Every generated report must include an appendix listing:
* Software Version & Library Versions (Pandas/Scikit-learn versions).
* Random Seed used for stochastic processes (Isolation Forest, train/test split).
* Sequence of applied filters (e.g., "Row 45 excluded due to Z-score > 3").
### Industry Standards & Best Practices
* **Statistical Standards:** Use `statsmodels` for classical regression (p-values, confidence intervals) to match traditional statistical expectations, and `scikit-learn` for predictive tasks.
* **Visual Standards:** Error bars, confidence bands, and residual plots must follow standard scientific visualization conventions (e.g., Q-Q plots for normality).
### Required Expertise & Validation
* **Validation Methodology:**
* **Unit Tests for Math:** Verify regression outputs against known standard datasets (e.g., Anscombe's quartet).
* **Drift Detection:** Alert users if data distribution significantly deviates from assumptions (e.g., normality check for linear regression).
### Implementation Considerations
* **Asynchronous Processing:** Heavy computations (Feature Selection on >10k rows) must be offloaded to a background worker (Celery/Redis) to maintain UI responsiveness.
* **Fixed Random Seeds:** All stochastic algorithms must imply a fixed random state by default to ensure consistency, with an option for the user to change it.
## Innovation & Novel Patterns
### Detected Innovation Areas
* **Hybrid "Spreadsheet-Notebook" Interface:**
* **Concept:** Combines the low barrier to entry of a spreadsheet (Excel) with the computational power and reproducibility of a code notebook (Jupyter), without requiring the user to write code.
* **Differentiation:** Traditional tools are either "click-heavy" (Minitab) or "code-heavy" (Python/R). Data_analysis sits in the "sweet spot" of **No-Code Data Science** with full transparency.
* **Guided "GPS" Workflow:**
* **Concept:** Instead of a passive toolbox, the system actively guides the analysis. It doesn't just ask "What model do you want?", it suggests "Your data has outliers, let's fix them first" and "These 3 variables are the most predictive."
* **Differentiation:** Moves from "User-Driven Analysis" to **"Assisted Analysis"**, reducing the risk of statistical errors by non-experts.
* **Explainable AI (XAI) for Quality:**
* **Concept:** Using advanced algorithms (Isolation Forest) not just to *remove* bad data, but to *explain* why it's bad visually.
* **Differentiation:** Makes complex ML concepts accessible to domain experts (e.g., Quality Engineers) who understand the *context* but not necessarily the *algorithm*.
### Market Context & Competitive Landscape
* **Legacy Players:** Minitab, SPSS (Powerful but expensive, dated UI, steep learning curve).
* **Modern Data Tools:** Tableau, PowerBI (Great for visualization, weak for advanced statistical regression).
* **Code-Based:** Jupyter, Streamlit (Powerful but requires coding skills).
* **Opportunity:** Data_analysis fills the gap for a **Modern, Web-Based, Statistical Power Tool** for non-coders.
### Validation Approach
* **User Testing:** Compare time-to-insight between Data_analysis and Minitab for a standard regression task.
* **Side-by-Side Benchmark:** Run the same dataset through Minitab and Data_analysis to validate numerical accuracy (ensure results match to 4 decimal places).
### Risk Mitigation
* **"Black Box" Trust:** Users might not trust automated suggestions.
* *Mitigation:* Always provide a "Show Details" view with raw statistical metrics (p-values) to prove the "why".
* **Performance:** Python backend might lag on large Excel files.
* *Mitigation:* Implemented async task queue (Celery) and progressive loading for the frontend grid.
## Web App Specific Requirements
### Project-Type Overview
As a scientific web application, Data_analysis prioritizes data integrity and high-performance interactivity. The technical architecture must support heavy client-side state management (for the grid) while leveraging robust backend statistical processing.
### Technical Architecture Considerations
* **Rendering Strategy:**
* **Shell & Reports:** Next.js Server Components for optimized performance and SEO (if public).
* **Data Grid:** React Client Components to manage complex state transitions, cell editing, and local filtering with sub-second latency.
* **Data Persistence:**
* **Session-based Workspace:** Users work on a "Project" basis; files are uploaded to temporary storage for analysis, with an option to persist to a database (PostgreSQL) for long-term tracking.
* **Browser Strategy:** Support for modern "Evergreen" browsers (Chrome, Edge, Firefox, Safari). High-performance features like Web Workers may be used for local data transformations.
### Functional Requirements (Web-Specific)
* **Excel-like Interactions:** Support for keyboard shortcuts (Ctrl+C/V, Undo/Redo), drag-to-fill (Growth), and multi-cell selection.
* **Responsive Analysis:** The interface must adapt for "Marc's Journey" (Manager/Consumer) on tablets, while ensuring "Julien's Journey" (Analyst) is optimized for high-resolution desktop displays.
* **Accessibility:** Adherence to WCAG 2.1 principles for the UI shell, with specific focus on keyboard-only navigation for the data entry grid.
### Implementation Considerations
* **Security:** JWT-based authentication for Sarah's (Admin) user management. All data uploads must be scanned for malicious macros/content.
* **Stateless Backend:** The Python API (FastAPI) will remain largely stateless, receiving data via secure requests and returning analytical results/visualizations in JSON/Base64 format.
## Project Scoping & Phased Development
### MVP Strategy & Philosophy
**MVP Approach:** Experience MVP - Stateless & Fast.
**Core Value:** Deliver a "Zero-Setup" analytical tool where users get results instantly without creating accounts or managing projects. Focus on the *quality* of the interaction and the analysis report.
### MVP Feature Set (Phase 1)
**Core User Journeys Supported:**
* **Julien (Analyst):** Full flow from upload to regression report.
* **Marc (Manager):** Reading the generated PDF report.
* *(Deferred)* **Sarah (Admin):** No admin dashboard needed yet as the system is stateless/public.
**Must-Have Capabilities:**
* **Input:** Drag & Drop Excel/CSV parser (Pandas).
* **Interaction:** Interactive Data Grid (Read/Write) for quick cleaning/filtering.
* **Analysis Core:**
* Automated Outlier Detection (Isolation Forest).
* Automated Feature Selection (RFE).
* Models: Linear Regression (Simple/Multiple), Logistic Regression, Correlation Matrix.
* **Output:** Interactive Web Report + One-click PDF Export.
### Post-MVP Features
**Phase 2 (Growth - "Project Mode"):**
* User Accounts & Project Persistence (PostgreSQL).
* Admin Dashboard for resource monitoring.
* Advanced Models: Time Series, ANOVA.
**Phase 3 (Expansion - "Enterprise"):**
* Collaboration (Real-time editing).
* Direct connectors (SQL Database, Salesforce).
* On-premise deployment options (Docker).
### Risk Mitigation Strategy
**Technical Risks:**
* **Grid Performance:** Using a robust React Data Grid library (TanStack Table or AG Grid Community) to handle DOM virtualization for 50k rows.
* **Stateless Memory:** Limiting file upload size (e.g., 50MB) to prevent RAM saturation since we aren't using a DB yet.
**Market Risks:**
* **Trust:** Ensuring the PDF report looks professional enough to be accepted in a formal meeting (Marc's journey).
## Functional Requirements
### Data Ingestion & Management
- **FR1:** Users can upload datasets in .xlsx, .xls, and .csv formats via drag-and-drop or file selection.
- **FR2:** System automatically detects column data types (numeric, categorical, datetime) upon ingest.
- **FR3:** Users can manually override detected data types if the inference is incorrect.
- **FR4:** Users can rename columns directly in the interface to sanitize inputs.
### Interactive Data Grid (Workspace)
- **FR5:** Users can view loaded data in a paginated, virtualized grid capable of displaying 50,000+ rows.
- **FR6:** Users can edit cell values directly (double-click to edit) with inputs validated against the column type.
- **FR7:** Users can sort columns (asc/desc) and filter rows based on values/conditions (e.g., "> 100").
- **FR8:** Users can perform Undo/Redo operations (Ctrl+Z/Ctrl+Y) on data edits within the current session.
- **FR9:** Users can exclude specific rows from analysis without deleting them (soft delete/toggle).
### Automated Data Preparation (Smart Prep)
- **FR10:** System automatically identifies univariate outliers using IQR/Z-score and visualizes them in the grid/plots.
- **FR11:** System automatically identifies multivariate outliers using Isolation Forest upon user request.
- **FR12:** Users can accept or reject outlier exclusion proposals individually or in bulk.
- **FR13:** Users can select a Target Variable (Y) to trigger an automated Feature Importance analysis.
- **FR14:** System recommends the Top-N predictive features based on RFE (Recursive Feature Elimination) or Random Forest importance.
### Statistical Modeling (Core Analytics)
- **FR15:** Users can configure a Linear Regression (Simple/Multiple) by selecting Dependent (Y) and Independent (X) variables.
- **FR16:** Users can configure a Binary Logistic Regression for categorical target variables.
- **FR17:** System generates a "Model Summary" including R-squared, Adjusted R-squared, F-statistic, and P-values for coefficients.
- **FR18:** System generates standard diagnostic plots: Residuals vs Fitted, Q-Q Plot, and Scale-Location.
- **FR19:** Users can view a Correlation Matrix (Heatmap) for selected numeric variables.
### Reporting & Reproducibility
- **FR20:** Users can view an interactive "Analysis Report" dashboard summarizing data health, methodology, and model results.
- **FR21:** Users can export the full report as a branded PDF document.
- **FR22:** System appends an "Audit Trail" to the report listing library versions, random seeds, and data exclusion steps for reproducibility.
## Non-Functional Requirements
### Performance
* **Grid Latency:** The interactive data grid must render 50,000 rows with filtering/sorting response times under 200ms (Client-Side Virtualization).
* **Analysis Throughput:** Automated analysis (Outlier Detection + Feature Selection) on standard datasets (<10MB) must complete in under 15 seconds.
* **Upload Speed:** Parsing and validation of a 5MB Excel file should complete in under 3 seconds.
### Security & Privacy
* **Data Ephemerality:** All user datasets uploaded to the temporary workspace must be permanently purged from the server memory/storage after 1 hour of inactivity or immediately upon session termination.
* **Transport Security:** All data transmission between Client and Server must be encrypted via TLS 1.3.
* **Input Sanitization:** The file parser must strictly validate MIME types and file signatures to prevent malicious code execution (e.g., Excel Macros).
### Reliability & Stability
* **Graceful Degradation:** The system must handle "bad data" (NaNs, infinite values) by providing clear error messages rather than crashing the Python backend (500 Internal Error).
* **Concurrency:** The backend must support at least 50 concurrent analysis requests without performance degradation, using an asynchronous task queue (Celery).
### Accessibility
* **Keyboard Navigation:** The data grid must be fully navigable via keyboard (Arrows, Tab, Enter) to support "Power User" workflows efficiently.

192
_bmad-output/planning-artifacts/ux-design-directions.html Normal file
View File

@@ -0,0 +1,192 @@
<!DOCTYPE html>
<html lang="fr" class="light">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Data_analysis - Design System Showcase</title>
<script src="https://cdn.tailwindcss.com"></script>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
<script>
tailwind.config = {
darkMode: 'class',
theme: {
extend: {
fontFamily: {
sans: ['Inter', 'sans-serif'],
mono: ['JetBrains Mono', 'monospace'],
},
colors: {
indigo: {
50: '#eef2ff', 100: '#e0e7ff', 200: '#c7d2fe', 300: '#a5b4fc',
400: '#818cf8', 500: '#6366f1', 600: '#4f46e5', 700: '#4338ca',
800: '#3730a3', 900: '#312e81', 950: '#1e1b4b',
},
slate: {
50: '#f8fafc', 100: '#f1f5f9', 200: '#e2e8f0', 300: '#cbd5e1',
400: '#94a3b8', 500: '#64748b', 600: '#475569', 700: '#334155',
800: '#1e293b', 900: '#0f172a', 950: '#020617',
}
}
}
}
}
</script>
<style>
.grid-cell { font-family: 'JetBrains Mono', monospace; font-size: 13px; }
.transition-theme { transition: background-color 0.3s ease, color 0.3s ease, border-color 0.3s ease; }
</style>
</head>
<body class="bg-slate-50 dark:bg-slate-950 transition-theme min-h-screen p-8 font-sans">
<div class="max-w-6xl mx-auto space-y-12">
<!-- Header -->
<header class="flex items-center justify-between border-b border-slate-200 dark:border-slate-800 pb-6">
<div>
<h1 class="text-3xl font-bold text-slate-900 dark:text-white">Design System Showcase</h1>
<p class="text-slate-500 dark:text-slate-400 mt-1">Composants clés pour Data_analysis • Shadcn UI style</p>
</div>
<button onclick="toggleDarkMode()" class="bg-white dark:bg-slate-900 border border-slate-200 dark:border-slate-700 px-4 py-2 rounded-lg shadow-sm flex items-center gap-2 text-sm font-medium transition-colors hover:bg-slate-50 dark:hover:bg-slate-800 dark:text-white">
<span class="dark:hidden text-slate-600">🌙 Passer au Mode Sombre</span>
<span class="hidden dark:block text-slate-300">☀️ Passer au Mode Clair</span>
</button>
</header>
<!-- Section 1: Buttons & Controls -->
<section class="space-y-6">
<h2 class="text-xl font-semibold dark:text-white flex items-center gap-2">
<span class="w-1.5 h-6 bg-indigo-600 rounded-full"></span>
Contrôles & Actions
</h2>
<div class="grid grid-cols-1 md:grid-cols-2 gap-8">
<!-- Buttons -->
<div class="bg-white dark:bg-slate-900 p-6 rounded-xl border border-slate-200 dark:border-slate-800 space-y-4">
<p class="text-xs font-bold text-slate-400 uppercase tracking-widest mb-4">Boutons (Buttons)</p>
<div class="flex flex-wrap gap-3">
<button class="bg-indigo-600 hover:bg-indigo-700 text-white px-4 py-2 rounded-md text-sm font-medium transition-all shadow-sm shadow-indigo-200 dark:shadow-none">Primary Action</button>
<button class="bg-white dark:bg-slate-800 border border-slate-200 dark:border-slate-700 text-slate-700 dark:text-slate-200 px-4 py-2 rounded-md text-sm font-medium hover:bg-slate-50 dark:hover:bg-slate-700 transition-colors">Secondary</button>
<button class="text-slate-500 dark:text-slate-400 hover:text-slate-900 dark:hover:text-white px-4 py-2 text-sm font-medium">Ghost</button>
<button class="bg-rose-50 dark:bg-rose-950/30 text-rose-600 dark:text-rose-400 border border-rose-100 dark:border-rose-900/50 px-4 py-2 rounded-md text-sm font-medium hover:bg-rose-100 dark:hover:bg-rose-900/50 transition-colors">Destructive</button>
</div>
</div>
<!-- Badges -->
<div class="bg-white dark:bg-slate-900 p-6 rounded-xl border border-slate-200 dark:border-slate-800 space-y-4">
<p class="text-xs font-bold text-slate-400 uppercase tracking-widest mb-4">Statuts & Badges</p>
<div class="flex flex-wrap gap-3">
<span class="bg-emerald-100 dark:bg-emerald-950/30 text-emerald-700 dark:text-emerald-400 px-2.5 py-0.5 rounded-full text-xs font-semibold border border-emerald-200 dark:border-emerald-900/50">Valid Data</span>
<span class="bg-rose-100 dark:bg-rose-950/30 text-rose-700 dark:text-rose-400 px-2.5 py-0.5 rounded-full text-xs font-semibold border border-rose-200 dark:border-rose-900/50">Outlier Detected</span>
<span class="bg-indigo-100 dark:bg-indigo-950/30 text-indigo-700 dark:text-indigo-400 px-2.5 py-0.5 rounded-full text-xs font-semibold border border-indigo-200 dark:border-indigo-900/50">Target (Y)</span>
<span class="bg-slate-100 dark:bg-slate-800 text-slate-600 dark:text-slate-400 px-2.5 py-0.5 rounded-full text-xs font-semibold">Numeric</span>
</div>
</div>
</div>
</section>
<!-- Section 2: The Smart Grid -->
<section class="space-y-6">
<h2 class="text-xl font-semibold dark:text-white flex items-center gap-2">
<span class="w-1.5 h-6 bg-indigo-600 rounded-full"></span>
La Smart Grid (TanStack Table Style)
</h2>
<div class="bg-white dark:bg-slate-900 rounded-xl border border-slate-200 dark:border-slate-800 overflow-hidden shadow-sm">
<table class="w-full border-separate border-spacing-0">
<thead class="bg-slate-50 dark:bg-slate-800/50">
<tr>
<th class="border-b border-r border-slate-200 dark:border-slate-700 p-2 text-center text-xs font-mono text-slate-400">#</th>
<th class="border-b border-r border-slate-200 dark:border-slate-700 p-3 text-left">
<div class="flex flex-col gap-1">
<span class="text-xs font-bold text-slate-900 dark:text-white">Temperature_C</span>
<span class="text-[10px] font-mono text-slate-400 uppercase tracking-tighter">C1 • Numeric</span>
</div>
</th>
<th class="border-b border-r border-slate-200 dark:border-slate-700 p-3 text-left">
<div class="flex flex-col gap-1">
<div class="flex items-center gap-2">
<span class="text-xs font-bold text-slate-900 dark:text-white">Pressure_Bar</span>
<span class="w-2 h-2 bg-rose-500 rounded-full animate-pulse"></span>
</div>
<span class="text-[10px] font-mono text-slate-400 uppercase tracking-tighter">C2 • Numeric</span>
</div>
</th>
<th class="border-b border-slate-200 dark:border-slate-700 p-3 text-left bg-indigo-50/30 dark:bg-indigo-900/10">
<div class="flex flex-col gap-1">
<span class="text-xs font-bold text-indigo-600 dark:text-indigo-400">Yield_Output</span>
<span class="text-[10px] font-mono text-indigo-400 uppercase tracking-tighter italic">Target Variable</span>
</div>
</th>
</tr>
</thead>
<tbody class="divide-y divide-slate-100 dark:divide-slate-800">
<!-- Normal Row -->
<tr class="hover:bg-slate-50 dark:hover:bg-slate-800/50 transition-colors">
<td class="p-2 text-center text-xs font-mono text-slate-400 border-r border-slate-100 dark:border-slate-800">1</td>
<td class="p-3 grid-cell text-slate-700 dark:text-slate-300 border-r border-slate-100 dark:border-slate-800">24.50</td>
<td class="p-3 grid-cell text-slate-700 dark:text-slate-300 border-r border-slate-100 dark:border-slate-800">1.02</td>
<td class="p-3 grid-cell font-bold text-slate-900 dark:text-white bg-indigo-50/10 dark:bg-indigo-900/5">98.2</td>
</tr>
<!-- Outlier Row -->
<tr class="bg-rose-50/50 dark:bg-rose-900/10">
<td class="p-2 text-center text-xs font-mono text-rose-400 border-r border-rose-100 dark:border-rose-900/20">2</td>
<td class="p-3 grid-cell text-slate-700 dark:text-slate-300 border-r border-rose-100 dark:border-rose-900/20">24.52</td>
<td class="p-3 grid-cell font-bold text-rose-600 dark:text-rose-400 border-r border-rose-100 dark:border-rose-900/20 shadow-inner">9.99*</td>
<td class="p-3 grid-cell font-bold text-slate-900 dark:text-white opacity-40">45.1</td>
</tr>
<!-- Editing Row -->
<tr class="bg-white dark:bg-slate-900">
<td class="p-2 text-center text-xs font-mono text-slate-400 border-r border-slate-100 dark:border-slate-800">3</td>
<td class="p-3 border-r border-slate-100 dark:border-slate-800">
<input type="text" value="24.48" class="w-full bg-indigo-50 dark:bg-indigo-900/30 border border-indigo-500 dark:border-indigo-400 rounded px-2 py-1 text-sm font-mono dark:text-white outline-none">
</td>
<td class="p-3 grid-cell text-slate-700 dark:text-slate-300 border-r border-slate-100 dark:border-slate-800">1.01</td>
<td class="p-3 grid-cell font-bold text-slate-900 dark:text-white">97.9</td>
</tr>
</tbody>
</table>
</div>
</section>
<!-- Section 3: Smart Insights Panel -->
<section class="space-y-6">
<h2 class="text-xl font-semibold dark:text-white flex items-center gap-2">
<span class="w-1.5 h-6 bg-indigo-600 rounded-full"></span>
Insight Panel (Explainable AI)
</h2>
<div class="max-w-md bg-white dark:bg-slate-900 rounded-xl border border-slate-200 dark:border-slate-800 shadow-xl overflow-hidden">
<div class="p-4 bg-indigo-600 flex items-center justify-between text-white">
<span class="font-bold text-sm">Smart Insight</span>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><path d="m9 12 2 2 4-4"/></svg>
</div>
<div class="p-6 space-y-6">
<div>
<p class="text-[10px] font-bold text-slate-400 dark:text-slate-500 uppercase tracking-widest mb-2">Observation</p>
<p class="text-sm dark:text-slate-300">Column <span class="font-mono text-indigo-600 dark:text-indigo-400 font-bold">Pressure_Bar</span> has 34 outliers.</p>
</div>
<!-- Simulated Chart -->
<div class="h-24 bg-slate-50 dark:bg-slate-800 rounded-lg flex items-end gap-1 p-2 border border-slate-100 dark:border-slate-700">
<div class="bg-indigo-400/30 w-full h-[20%] rounded-t"></div>
<div class="bg-indigo-400/30 w-full h-[40%] rounded-t"></div>
<div class="bg-indigo-400/30 w-full h-[100%] rounded-t"></div>
<div class="bg-indigo-400/30 w-full h-[60%] rounded-t"></div>
<div class="bg-rose-500 w-full h-[15%] rounded-t border-t-2 border-rose-600 shadow-[0_0_8px_rgba(244,63,94,0.4)]"></div>
</div>
<div class="space-y-3">
<p class="text-xs text-slate-500 italic dark:text-slate-400">Excluding these will increase your model accuracy (R²) by <strong>26%</strong>.</p>
<button class="w-full bg-indigo-600 hover:bg-indigo-700 text-white py-2 rounded-lg text-sm font-bold transition-all shadow-lg shadow-indigo-100 dark:shadow-none">Appliquer la Correction</button>
</div>
</div>
</div>
</section>
</div>
<script>
function toggleDarkMode() {
document.documentElement.classList.toggle('dark');
}
</script>
</body>
</html>

392
_bmad-output/planning-artifacts/ux-design-specification.md Normal file
View File

@@ -0,0 +1,392 @@
---
stepsCompleted: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
inputDocuments: ['_bmad-output/planning-artifacts/prd.md']
---
# UX Design Specification Data_analysis
**Author:** Sepehr
**Date:** 2026-01-10
---
<!-- UX design content will be appended sequentially through collaborative workflow steps -->
## Executive Summary
### Project Vision
Create a modern, web-based, "No-Code" alternative to Minitab. The goal is to empower domain experts (engineers, analysts) to perform rigorous statistical regressions via a hybrid interface combining the simplicity of Excel with the computational power of Python.
### Target Users
* **Julien (Analyst/Engineer):** Domain expert user, seeks efficiency and rigor without coding. Primarily uses a desktop computer.
* **Marc (Decision Maker):** Result consumer, needs clear, mobile-friendly reports to validate production decisions.
### Key Design Challenges
* **Grid Performance:** Maintain fluid interactivity with large data volumes (virtualization).
* **Statistical Vulgarization:** Make variable selection and outlier detection concepts intuitive through visual design.
* **Guided Workflow:** Design a conversion funnel (from raw file to final report) that reduces cognitive load.
### Design Opportunities
* **Familiar Interface:** Leverage Microsoft Excel design patterns to reduce initial friction.
* **"Mobile-First" Reports:** Create a competitive advantage with report exports and views optimized for tablets.
## Core User Experience
### Defining Experience
The core of Data_analysis is the **"Smart Grid"**. Unlike a static HTML table, this grid feels alive. It's the command center where data ingestion, cleaning, and exploration happen seamlessly. Users don't "run scripts"; they interact with their data directly, with the system acting as an intelligent co-pilot suggesting corrections and insights.
### Platform Strategy
* **Desktop (Primary):** Optimized for mouse/keyboard inputs. High density of information. Supports "Power User" shortcuts (Ctrl+Z, Arrows).
* **Tablet (Secondary):** Optimized for touch. "Read-only" mode for reports and dashboards. Lower density, larger touch targets.
### Effortless Interactions
* **Zero-Config Import:** Drag-and-drop Excel ingestion with auto-detection of headers, types, and delimiters. No wizard fatigue.
* **One-Click Hygiene:** Automated detection of data anomalies (NaNs, wrong types) with single-click remediation actions ("Fix all", "Drop rows").
### Critical Success Moments
* **The "Clarity" Moment:** When the "Smart Feature Selection" reduces a chaotic 50-column dataset to the 3-4 variables that actually matter, visualized clearly.
* **The "Confidence" Moment:** When the system confirms "No outliers detected" or "Model assumptions met" via clear green indicators before generating the report.
### Experience Principles
1. **Direct Manipulation:** Don't hide data behind menus. Let users click, edit, and filter right where the data lives.
2. **Proactive Intelligence:** Don't wait for the user to find errors. Highlight them immediately and offer solutions.
3. **Visual First:** Show the data distribution (mini-histograms) in the headers. Show the outliers on a plot, not just a list of row numbers.
## Desired Emotional Response
### Primary Emotional Goals
The primary emotional goal of Data_analysis is to move the user from **Anxiety to Confidence**. Statistics can be intimidating; our interface must act as a reassuring expert co-pilot.
### Emotional Journey Mapping
* **Discovery:** **Curiosity & Hope.** "Can this really replace my manual Excel cleaning?"
* **Data Ingestion:** **Relief.** "It parsed my file instantly without errors."
* **Data Cleaning:** **Surprise & Empowerment.** "I didn't know I had outliers, now I see them clearly."
* **Analysis/Reporting:** **Confidence & Pride.** "This report looks professional and I understand every part of it."
### Micro-Emotions
* **Trust vs. Skepticism:** Built through "Explainable AI" tooltips.
* **Calm vs. Frustration:** Achieved through smooth animations and non-blocking background tasks.
* **Mastery vs. Confusion:** Delivered by guiding the user through a linear logical workflow.
### Design Implications
* **Confidence** → Use a sober, professional color palette (Blues/Grays). Provide clear "Validation" checkmarks when data is clean.
* **Relief** → Automate tedious tasks like type-casting and missing value detection. Use "Undo" to remove the fear of making mistakes.
* **Empowerment** → Use natural language labels instead of cryptic statistical abbreviations (e.g., "Predictive Power" instead of "Coefficient of Determination").
### Emotional Design Principles
1. **Safety Net:** Users should never feel like they can "break" the data. Every action is reversible.
2. **No Dead Ends:** If an error occurs (e.g., singular matrix), explain *why* in plain French and how to fix it.
3. **Visual Rewards:** Use subtle success animations when a model is successfully trained.
## UX Pattern Analysis & Inspiration
### Inspiring Products Analysis
* **Microsoft Excel:** The standard for grid interaction. Users expect double-click editing, arrow-key navigation, and "fill-down" patterns.
* **Airtable:** Revolutionized the data grid with modern UI patterns. We adopt their clean column headers, visual data types (badges, progress bars), and intuitive filtering.
* **Linear / Vercel:** The benchmark for high-performance developer tools. We draw inspiration from their minimalist aesthetic, exceptional Dark Mode, and keyboard-first navigation.
### Transferable UX Patterns
* **Navigation:** **Sidebar-less / Hub & Spoke.** Focus on the data grid as the central workspace with floating or collapsible side panels for analysis tools.
* **Interaction:** **"Sheet-to-Report" Pipeline.** A clear horizontal or vertical progression from raw data to a finalized interactive report.
* **Visual:** **Statistical Overlays.** Using "Sparklines" (mini-histograms) in column headers to show data distribution at a glance.
### Anti-Patterns to Avoid
* **The Modal Maze:** Opening a new pop-up window for every statistical setting. We prefer slide-over panels or inline settings to keep the context visible.
* **Opaque Processing:** Showing a generic spinner during long calculations. We will use a "Step-by-Step" status bar (e.g., "1. Parsing -> 2. Detecting Outliers -> 3. Selecting Features").
### Design Inspiration Strategy
* **Adopt:** The "TanStack Table" logic for grid virtualization (Excel speed) combined with Shadcn UI components (Vercel aesthetic).
* **Adapt:** Excel's right-click menu to include specific statistical actions like "Exclude from analysis" or "Set as Target (Y)".
* **Avoid:** Complex "Dashboard Builders." Users want a generated report, not a canvas they have to design themselves.
## Design System Foundation
### 1.1 Design System Choice
The project will use **Shadcn UI** as the primary UI library, built on top of **Tailwind CSS** and **Radix UI**. The core data interaction will be powered by **TanStack Table (headless)** to create a custom, high-performance "Smart Grid."
### Rationale for Selection
* **Performance:** TanStack Table allows for massive data virtualization (50k+ rows) without the overhead of heavy UI frameworks.
* **Aesthetic Consistency:** Shadcn provides the "Vercel-like" minimalist and professional aesthetic defined in our inspiration phase.
* **Accessibility:** Leveraging Radix UI primitives ensures that complex components (popovers, dropdowns, dialogs) are fully WCAG compliant.
* **Developer Experience:** Direct ownership of component code allows for deep customization of statistical-specific UI elements.
### Implementation Approach
* **Shell:** Standard Shadcn layout components (Sidebar, TopNav).
* **Data Grid:** A custom-built component using TanStack Table's hook logic, styled with Shadcn Table primitives.
* **Charts:** Integration of **Recharts** or **Tremor** (which matches Shadcn's style) for statistical visualizations.
### Customization Strategy
* **Tokens:** Neutral gray base with "Scientific Blue" as the primary action color.
* **Typography:** Sans-serif (Geist or Inter) for the UI; Monospace (JetBrains Mono) for data cells and statistical metrics.
* **Density:** "High-Density" mode by default for the grid (small cell padding) to maximize data visibility.
## 2. Core User Experience
### 2.1 Defining Experience
The defining interaction of Data_analysis is the **"Guided Data Hygiene Loop"**. It transforms the tedious task of cleaning data into a rapid, rewarding conversation with the system. Users don't "edit cells"; they respond to intelligent insights that actively improve their model's quality in real-time.
### 2.2 User Mental Model
* **Current Model:** "I have to manually hunt for errors row by row in Excel, then delete them and hope I didn't break anything."
* **Target Model:** "The system is my Quality Assistant. It points out the issues, I make the executive decision, and I instantly see the result."
### 2.3 Success Criteria
* **Speed:** Reviewing and fixing 50 outliers should take less than 30 seconds.
* **Safety:** Users must feel that "excluding" data is non-destructive (reversible).
* **Reward:** Every fix must trigger a positive visual feedback (e.g., model accuracy score pulsing green).
### 2.4 Novel UX Patterns
* **"Contextual Insight Panel":** Instead of modal popups, a slide-over panel allows users to see the specific rows in question (highlighted in the grid) while reviewing the statistical explanation (boxplot/histogram) side-by-side.
* **"Live Impact Preview":** Before confirming an exclusion, hover over the button to see a "Ghost Curve" showing how the regression line *will* change.
### 2.5 Experience Mechanics
1. **Initiation:** System highlights "dirty" columns with a subtle warning badge in the header.
2. **Interaction:** User clicks the header badge. The Insight Panel slides in.
3. **Feedback:** The panel shows "34 values are > 3 Sigma". The grid highlights these 34 rows.
4. **Action:** User clicks "Exclude All". Rows fade to gray. The Regression R² badge updates from 0.65 to 0.82 with a celebration animation.
5. **Completion:** The column header badge turns to a green checkmark.
## Visual Design Foundation
### Color System
* **Neutral:** Slate (50-900) - Technical, cold background for heavy data.
* **Primary:** Indigo (600) - For primary actions ("Run Regression").
* **Semantic Data Colors:**
* **Rose (500):** Outliers/Errors (Soft alert).
* **Emerald (500):** Valid Data/Success (Reassurance).
* **Amber (500):** Warnings/Missing Values.
* **Modes:** Fully supported Dark Mode using Slate-900 backgrounds and Indigo-400 primary accents.
### Typography System
* **Interface:** `Inter` (or Geist Sans) - Clean, legible at small sizes.
* **Data:** `JetBrains Mono` - Mandatory for the grid to ensure tabular alignment of decimals.
### Spacing & Layout Foundation
* **Grid Density:** Ultra-compact (4px y-padding) to maximize data visibility.
* **Panel Density:** Comfortable (16px padding) for reading insights.
* **Layout:** Full-width liquid layout. No wasted margins.
### Accessibility Considerations
* **Contrast:** Ensure data text (Slate-700) on row backgrounds meets AA standards.
* **Focus States:** High-visibility focus rings (Indigo-500 ring) for keyboard navigation in the grid.
## Design Direction Decision
### Design Directions Explored
Multiple design approaches were evaluated to balance density, readability, and modern aesthetics:
* **"Corporate Legacy":** Mimicking Minitab/Excel directly (too cluttered).
* **"Creative Canvas":** Like Notion/Miro (too open-ended).
* **"Lab & Tech":** A hybrid of Vercel's minimalism and Excel's density.
### Chosen Direction
**"Lab & Tech" with Shadcn UI & TanStack Table**
* **Visual Style:** Minimalist, data-first, with a strong Dark Mode.
* **Components:** Shadcn UI for the shell, TanStack Table for the grid.
* **Palette:** Slate + Indigo + Rose/Emerald semantic indicators.
### Design Rationale
* **User Fit:** Matches Julien's need for a professional, distraction-free environment.
* **Modernity:** Positions the tool as a "Next-Gen" product compared to legacy competitors.
* **Scalability:** The component library allows for easy addition of complex statistical widgets later.
### Implementation Approach
* **CSS Framework:** Tailwind CSS.
* **Component Library:** Shadcn UI (Radix based).
* **Icons:** Lucide React.
* **Charts:** Recharts.
## User Journey Flows
### Journey 1: Julien - The Guided Hygiene Loop
This flow details how Julien interacts with the system to clean his data. The focus is on the "Ping-Pong" interaction between the Grid and the Insight Panel.
```mermaid
graph TD
A[Start: File Uploaded] --> B{System Checks}
B -->|Clean| C[Grid View: Standard]
B -->|Issues Found| D[Grid View: Warning Badge on Header]
D --> E(User Clicks Badge)
E --> F[Action: Open Insight Panel]
subgraph Insight Panel Interaction
F --> G[Display: Issue Description + Chart]
G --> H[Display: Proposed Fix]
H --> I{User Decision}
I -->|Ignore| J[Close Panel & Remove Badge]
I -->|Apply Fix| K[Action: Update Grid Data]
end
K --> L[Feedback: Toast 'Fix Applied']
L --> M[Update Model Score R²]
M --> N[End: Ready for Regression]
```
### Journey 2: Marc - Mobile Decision Making
Optimized for touch and "Read-Only" consumption. No dense grids, just insights.
```mermaid
graph TD
A[Start: Click Link in Email] --> B[View: Mobile Dashboard]
B --> C[Display: Key Metrics Cards]
B --> D[Display: Regression Chart]
D --> E(User Taps Data Point)
E --> F[Action: Show Tooltip Details]
subgraph Decision
F --> G{Is Data Valid?}
G -->|No| H[Action: Add Comment 'Check this']
G -->|Yes| I[Action: Click 'Approve Analysis']
end
H --> J[Notify Julien]
I --> K[Generate PDF & Archive]
```
### Journey 3: Error Handling - The "Graceful Fail"
Ensuring the system handles bad inputs without crashing the Python backend.
```mermaid
graph TD
A[Start: Upload 50MB .xlsb] --> B{Validation Service}
B -->|Success| C[Proceed to Parsing]
B -->|Fail: Macros Detected| D[State: Upload Error]
D --> E[Display: Error Modal]
E --> F[Content: 'Security Risk Detected']
E --> G[Action: 'Sanitize & Retry' Button]
G --> H{Sanitization}
H -->|Success| C
H -->|Fail| I[Display: 'Please upload .xlsx or .csv']
```
### Flow Optimization Principles
1. **Non-Blocking Errors:** Warnings (like outliers) should never block the user from navigating. They are "suggestions", not "gates".
2. **Context Preservation:** When opening the Insight Panel, the relevant grid columns must scroll into view automatically.
3. **Optimistic UI:** When Julien clicks "Apply Fix", the UI updates instantly (Gray out rows) even while the backend saves the state.
## Component Strategy
### Design System Components (Shadcn UI)
We will rely on the standard library for:
* **Layout:** `Sheet` (for Insight Panel), `ScrollArea`, `Resizable`.
* **Forms:** `Button`, `Input`, `Select`, `Switch`.
* **Feedback:** `Toast`, `Progress`, `Skeleton` (for loading states).
### Custom Components Specification
#### 1. `<SmartGrid />`
The central nervous system of the app.
* **Purpose:** Virtualized rendering of massive datasets with Excel-like interactions.
* **Core Props:**
* `data: any[]` - The raw dataset.
* `columns: ColumnDef[]` - Definitions including types and formatters.
* `onCellEdit: (rowId, colId, value) => void` - Handler for data mutation.
* `highlightedRows: string[]` - IDs of rows to highlight (e.g., outliers).
* **Key States:** `Loading`, `Empty`, `Filtering`, `Editing`.
#### 2. `<InsightPanel />`
The container for Explainable AI interactions.
* **Purpose:** Contextual sidebar for statistical insights and data cleaning.
* **Core Props:**
* `isOpen: boolean` - Visibility state.
* `insight: InsightObject` - Contains `{ type: 'outlier' | 'correlation', description: string, chartData: any }`.
* `onApplyFix: () => Promise<void>` - Async handler for the fix action.
* **Anatomy:** Header (Title + Close), Body (Text + Recharts Graph), Footer (Action Buttons).
#### 3. `<ColumnHeader />`
A rich header component for the grid.
* **Purpose:** Show name, type, and distribution summary.
* **Core Props:**
* `label: string`.
* `type: 'numeric' | 'categorical' | 'date'`.
* `distribution: number[]` - Data for the sparkline mini-chart.
* `hasWarning: boolean` - Triggers the red badge.
### Implementation Roadmap
1. **Phase 1 (Grid Core):** Implement `SmartGrid` with read-only virtualization (TanStack Table).
2. **Phase 2 (Interaction):** Add `ColumnHeader` visualization and `onCellEdit` logic.
3. **Phase 3 (Intelligence):** Build the `InsightPanel` and connect it to the outlier detection logic.
## UX Consistency Patterns
### Button Hierarchy
* **Primary (Indigo):** Reserved for "Positive Progression" actions (Run Regression, Save, Export). Only one per view.
* **Secondary (White/Outline):** For "Alternative" actions (Cancel, Clear Filter, Close Panel).
* **Destructive (Rose):** For "Irreversible" actions (Exclude Data, Delete Project). Always requires a confirmation step if significant.
* **Ghost (Transparent):** For tertiary actions inside toolbars (e.g., "Sort Ascending" icon button) to reduce visual noise.
### Feedback Patterns
* **Toasts (Ephemeral):** Used for success confirmations ("Data saved", "Model updated"). Position: Bottom-Right. Duration: 3s.
* **Inline Validation:** Used for data entry errors within the grid (e.g., entering text in a numeric column). Immediate red border + tooltip.
* **Global Status:** A persistent "Status Bar" at the top showing the system state (Ready / Processing... / Done).
### Grid Interaction Patterns (Excel Compatibility)
* **Navigation:** Arrow keys move focus between cells. Tab moves right. Enter moves down.
* **Selection:** Click to select cell. Shift+Click to select range. Click row header to select row.
* **Editing:** Double-click or `Enter` starts editing. `Esc` cancels. `Enter` saves.
* **Context Menu:** Right-click triggers action menu specific to the selected object (Cell vs Row vs Column).
### Empty States
* **No Data:** Don't show an empty grid. Show a "Drop Zone" with a clear CTA ("Upload Excel File") and sample datasets for exploration.
* **No Selection:** When the Insight Panel is open but nothing is selected, show a helper illustration ("Select a column to see stats").
## Responsive Design & Accessibility
### Responsive Strategy
* **Desktop Only:** The application is strictly optimized for high-resolution desktop displays (1366px width minimum). No responsive breakpoints for mobile or tablet will be implemented.
* **Layout Focus:** Use a fixed Sidebar + Liquid Grid layout. The grid will expand to fill all available horizontal space.
### Breakpoint Strategy
* **Default:** 1440px+ (Optimized).
* **Minimum:** 1280px (Functional). Below this, a horizontal scrollbar will appear for the entire app shell to preserve data integrity.
### Accessibility Strategy
* **Compliance:** WCAG 2.1 Level AA.
* **Keyboard First:** Full focus on making the Data Grid and Insight Panel navigable without a mouse.
* **Screen Reader support:** Required for statistical summaries and report highlights.
### Testing Strategy
* **Browsers:** Chrome, Edge, and Firefox (latest 2 versions).
* **Devices:** Standard laptops (13" to 16") and external monitors (24"+).
### Implementation Guidelines
* **Container Query:** Use `@container` for complex widgets (like the Insight Panel) to adapt their layout based on the sidebar's width rather than the screen width.
* **Focus Management:** Ensure the focus ring is never hidden and follows a logical order (Sidebar -> Grid -> Insight Panel).

256
_bmad-output/planning-artifacts/ux-visual-visualizer.html Normal file
View File

@@ -0,0 +1,256 @@
<!DOCTYPE html>
<html lang="fr">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Data_analysis - UX Visual Foundation</title>
<script src="https://cdn.tailwindcss.com"></script>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
<script>
tailwind.config = {
theme: {
extend: {
fontFamily: {
sans: ['Inter', 'sans-serif'],
mono: ['JetBrains Mono', 'monospace'],
},
colors: {
primary: '#4f46e5', // Indigo 600
success: '#10b981', // Emerald 500
danger: '#f43f5e', // Rose 500
surface: '#f8fafc', // Slate 50
border: '#e2e8f0', // Slate 200
}
}
}
}
</script>
<style>
.grid-cell { font-family: 'JetBrains Mono', monospace; font-size: 13px; }
.dense-padding { padding: 4px 8px; }
.shimmer {
background: linear-gradient(90deg, #f1f5f9 25%, #e2e8f0 50%, #f1f5f9 75%);
background-size: 200% 100%;
animation: shimmer 1.5s infinite;
}
@keyframes shimmer {
0% { background-position: 200% 0; }
100% { background-position: -200% 0; }
}
</style>
</head>
<body class="bg-slate-50 font-sans text-slate-900 flex h-screen overflow-hidden">
<!-- Sidebar Simulation -->
<aside class="w-64 border-r border-slate-200 bg-white flex flex-col">
<div class="p-6 border-b border-slate-200">
<h1 class="text-xl font-bold text-indigo-600 flex items-center gap-2">
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-bar-chart-big"><path d="M3 3v18h18"/><rect width="4" height="7" x="7" y="10" rx="1"/><rect width="4" height="12" x="15" y="5" rx="1"/></svg>
Data_analysis
</h1>
</div>
<nav class="p-4 flex-1 space-y-1">
<a href="#" class="flex items-center gap-3 px-3 py-2 text-sm font-medium text-slate-900 bg-slate-100 rounded-md">
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="18" height="18" x="3" y="3" rx="2"/><path d="M3 9h18"/><path d="M9 3v18"/></svg>
Workspace (Grid)
</a>
<a href="#" class="flex items-center gap-3 px-3 py-2 text-sm font-medium text-slate-600 hover:bg-slate-50 rounded-md transition-colors">
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M3 3v18h18"/><path d="m19 9-5 5-4-4-3 3"/></svg>
Regressions
</a>
<a href="#" class="flex items-center gap-3 px-3 py-2 text-sm font-medium text-slate-600 hover:bg-slate-50 rounded-md transition-colors">
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14.5 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V7.5L14.5 2z"/><polyline points="14 2 14 8 20 8"/></svg>
Reports
</a>
</nav>
<div class="p-4 border-t border-slate-200">
<div class="bg-slate-100 p-3 rounded-lg text-xs space-y-2">
<p class="font-semibold text-slate-500 uppercase tracking-wider">System Status</p>
<div class="flex items-center gap-2">
<span class="w-2 h-2 rounded-full bg-success"></span>
<span>Python Backend: IDLE</span>
</div>
</div>
</div>
</aside>
<!-- Main Workspace -->
<main class="flex-1 flex flex-col overflow-hidden bg-white">
<!-- Top Toolbar -->
<header class="h-14 border-b border-slate-200 flex items-center justify-between px-6">
<div class="flex items-center gap-4">
<span class="text-sm font-semibold text-slate-500">Project:</span>
<span class="text-sm font-medium">Production_Quality_Jan2026.xlsx</span>
<span class="px-2 py-0.5 bg-indigo-50 text-indigo-700 text-[10px] font-bold rounded uppercase tracking-wider">Stateless Session</span>
</div>
<div class="flex items-center gap-3">
<button class="flex items-center gap-2 px-3 py-1.5 text-xs font-medium text-slate-600 hover:bg-slate-50 border border-slate-200 rounded transition-colors">
<svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"/><polyline points="7 10 12 15 17 10"/><line x1="12" x2="12" y1="15" y2="3"/></svg>
Download PDF
</button>
<button class="bg-indigo-600 text-white px-4 py-1.5 text-xs font-semibold rounded hover:bg-indigo-700 transition-shadow shadow-sm shadow-indigo-200">
Run Regression
</button>
</div>
</header>
<!-- The "Smart Grid" -->
<div class="flex-1 overflow-auto bg-slate-50 relative">
<table class="w-full border-separate border-spacing-0 bg-white">
<thead class="sticky top-0 bg-white z-10">
<tr>
<th class="border-b border-r border-slate-200 dense-padding bg-slate-50 w-10"></th>
<th class="border-b border-r border-slate-200 dense-padding text-left group">
<div class="flex flex-col gap-1">
<div class="flex items-center justify-between text-[11px] text-slate-500">
<span class="font-mono uppercase">C1</span>
<span class="bg-blue-50 text-blue-600 px-1 rounded">Num</span>
</div>
<span class="text-sm">Temperature_C</span>
<div class="h-4 flex items-end gap-0.5 mt-1">
<div class="bg-indigo-200 w-full h-[20%]"></div>
<div class="bg-indigo-200 w-full h-[40%]"></div>
<div class="bg-indigo-400 w-full h-[90%]"></div>
<div class="bg-indigo-400 w-full h-[100%]"></div>
<div class="bg-indigo-200 w-full h-[30%]"></div>
</div>
</div>
</th>
<th class="border-b border-r border-slate-200 dense-padding text-left group relative">
<div class="flex flex-col gap-1">
<div class="flex items-center justify-between text-[11px] text-slate-500">
<span class="font-mono uppercase">C2</span>
<span class="bg-blue-50 text-blue-600 px-1 rounded">Num</span>
</div>
<span class="text-sm">Pressure_Bar</span>
<div class="h-4 flex items-end gap-0.5 mt-1">
<div class="bg-indigo-200 w-full h-[60%]"></div>
<div class="bg-indigo-400 w-full h-[100%]"></div>
<div class="bg-indigo-200 w-full h-[40%]"></div>
<div class="bg-rose-400 w-full h-[10%]"></div> <!-- Outlier peak -->
</div>
</div>
<!-- Warning Badge -->
<div class="absolute -top-1 -right-1 bg-rose-500 text-white w-4 h-4 rounded-full flex items-center justify-center text-[10px] font-bold shadow-sm cursor-pointer hover:scale-110 transition-transform">!</div>
</th>
<th class="border-b border-r border-slate-200 dense-padding text-left">
<div class="flex flex-col gap-1">
<div class="flex items-center justify-between text-[11px] text-slate-500">
<span class="font-mono uppercase">C3</span>
<span class="bg-amber-50 text-amber-600 px-1 rounded">Cat</span>
</div>
<span class="text-sm">Machine_ID</span>
<div class="flex gap-1 mt-1">
<span class="w-full h-1 bg-slate-200 rounded"></span>
<span class="w-full h-1 bg-slate-200 rounded"></span>
<span class="w-full h-1 bg-slate-200 rounded"></span>
</div>
</div>
</th>
<th class="border-b border-slate-200 dense-padding text-left bg-indigo-50/50">
<div class="flex flex-col gap-1">
<div class="flex items-center justify-between text-[11px] text-indigo-500">
<span class="font-mono uppercase">Target</span>
<span class="bg-indigo-100 text-indigo-600 px-1 rounded">Y</span>
</div>
<span class="text-sm font-bold text-indigo-900">Yield_Output</span>
<div class="h-4 flex items-end gap-0.5 mt-1">
<div class="bg-indigo-400 w-full h-[100%]"></div>
<div class="bg-indigo-300 w-full h-[80%]"></div>
<div class="bg-indigo-200 w-full h-[50%]"></div>
</div>
</div>
</th>
</tr>
</thead>
<tbody>
<!-- Row 1 -->
<tr class="hover:bg-slate-50 transition-colors cursor-pointer group">
<td class="border-b border-r border-slate-100 dense-padding text-center text-[10px] text-slate-400 font-mono">1</td>
<td class="border-b border-r border-slate-100 dense-padding grid-cell">24.50</td>
<td class="border-b border-r border-slate-100 dense-padding grid-cell">1.02</td>
<td class="border-b border-r border-slate-100 dense-padding text-[12px]">
<span class="bg-slate-100 px-2 py-0.5 rounded text-slate-600">MAC-01</span>
</td>
<td class="border-b border-slate-100 dense-padding grid-cell font-bold bg-indigo-50/20">98.2</td>
</tr>
<!-- Row 2: OUTLIER -->
<tr class="bg-rose-50 transition-colors cursor-pointer group">
<td class="border-b border-r border-rose-100 dense-padding text-center text-[10px] text-rose-400 font-mono">2</td>
<td class="border-b border-r border-rose-100 dense-padding grid-cell">24.52</td>
<td class="border-b border-r border-rose-100 dense-padding grid-cell font-bold text-rose-600 bg-rose-100/50">9.99*</td>
<td class="border-b border-r border-rose-100 dense-padding text-[12px]">
<span class="bg-slate-100 px-2 py-0.5 rounded text-slate-600">MAC-01</span>
</td>
<td class="border-b border-rose-100 dense-padding grid-cell font-bold bg-indigo-50/20 opacity-50">45.1</td>
</tr>
<!-- Row 3 -->
<tr class="hover:bg-slate-50 transition-colors cursor-pointer group">
<td class="border-b border-r border-slate-100 dense-padding text-center text-[10px] text-slate-400 font-mono">3</td>
<td class="border-b border-r border-slate-100 dense-padding grid-cell">24.48</td>
<td class="border-b border-r border-slate-100 dense-padding grid-cell">1.01</td>
<td class="border-b border-r border-slate-100 dense-padding text-[12px]">
<span class="bg-slate-100 px-2 py-0.5 rounded text-slate-600">MAC-02</span>
</td>
<td class="border-b border-slate-100 dense-padding grid-cell font-bold bg-indigo-50/20">97.9</td>
</tr>
<!-- Row 4: LOADING Simulation -->
<tr class="">
<td class="border-b border-r border-slate-100 dense-padding text-center text-[10px] text-slate-400 font-mono">4</td>
<td class="border-b border-r border-slate-100 p-2"><div class="h-4 shimmer rounded w-16"></div></td>
<td class="border-b border-r border-slate-100 p-2"><div class="h-4 shimmer rounded w-16"></div></td>
<td class="border-b border-r border-slate-100 p-2"><div class="h-4 shimmer rounded w-20"></div></td>
<td class="border-b border-slate-100 p-2"><div class="h-4 shimmer rounded w-12"></div></td>
</tr>
</tbody>
</table>
</div>
<!-- Floating Insight Panel (Simulation) -->
<div class="absolute right-6 top-20 w-80 bg-white border border-slate-200 rounded-xl shadow-2xl shadow-indigo-100 overflow-hidden flex flex-col animate-slide-in">
<div class="p-4 bg-indigo-600 text-white flex items-center justify-between">
<h3 class="font-bold flex items-center gap-2">
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="12" x2="12" y1="8" y2="12"/><line x1="12" x2="12.01" y1="16" y2="16"/></svg>
Smart Insights
</h3>
<button class="opacity-70 hover:opacity-100">&times;</button>
</div>
<div class="p-5 space-y-4">
<div class="space-y-1">
<p class="text-[10px] uppercase font-bold text-slate-400 tracking-tight">Detected Anomalies</p>
<p class="text-sm text-slate-700">Found <span class="font-bold text-rose-600">34 outliers</span> in column <span class="font-mono bg-slate-100 px-1 rounded text-xs">Pressure_Bar</span>.</p>
</div>
<div class="bg-slate-50 border border-slate-100 rounded-lg p-3 space-y-2">
<p class="text-xs text-slate-500 font-medium italic">Why? Values are > 3.5 standard deviations from the mean (9.99 bar vs avg 1.05 bar).</p>
<div class="flex items-center gap-2">
<button class="bg-rose-100 text-rose-700 px-3 py-1.5 rounded text-[11px] font-bold hover:bg-rose-200 transition-colors flex-1">Exclude Data</button>
<button class="bg-white border border-slate-200 text-slate-600 px-3 py-1.5 rounded text-[11px] font-bold hover:bg-slate-50 transition-colors">Ignore</button>
</div>
</div>
<div class="pt-2 border-t border-slate-100">
<p class="text-[10px] uppercase font-bold text-slate-400 tracking-tight mb-2">Impact on Model</p>
<div class="flex items-center justify-between mb-1">
<span class="text-xs text-slate-600 font-medium">R-Squared (Current)</span>
<span class="text-xs font-mono font-bold">0.65</span>
</div>
<div class="flex items-center justify-between">
<span class="text-xs text-slate-600 font-medium">R-Squared (Post-fix)</span>
<span class="text-xs font-mono font-bold text-success">0.82 (+26%)</span>
</div>
</div>
</div>
</div>
<!-- Notification Toast -->
<div class="absolute bottom-6 left-1/2 -translate-x-1/2 bg-slate-900 text-white px-6 py-3 rounded-full shadow-lg flex items-center gap-3 text-sm animate-bounce">
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="success" stroke-width="3"><polyline points="20 6 9 17 4 12"/></svg>
Dataset "Production_Jan" loaded with 52,430 rows successfully.
</div>
</main>
</body>
</html>