You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# DataForge AI (Web Edition)
DataForge AI is a web-based data-quality and cleaning application built with React (frontend) and Flask + pandas (backend), with optional local LLM reasoning via Ollama.
## What It Does
- Upload CSV or Excel files (with robust handling of messy, real-world data)
- Profile dataset quality (missing values, duplicates, anomalies)
- Generate AI-assisted insights and cleaning suggestions
- Apply deterministic cleaning rules via backend `/process`
- Show cleaned dataset preview in Results
- Export cleaned dataset CSV from Results
- Auto-format email syntax, date formats, and numeric types
- Data drift protection: train models, run predictions, detect drift (KS test), and retrain
## Architecture
```text
React + Vite Web UI
|
| Axios HTTP
v
Flask API (python/app.py)
|- /health
|- /upload
|- /analyze
|- /process
|- /train
|- /predict
|- /detect-drift
|- /retrain
|
| Optional local LLM call
v
Ollama (localhost:11434)
```
## Stack
- Frontend: React, TypeScript, Vite
- Backend: Flask, pandas, numpy
- AI: Ollama local API (`/api/generate`)
- ML/anomaly: deterministic fallback + numeric outlier checks
## Setup
### 1. Install Node dependencies
```bash
npm install
```
### 2. Install Python dependencies
```bash
pip install -r python/requirements.txt
```
Recommended: use a virtual environment (`.venv`).
### 3. (Optional) Install and run Ollama
```bash
ollama serve
ollama pull smollm
```
The app works without Ollama by using fallback analysis.
## Run (Website + API)
### Terminal 1: start backend API
From project root:
```bash
python python/app.py
```
or with venv Python:
```bash
.venv/Scripts/python.exe python/app.py
```
### Terminal 2: start web app
```bash
npm run dev
```
Then open the printed local Vite URL in your browser (default: `http://127.0.0.1:8087`).
## Build for Web
```bash
npm run build
npm run preview
```
## API Endpoints
- `GET /health`
- `POST /upload`
- `POST /analyze`
- `POST /process`
- `POST /train`
- `POST /predict`
- `POST /detect-drift`
- `POST /retrain`
## Data Requirements
- Formats: CSV, XLSX
- Row limit: up to 50,000 rows per upload
- Handles mixed data types, invalid numerics/dates, missing values, outliers, duplicates
## Troubleshooting
### Backend connection error
1. Ensure backend is running: `python python/app.py`
2. Check `http://127.0.0.1:5000/health`
3. Verify firewall allows local loopback to `127.0.0.1:5000`
### Upload fails with max row error
- Dataset is over 50,000 rows
- Split dataset or reduce rows
### Results show raw data
- Run `Fix & Prepare Data` in Processing step to trigger `/process`
## Notes
- Session state is per browser tab/session storage.
- On refresh/new browser session, upload state resets.
- Desktop packaging files may still exist in the repo for legacy compatibility, but the active workflow is website-only.
# DataDriftProtection
