Turn sql.md (multiple SQL blocks in markdown) into a runnable BI prototype:

• SQL parsing
• semantic inference
• DSL filter extraction (P0)
• chart recommendation
• dashboard spec generation
• backend + frontend service integration

• Skill pipeline: skills/sql-to-bi-builder/scripts/*
• Backend service:
  • full mode: FastAPI + DuckDB + Polars + Pandas (services/backend/app.py)
  • lite mode: stdlib HTTP fallback (services/backend/app_lite.py)
• Frontend service:
  • full mode: React + Vite + ECharts (services/frontend)
  • lite mode: static HTML/JS + ECharts CDN (services/frontend-lite)
• Test fixtures:
  • SQL markdown: testdata/sql/demo.sql.md
  • DuckDB init SQL: testdata/duckdb/init.sql
  • Demo DB file: testdata/duckdb/sql2bi_demo.duckdb

Use Python 3.12 for local development and integration testing.

Recommended setup with uv:

uv python pin 3.12
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -r requirements-dev.txt
uv pip install -r services/backend/requirements.txt

If uv is unavailable, use built-in venv:

python3.12 -m venv .venv312
source .venv312/bin/activate
python -m pip install -r requirements-dev.txt
python -m pip install -r services/backend/requirements.txt

Default ports:

• backend: 127.0.0.1:18000
• frontend: 127.0.0.1:15173
• skill-agent (LangChain + SSE): 127.0.0.1:18100

Start:

bash services/start_backend.sh
bash services/start_frontend.sh
bash services/start_skill_agent.sh

Behavior:

• If backend Python deps are installed, backend runs in full mode.
• Otherwise backend falls back to lite mode automatically.
• If frontend node_modules exists, frontend runs React/Vite mode.
• Otherwise frontend falls back to static lite mode automatically.

Run the skill pipeline (requires Python 3.12 runtime):

python3.12 skills/sql-to-bi-builder/scripts/run_pipeline.py \
  --input /abs/path/sql.md \
  --out /abs/path/out \
  --with-services

Generated files:

• query_catalog.json
• semantic_catalog.json (includes dsl_filters)
• chart_plan.json
• dashboard.json (includes page-level global_filters)
• ui/ static scaffold
• services/ generated service bundle

• GET /api/health
• GET /api/v1/datasources
• GET /api/v1/datasources/{id}
• POST /api/v1/datasources
• PUT /api/v1/datasources/{id}
• DELETE /api/v1/datasources/{id}
• POST /api/v1/datasources/{id}/test
• POST /api/v1/import/sql-md with body:
  • {"sql_md_path": "/abs/path/to/sql.md"}
• GET /api/v1/dashboard/current
• GET /api/v1/filters
• GET /api/v1/queries/{query_id}/data?include_filters=true&field=value
• GET /api/v1/queries/{query_id}/export.csv?include_filters=true&field=value
• POST /api/v1/reports/query/{query_id} with body:
  • {"theme":"growth-acquisition","version":"v20260311.001","filters":{"country":"CN"},"include_csv":true,"chart_png_data_url":"data:image/png;base64,..."}

Query data API now executes stored SQL against configured datasource (read-only only), and persists audit artifacts to:

• audit/<session_id>/sql.md
• audit/<session_id>/sql/<query_id>.sql
• audit/<session_id>/sql_audit_report.json

Report generation API persists versioned report artifacts to:

• reports/<theme>/<version>/report.md
• reports/<theme>/<version>/report.json
• reports/<theme>/<version>/analysis_trace.md
• reports/<theme>/<version>/evidence_index.json
• reports/<theme>/<version>/metadata.json
• optional exports under reports/<theme>/<version>/exports/ and reports/<theme>/<version>/charts/

Example import:

curl -X POST http://127.0.0.1:18000/api/v1/import/sql-md \
  -H 'Content-Type: application/json' \
  -d '{"sql_md_path":"/Users/lyg/software/sql2bi/testdata/sql/demo.sql.md"}'

Current backend integration tests cover:

• sql.md import -> query execution -> session SQL audit files persistence
• read-only SQL guard (DDL is blocked)
• CSV export + versioned report artifact persistence
• end-to-end path: sql.md -> pipeline -> backend API -> frontend-lite DOM render

Run with Python 3.12 environment:

python -m unittest tests/integration/test_backend_audit_integration.py

For frontend render e2e test, install JS test dependency first:

npm --prefix tests/e2e install
python -m unittest tests/integration/test_sqlmd_to_frontend_e2e.py

Skill-agent stream integration test:

python -m pip install -r services/skill_agent/requirements.txt
python -m unittest tests/integration/test_skill_agent_stream.py

Run:

python -m pip install -r services/skill_agent/requirements.txt
bash services/start_skill_agent.sh

Stream call example (text/event-stream / SSE):

curl -N -X POST "http://127.0.0.1:18100/api/v1/skills/stream" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "prompt":"请先构建 BI dashboard，再给出业务分析计划",
    "skills":["sql-to-bi-builder","starrocks-mcp-analyst"],
    "sql_md_path":"/Users/lyg/software/sql2bi/sample.sql.md"
  }'

Install repo hooks:

bash scripts/install-git-hooks.sh

Hooks:

• pre-commit: conflict markers, shell/python syntax, requirement pin checks
• pre-push: smoke pipeline run on sample.sql.md

This repo now includes a static project intro site in website/:

• website/index.html
• website/styles.css
• website/main.js

Local preview:

python3 -m http.server 8080 --directory website

Then open:

• http://127.0.0.1:8080

This repo includes workflow:

• .github/workflows/deploy-website-pages.yml

Behavior:

• Trigger on push to main when website/** changes
• Upload website/ as Pages artifact
• Deploy automatically to GitHub Pages

One-time setup:

1. In repository settings, open Pages.
2. Set Build and deployment / Source to GitHub Actions.
3. Push changes to website/ on main (or run workflow manually via Actions tab).

1. Connect your repository to Cloudflare Pages.
2. Set build command as empty, output directory as website.
3. Deploy and bind your domain if needed (free SSL included).