df1f208cd1
docs: requirements.md, offline toast, showWarning in base
...
- Add requirements.md documenting all changes
- Add showWarning() to base.html with .toast-warning CSS
- Add offline tile fallback message when all providers fail
- All vendor assets local, GeoIP from local file
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-23 13:34:53 +08:00
38e8985718
Merge branch 'master' of https://gitea.cattysteve.top/HJQ/tianxuan into beta
2026-07-22 23:38:45 +08:00
46013afa74
feat: add simple analysis module with global map visualization
...
- Add independent 'simple_analysis' Django app with full workflow:
1. Upload/merge CSV files with auto column detection
2. Filter by cnam + cnrs + isrs (optional)
3. Clustering via HDBSCAN+haversine (geo) or /24 subnet (IP)
4. Interactive Leaflet global map with cluster-colored data points
- Integrate into nav bar, settings, and URL routing without
modifying existing analysis logic
- Cluster detail panel with IPs, top dst IPs, TLS/cipher profiles
- Chart.js bar/pie charts for cluster size distribution
- E2E Playwright tests for all workflow steps
- Support multiple CSV upload with diagonal_relaxed merge
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 22:39:16 +08:00
29484ef033
fix: defer background processing until all upload batches complete (race condition fix)
...
Add finalize_upload endpoint that starts _background_process only after
all file batches have been saved to disk. Fixes race where glob(*.csv)
captured only first batch's files.
- upload_csv: no longer starts thread, sets status='pending'
- finalize_upload: validates dir, starts processing thread
- upload.html: calls finalize after batch upload loop completes
2026-07-22 22:34:54 +08:00
7b9084b33a
chore: clean PCA references, integrate SVD pipeline, update AGENTS.md
2026-07-22 21:10:13 +08:00
0264bade2e
refactor: Wave 2 - auto.html remove dataset selector + timeline merge + file_count fix
2026-07-22 20:46:10 +08:00
2a64f554c4
fix: Wave 1 - 3 MCP crashes + delete whitelist + exception propagation
2026-07-22 20:41:25 +08:00
f18f0d84cf
fix: reduce upload batch size to 5 files
2026-07-22 13:48:23 +08:00
d20b1c3b28
v1.1.3: default server 0.0.0.0:80 + row-level clustering + filter UI + dataset selector
2026-07-21 14:15:56 +08:00
ec83d5df85
v1.1.3: row-level clustering + dataset selector with status filter
...
- Add cluster_mode toggle (raw row-level vs entity-aggregated) in manual.html
- manual_run_analysis supports cluster_mode='raw' skipping entity aggregation
- Replace simple dataset dropdown with selector showing status badges and filter tabs
- Filter tabs: 全部/待分析/已分析/错误, default 待分析
- E2E verified: raw mode entity_count=total_flows, entity mode regression OK
2026-07-21 14:00:18 +08:00
745b1481cd
v1.1.3: entity_count fix + geoip database + multi-file test data + csv_glob display fix
...
- Fix entity_count not populated after clustering pipeline
- Create comprehensive geoip_data.txt (803 IP ranges, 49 cities)
- Generate 100 multi_upload test files with missing columns
- Replace csv_glob path with SQLite table name in frontend
- Fix globe zoom-drag point-fixing (camera.z/12.0 scaling)
- Fix orphaned manual_run_analysis function
2026-07-21 13:21:20 +08:00
917e065609
fix: globe zoom-drag point fixing + pipeline manual_run_analysis bug
...
- Globe drag now scales rotation by camera.z/12.0 so the point under cursor stays fixed at all zoom levels (drag AND inertia)
- Fixed orphaned manual_run_analysis function that was trapped inside _run_clustering_pipeline's except handler, causing NameError
- Updated urls.py route to correct function name
2026-07-21 12:52:44 +08:00
3617b7aabc
release: v1.1.3 — 13 fixes across build, frontend, pipeline, and data handling
...
Build: fix incremental package (update_info.json SHA256, __delete__, test exclude, files/ prefix), fix bump_version for 3-segment, PYTHONUTF8 in update.bat
Frontend: upload→manual redirect, LLM→cluster_overview redirect, filter UI before clustering, 地球→态势 rename, fix globe drag direction+inertia
Pipeline: LLM→auto clustering+PCA, column union handling (keep extra cols, no dropping), FFT spectral analysis tool
Quality: fix 24 bare except:pass patterns, enrich LLM context with TlsDB metadata, auto-save workflows with pinning, TlsDB-based column types
2026-07-21 12:01:52 +08:00
9314ac6942
chore: bump version to v1.1.2
...
- New timeline UI: interleaved LLM reasoning + tool call display
- Fix: run_log truncation for large tool outputs
- Fix: lat/lon + values reduced to ~1% (was 99%)
- Fix: @csrf_exempt on upload/delete endpoints
- Fix: DATA_UPLOAD_MAX_NUMBER_FILES for 1998-file upload
- Fix: dtype alignment for multi-batch CSV processing
- New: tests/test_e2e_full.py with real DeepSeek LLM
- New: 1998-file test data generator
2026-07-20 23:32:32 +08:00
8fa36b774e
v7: 19 issues fixed — SQLite storage, new clustering pipeline, display_id, globe rewrite, Chinese tools
...
Key changes:
- New: data_loader.py SQLite persistence with drop_sqlite_table
- New: db_utils.py retry_on_lock decorator (3 retries, exponential backoff)
- New: tool_registry.py with 27 MCP tools (filter_and_cluster, compute_scores, etc.)
- New: tls_ref.py for TLS cipher/reference data
- New: import_tlsdb.py management command
- New: scripts/start_server.py for portable runtime
- New: migrations 0003-0007 for SQLite table, display_id, llm fields
- Changed: views.py unified pipeline worker, retry_run, display_id everywhere
- Changed: models.py with display_id auto-assignment, run_type, sqlite_table, llm_thinking, tool_calls_json
- Changed: urls.py added retry_run route
- Changed: session_store.py robust JSON persistence
- Changed: AGENTS.md v7 fix summary added
- Changed: templates — globe rewrite (inertia/polar flip), auto.html (thinking/tool accordions), base.html (toast/config), all pages use display_id
- Changed: run.bat PYTHONUTF8=1
- Deleted: entity_detector.py, entity_aggregator.py (replaced by filter_and_cluster clustering pipeline)
- Test: 92/92 unit tests passing
2026-07-20 13:33:13 +08:00
02f18944d5
fix: manual page dataset selector uses dataset_id not ds.id
2026-07-19 23:33:06 +08:00
6e89107a30
feat: save/load workflow plans + run all sequentially + step reorder (manual workflow builder)
2026-07-19 23:15:50 +08:00
4fb1e4fdda
refactor: integrate 27 MCP tools into manual analysis workflow builder (remove separate tool_lab)
2026-07-19 23:12:16 +08:00
06db287c0e
feat: MCP tool lab page + workflow docs + nav link
2026-07-19 23:09:13 +08:00
6ed109dbfe
perf: PCA downsample to 50K for 10M+ entities + directory upload via webkitdirectory
2026-07-19 21:46:34 +08:00
b90dcbf11c
fix: cluster charts cap at 50 bars + silhouette label fix + manual page progress bar
2026-07-19 21:36:51 +08:00
3e98acecf9
fix: lower arc to R=5.06 e=0.65 (clearance 0.06, not floating)
2026-07-19 21:24:53 +08:00
3c63fe9d23
fix: globe run selection fallback + arc start height 5.2 / ecc 0.85 to avoid earth intersection
2026-07-18 21:51:26 +08:00
3664d2d8ee
fix: use direct parametric ellipse equation for trail sphere animation
2026-07-18 21:44:57 +08:00
b1bd7ad5bb
fix: replace QuadraticBezierCurve with centered elliptical arc (no earth intersection)
2026-07-18 21:43:56 +08:00
aab354cc51
v4: progress bar position + sklearn warning + N+1 fix + PRAGMA opt + zero-var fallback
2026-07-17 19:07:57 +08:00
45d9f508ab
feat: add proxy/anomaly detection scoring + retry analysis for failed runs
2026-07-17 13:33:02 +08:00
4e6339253d
fix(globe): arc sphere size independent of camera zoom using inverse distance scaling
2026-07-17 13:27:06 +08:00
1af5704796
fix(cluster_overview): responsive charts with adaptive bandwidth and overflow-safe legend
2026-07-17 13:25:08 +08:00
b58d01c7f2
cleanup: remove v6 garbage, restore original files, unify views.py workers
...
- Delete 15 garbage scripts (wnl_to_user_csv, eval_clustering, debug_db2/3,
debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_*,
test_e2e_full, test_32b, test_chinese_path, verify_runtime)
- Delete data/wnl_converted.csv
- Delete analysis/value_normalizer.py + tests/test_value_normalizer.py
- Remove normalize_lf references from data_loader.py
- Remove _norm column lookups from entity_aggregator.py
- Fix settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB -> 100MB
- Fix upload.html: add per-file X delete button + removeFile()
- Refactor views.py: unify _run_analysis_worker/_run_llm_analysis into
_run_pipeline_worker with callback pattern
- Wrap sync ORM call in tool_registry.py with sync_to_async
- Update run_pipeline.py to use _run_pipeline_worker
- Rebuild gen_test_data.py as simple version (replace 3-profile v6 version)
- Add runtime/ db.sqlite3 logs/ .omo/ to .gitignore
- Delete old .omo/plans/ plan files (tianxiu-v4/v5/v6, globe-v6)
- 120/120 tests passing
2026-07-16 22:53:07 +08:00
814f29645a
chore: initial commit of 天璇 project state before cleanup
2026-07-16 22:41:26 +08:00
TianXuan Developer
PM-pinou