8ba99292b7
fix: MMDB中文路径解决方案 + 聚类优先使用最新GeoIP数据
...
- geoip.py: maxminddb C库不支持Windows中文路径，复制MMDB到ASCII临时目录
- views.py: _cluster_network 优先用 geoip_lookup()(MMDB) 而非上传时缓存(text)
- 效果: IP地理坐标从 ~55城市粗定位 → 全球百万级精确城市定位
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-23 23:59:27 +08:00
e0a7400bd7
feat: 高精度离线GeoIP — MMDB支持 + 中文路径修复 + 分块自解压
...
- geoip.py: 修复中文路径 str()→Path 编码问题 (maxminddb Latin-1 bug)
- geoip.py: 启动自动重组geoip_chunks分块 + gunzip解压
- scripts/setup_geoip.py: 手动重组/解压脚本
- .gitignore: 忽略解压后的 .mmdb 文件
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-23 23:42:38 +08:00
32c15f5365
feat: v3 refactor with MMDB GeoIP, network clustering, node/edge viz, offline tile cache, dual export
...
- GeoIP: MMDB (GeoLite2-City + ASN) support with text fallback
- Upload: two-phase flow with advanced preset filter builder
- Clustering: IP communication network community discovery (/24 + HDBSCAN)
- Map: per-IP node rendering, node click detail, edge click detail
- Offline: Cache API tile caching with fallback
- Export: node table + edge table CSV download
- Fix: null-guard DOM operations throughout
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-23 19:35:45 +08:00
7d6d29475d
fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True)
2026-07-22 23:39:37 +08:00
035d22b234
fix: make all entity_values unique by appending row index to support bulk_create with ignore_conflicts
2026-07-22 23:17:09 +08:00
0d92a5a170
fix: unique entity_value fallback to prevent bulk_create ignore_conflicts from skipping duplicates
2026-07-22 23:14:00 +08:00
c1db7ad7eb
fix: fallback entity value when column is empty to ensure all UMAP profiles are saved
2026-07-22 23:11:32 +08:00
2c5aece65d
fix: streaming load_from_db with fetchmany(10000) instead of fetchall() (fixes OOM on 4M rows)
2026-07-22 22:39:51 +08:00
29484ef033
fix: defer background processing until all upload batches complete (race condition fix)
...
Add finalize_upload endpoint that starts _background_process only after
all file batches have been saved to disk. Fixes race where glob(*.csv)
captured only first batch's files.
- upload_csv: no longer starts thread, sets status='pending'
- finalize_upload: validates dir, starts processing thread
- upload.html: calls finalize after batch upload loop completes
2026-07-22 22:34:54 +08:00
2c8d1e90ae
fix: per-file CSV scan for heterogeneous schemas, add upload file count logging
2026-07-22 22:31:49 +08:00
7b9084b33a
chore: clean PCA references, integrate SVD pipeline, update AGENTS.md
2026-07-22 21:10:13 +08:00
585dbb4ec4
refactor: Wave 3 - UMAP-2D replace PCA scatter, TruncatedSVD replace PCA preprocessing, update AGENTS.md
2026-07-22 21:07:03 +08:00
0264bade2e
refactor: Wave 2 - auto.html remove dataset selector + timeline merge + file_count fix
2026-07-22 20:46:10 +08:00
2a64f554c4
fix: Wave 1 - 3 MCP crashes + delete whitelist + exception propagation
2026-07-22 20:41:25 +08:00
e4689ed0a9
fix: batch upload no longer re-triggers _background_process to avoid DROP TABLE race
2026-07-22 14:01:56 +08:00
8c90bad9f0
v1.1.6: batch upload BATCH=5 + temp dir to D: + schema sort + update.bat fix
2026-07-22 13:58:19 +08:00
9ca5da1237
release: v1.1.6 — update.bat wildcard fix + schema column-order fix + gen_test format alignment + pipeline verify
2026-07-22 11:39:35 +08:00
6a633016d2
v1.1.4: gen_test reads TlsDB coverage + schema fixes
...
- gen_test_data.py: load_coverage() from TlsDB.csv, replaced all hardcoded _blank_or rates with coverage lookups
- gen_multi.py: coverage-based per-file column dropping
- collect_schema: force Utf8 for cross-file compatibility
- _background_process: pre-scan union columns, single CREATE TABLE, no ALTER TABLE
2026-07-21 19:34:14 +08:00
d20b1c3b28
v1.1.3: default server 0.0.0.0:80 + row-level clustering + filter UI + dataset selector
2026-07-21 14:15:56 +08:00
ec83d5df85
v1.1.3: row-level clustering + dataset selector with status filter
...
- Add cluster_mode toggle (raw row-level vs entity-aggregated) in manual.html
- manual_run_analysis supports cluster_mode='raw' skipping entity aggregation
- Replace simple dataset dropdown with selector showing status badges and filter tabs
- Filter tabs: 全部/待分析/已分析/错误, default 待分析
- E2E verified: raw mode entity_count=total_flows, entity mode regression OK
2026-07-21 14:00:18 +08:00
9de1d93146
fix: entity_count fallback + globe SQLite loading
...
- entity_count: fallback from ClusterResult sizes when PCA is skipped
- globe_view: load from SQLite table (_data_N) instead of csv_glob path
2026-07-21 13:26:50 +08:00
745b1481cd
v1.1.3: entity_count fix + geoip database + multi-file test data + csv_glob display fix
...
- Fix entity_count not populated after clustering pipeline
- Create comprehensive geoip_data.txt (803 IP ranges, 49 cities)
- Generate 100 multi_upload test files with missing columns
- Replace csv_glob path with SQLite table name in frontend
- Fix globe zoom-drag point-fixing (camera.z/12.0 scaling)
- Fix orphaned manual_run_analysis function
2026-07-21 13:21:20 +08:00
917e065609
fix: globe zoom-drag point fixing + pipeline manual_run_analysis bug
...
- Globe drag now scales rotation by camera.z/12.0 so the point under cursor stays fixed at all zoom levels (drag AND inertia)
- Fixed orphaned manual_run_analysis function that was trapped inside _run_clustering_pipeline's except handler, causing NameError
- Updated urls.py route to correct function name
2026-07-21 12:52:44 +08:00
3617b7aabc
release: v1.1.3 — 13 fixes across build, frontend, pipeline, and data handling
...
Build: fix incremental package (update_info.json SHA256, __delete__, test exclude, files/ prefix), fix bump_version for 3-segment, PYTHONUTF8 in update.bat
Frontend: upload→manual redirect, LLM→cluster_overview redirect, filter UI before clustering, 地球→态势 rename, fix globe drag direction+inertia
Pipeline: LLM→auto clustering+PCA, column union handling (keep extra cols, no dropping), FFT spectral analysis tool
Quality: fix 24 bare except:pass patterns, enrich LLM context with TlsDB metadata, auto-save workflows with pinning, TlsDB-based column types
2026-07-21 12:01:52 +08:00
9314ac6942
chore: bump version to v1.1.2
...
- New timeline UI: interleaved LLM reasoning + tool call display
- Fix: run_log truncation for large tool outputs
- Fix: lat/lon + values reduced to ~1% (was 99%)
- Fix: @csrf_exempt on upload/delete endpoints
- Fix: DATA_UPLOAD_MAX_NUMBER_FILES for 1998-file upload
- Fix: dtype alignment for multi-batch CSV processing
- New: tests/test_e2e_full.py with real DeepSeek LLM
- New: 1998-file test data generator
2026-07-20 23:32:32 +08:00
efc6e3ab9d
v8: fix server startup, CSV parse, upload flow, load_from_db types, integration test
2026-07-20 19:40:49 +08:00
0b6672781f
feat(settings): migrate db.sqlite3 to APPDATA dir, extract shared appdata.py
...
Migrate database path from project-local db.sqlite3 to %%APPDATA%%/TianXuan/ so data persists across updates.
- tianxuan/settings.py: DB path to APPDATA
- tianxuan/wsgi.py: use settings.DATABASES path instead of hardcoded db.sqlite3
- analysis/appdata.py: extract shared APPDATA path resolution
- analysis/session_store.py: use appdata.py for session store path
- analysis/views.py: use appdata.py for view paths
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent )
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai >
2026-07-20 14:14:07 +08:00
8fa36b774e
v7: 19 issues fixed — SQLite storage, new clustering pipeline, display_id, globe rewrite, Chinese tools
...
Key changes:
- New: data_loader.py SQLite persistence with drop_sqlite_table
- New: db_utils.py retry_on_lock decorator (3 retries, exponential backoff)
- New: tool_registry.py with 27 MCP tools (filter_and_cluster, compute_scores, etc.)
- New: tls_ref.py for TLS cipher/reference data
- New: import_tlsdb.py management command
- New: scripts/start_server.py for portable runtime
- New: migrations 0003-0007 for SQLite table, display_id, llm fields
- Changed: views.py unified pipeline worker, retry_run, display_id everywhere
- Changed: models.py with display_id auto-assignment, run_type, sqlite_table, llm_thinking, tool_calls_json
- Changed: urls.py added retry_run route
- Changed: session_store.py robust JSON persistence
- Changed: AGENTS.md v7 fix summary added
- Changed: templates — globe rewrite (inertia/polar flip), auto.html (thinking/tool accordions), base.html (toast/config), all pages use display_id
- Changed: run.bat PYTHONUTF8=1
- Deleted: entity_detector.py, entity_aggregator.py (replaced by filter_and_cluster clustering pipeline)
- Test: 92/92 unit tests passing
2026-07-20 13:33:13 +08:00
763bc144ea
test: full-link E2E test - 120 unit + 8 pipeline + 10 tool integration tests pass
2026-07-19 23:19:51 +08:00
6e89107a30
feat: save/load workflow plans + run all sequentially + step reorder (manual workflow builder)
2026-07-19 23:15:50 +08:00
4fb1e4fdda
refactor: integrate 27 MCP tools into manual analysis workflow builder (remove separate tool_lab)
2026-07-19 23:12:16 +08:00
06db287c0e
feat: MCP tool lab page + workflow docs + nav link
2026-07-19 23:09:13 +08:00
7fbf5e2720
feat: add 5 read-only analysis tools for LLM deep-dive (patterns/temporal/tls-health/geo/entity-detail)
2026-07-19 23:06:53 +08:00
aa238b9439
feat: add 7 diagnostic MCP tools for edge-case data analysis (validate/explore/outliers/diagnose/compare/export/repair)
2026-07-19 22:54:19 +08:00
27dd736443
feat: add 3 composable MCP tools (compute_scores/detect_anomalies/visualize_anomalies) for LLM-driven workflow
2026-07-19 22:51:16 +08:00
2caa09ed21
feat: adaptive anomaly scoring via PCA weight learning + IQR thresholds (no hardcoded params)
2026-07-19 22:44:20 +08:00
62ce147a05
perf: lazy filter _ds_idx before collect + pre-collect downsampling in _handle_run_clustering
2026-07-19 21:51:42 +08:00
6ed109dbfe
perf: PCA downsample to 50K for 10M+ entities + directory upload via webkitdirectory
2026-07-19 21:46:34 +08:00
3c63fe9d23
fix: globe run selection fallback + arc start height 5.2 / ecc 0.85 to avoid earth intersection
2026-07-18 21:51:26 +08:00
7fd081d991
fix: add 11 missing TlsDB columns to type_classifier + entity_aggregator
2026-07-18 21:05:44 +08:00
e2e09b1efd
fix: globe view skip missing files + TLS/bytes column matching for 0ver/8ack
2026-07-18 20:57:21 +08:00
aab354cc51
v4: progress bar position + sklearn warning + N+1 fix + PRAGMA opt + zero-var fallback
2026-07-17 19:07:57 +08:00
fa305515c1
fix: D5 downsampling index mismatch + add 1M-row medium-scale test to docs
2026-07-17 14:02:30 +08:00
45d9f508ab
feat: add proxy/anomaly detection scoring + retry analysis for failed runs
2026-07-17 13:33:02 +08:00
548ff8ca1b
perf: replace per-row EntityProfile save with bulk_create (views.py + tool_registry.py)
2026-07-17 13:28:27 +08:00
03be495593
fix: lat/lon column name matching (:ips.latd) + cnrs/isrs boolean conversion
2026-07-17 13:22:34 +08:00
2bb3b7273a
xiufu: fix globe ?data=, add pipeline tests, update settings, rewrite AGENTS.md
...
- Fix globe_view ?data= 500 error (try/except safety + DATA_UPLOAD_MAX_NUMBER_FIELDS=1000000)
- Add test_pipeline.py (8 tests: 3 datasets × 2 algorithms + edge cases)
- Update AGENTS.md with all 18 issue statuses
- Clean up server processes
2026-07-16 23:54:19 +08:00
b271626abc
feat: add downsampling support (head parameter for manual + LLM)
2026-07-16 23:35:10 +08:00
50f4d025ce
feat: SessionStore persistence to JSON file + startup auto recovery
2026-07-16 23:33:19 +08:00
29e21a6a23
fix: remove tb[:200] truncation, add TLS_HEX_MAP to type_classifier, fix + FLOAT detection
2026-07-16 23:31:35 +08:00
TianXuan Developer
PM-pinou