19 Commits

Author SHA1 Message Date
TianXuan Developer 829729943c feat: 社区交互增强 + 卫星地图 + 节点/边数据丰富 + 项目清理
核心改进:
- 节点数据丰富: TLS版本/加密套件/SNI/端口/流量统计/连接成功率
- 边数据丰富: 包数/持续时间/TLS/端口/SNI/分类标签
- 新增 edge_time_series 端点: 边流量时序图(北京时间)
- 新增 cluster_nodes_edges 端点: 社区聚焦(zoom+高亮+专属边)
- 社区点击: 自动缩放/高亮节点/显示社区边(可点击)
- 边交互修复: 事件从layerGroup移至hitPoly,三处边渲染全修复
- 地图升级: ESRI卫星影像 + 255国矢量边界叠加
- 离线回退: 卫星→CartoDB→暗色矢量地图,完整可用
- 节点面板: 删除无意义社区标签,移至Host+组织同行
- sanitizeName: 清理5个非ASCII国家名避免乱码
- .gitignore: 补充geoip_chunks/和嵌套CSV规则

项目清理:
- 删除 logs/ (20个调试文件,~2.5MB)
- 删除 test_*.py (4个临时测试脚本)
- 删除 _check_bool/_pipeline_test/_restart_server/start_server (硬编码路径)
- 删除 node_modules/package.json/package-lock.json (18MB,不用Node)
- 删除 tests/ 旧日志和调试脚本
- 删除 .pytest_cache/ .server_pid.txt RUNTIME_CHANGED.txt
- AGENTS.md → CLAUDE.md

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 13:06:14 +08:00
TianXuan Developer d4ec81a1dd feat: 修复世界地图 + 丰富节点/边数据 + 完善交互与导出
地图:
- 改用 CartoDB Positron 瓦片图层 (L.tileLayer),替代无效的 L.imageOverlay
- 复制 vendor 文件到项目级 static 目录

后端 (views.py):
- 新增 host 列别名 (host/hostname/host_name/remote_host/peer_name)
- _cluster_network 为每个 IP 收集 host、org、text、peer_ips 元数据
- 节点新增 host/org/texts/peer_ips/peer_count 字段
- 边新增 src/dst host/org/city/country/cluster + peak_hours/time_count 始终存在
- node_detail 返回 node_texts 和 enriched edges (peer_host/org/city)

前端 (simple_analysis.html):
- 实现 loadClusterDetail/showClusterDetailPanel 缺失函数
- renderCharts 在 goStep(4) 中调用
- 节点详情面板增强: host、org、text、对端 IP 列表
- 边详情面板增强: 端点 host/org/city/country、高峰时段
- 导出 CSV 增强: 节点新增 host/org/asn/peer_count/connected_ips,边新增全部端点字段
- Tooltip 新增 host 和 org 显示

测试: 更新 Playwright 测试适配两阶段上传 + 新 UI,全部 30 项通过

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 23:04:27 +08:00
TianXuan Developer dedb3fc0ae test: Playwright self-test passes all 8 checks + fix Beijing time
- Self-test: page load -> upload -> cluster -> map -> node click -> edge -> export
- All pass: 600 rows, 542 nodes, 535 markers, 300 edges, 542/500 CSV rows
- Fix toBeijing to handle .00000 microsecond timestamps

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 22:02:11 +08:00
TianXuan Developer e3a7cc20d0 test: Playwright E2E test for V3 simple analysis + fix initFilterBuilder
- Add comprehensive E2E test: upload → cluster → map → node/edge click → export
- Fix goStep initFilterBuilder guard (was !SA.colNames.length, now always called)
- All 9 test sections pass: upload, cluster, map, node click, edge display, edge click, dual export, charts

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 21:31:22 +08:00
TianXuan Developer 6f6581d38d fix: operator change handler enables/disables value dropdown
- Add onFilterOpChange handler on <select class=sa-filter-op>
- When operator is is_empty/not_empty: disable value select, show '— 无值 —'
- When switching back to eq/neq/contains/etc: re-enable and re-fetch values
- Fixes issue where changing operator before selecting column left value
  dropdown disabled permanently
- Remove operator check from onFilterColChange (now handled separately)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 13:14:03 +08:00
TianXuan Developer ab31471bd3 feat: value dropdown with top-50 by count + preset builder UI
Backend:
- advanced_filter supports mode='preset' — overwrites entry['df'] so
  preset filtering becomes the base data for subsequent steps

Frontend - Filter rows:
- Replace <input>+<datalist> with <select class=sa-filter-val>
- onFilterColChange populates dropdown with top 50 distinct values
  sorted by count, displayed as "value (count)"
- Disables value select for is_empty/not_empty operators

Frontend - Upload preset:
- Replace JSON textarea with full condition builder (addFilterCondition/
  addFilterGroup) — same format as advanced filter in Step 2
- Builder appears after upload with columns populated
- '应用预设筛选' button POSTs to filter/advanced/ with mode='preset'
- '清除预设' resets the builder
- Remove old upload-time preset_filters collection (now post-upload)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 13:04:23 +08:00
TianXuan Developer 8eacb38e9e feat: nested boolean group support in filter builder (parentheses logic)
Backend:
- Add _build_tree_condition() recursive function supporting AND/OR/NOT groups
- Modify advanced_filter to accept 'root' tree format alongside legacy flat format
- Tree supports arbitrary nesting depth: {type, logic, items}

Frontend:
- Add addFilterGroup() — creates visual ( ) group with own logic selector
- Add wrapRowInGroup() — wraps existing row into a group via [()]
- Add serializeFilterTree() — recursive JSON serialization
- Update applyAdvancedFilter to POST tree format
- Update updateFilterLogicTags for nested group labeling
- Switch filter row elements from IDs to class-based selectors
- CSS for group visual nesting (dashed left border, indentation)

Now supports: cnam='google' AND (isrs='+' OR cnrs='+')

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:42:17 +08:00
TianXuan Developer c5d9889a22 refactor: remove CNAM auto-selection from upload step
- Remove CNAM frequency display, auto-selection, and related CSS from upload UI
- Remove renderCnamList, selectCnam, runFilter functions (dead code)
- Remove topCnam from SA state object
- Keep preset filter cnam field for manual pre-filtering
- Users now use Step 2 advanced filter instead
- Update Playwright test assertion

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:24:38 +08:00
TianXuan Developer d81171cc1d test: Playwright E2E test for simple-analysis v2 + fix template JS syntax
- Fix extra closing brace in simple_analysis.html that broke all inline JS
- Add Playwright E2E test covering upload→filter→cluster→map→edges
- Add debug support script for Playwright issue isolation
- Use small 20-row test CSV for fast test execution
- All 31 assertions pass: page load, CSV upload, advanced filter,
  HDBSCAN clustering, Leaflet map, edge relationships

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:08:24 +08:00
PM-pinou 7d6d29475d fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True) 2026-07-22 23:39:37 +08:00
PM-pinou 1df4c3c2eb chore: bump VERSION to v2.0.0beta, reimport TlsDB, remove stale test files 2026-07-22 21:53:25 +08:00
PM-pinou 7aa847ba3e v1.1.6: cleanup — schema sort fix, batch upload, temp dir to D:, update.bat wildcard fix 2026-07-22 19:10:39 +08:00
PM-pinou 8c90bad9f0 v1.1.6: batch upload BATCH=5 + temp dir to D: + schema sort + update.bat fix 2026-07-22 13:58:19 +08:00
PM-pinou 9314ac6942 chore: bump version to v1.1.2
- New timeline UI: interleaved LLM reasoning + tool call display
- Fix: run_log truncation for large tool outputs
- Fix: lat/lon + values reduced to ~1% (was 99%)
- Fix: @csrf_exempt on upload/delete endpoints
- Fix: DATA_UPLOAD_MAX_NUMBER_FILES for 1998-file upload
- Fix: dtype alignment for multi-batch CSV processing
- New: tests/test_e2e_full.py with real DeepSeek LLM
- New: 1998-file test data generator
2026-07-20 23:32:32 +08:00
PM-pinou efc6e3ab9d v8: fix server startup, CSV parse, upload flow, load_from_db types, integration test 2026-07-20 19:40:49 +08:00
PM-pinou 7a670fd5e2 feat(scripts): add update.bat and rollback.bat launchers
Add user-facing launchers for the incremental update system:
- update.bat: one-click update launcher
- rollback.bat: rollback to previous version
- tests/test_update_integration.py: integration tests for the update system
- README.md: add incremental update guide (section 10)
- .gitignore: add .update_cache/ entry

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-07-20 14:14:17 +08:00
PM-pinou 2bb3b7273a xiufu: fix globe ?data=, add pipeline tests, update settings, rewrite AGENTS.md
- Fix globe_view ?data= 500 error (try/except safety + DATA_UPLOAD_MAX_NUMBER_FIELDS=1000000)
- Add test_pipeline.py (8 tests: 3 datasets × 2 algorithms + edge cases)
- Update AGENTS.md with all 18 issue statuses
- Clean up server processes
2026-07-16 23:54:19 +08:00
PM-pinou b58d01c7f2 cleanup: remove v6 garbage, restore original files, unify views.py workers
- Delete 15 garbage scripts (wnl_to_user_csv, eval_clustering, debug_db2/3,
  debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_*,
  test_e2e_full, test_32b, test_chinese_path, verify_runtime)
- Delete data/wnl_converted.csv
- Delete analysis/value_normalizer.py + tests/test_value_normalizer.py
- Remove normalize_lf references from data_loader.py
- Remove _norm column lookups from entity_aggregator.py
- Fix settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB -> 100MB
- Fix upload.html: add per-file X delete button + removeFile()
- Refactor views.py: unify _run_analysis_worker/_run_llm_analysis into
  _run_pipeline_worker with callback pattern
- Wrap sync ORM call in tool_registry.py with sync_to_async
- Update run_pipeline.py to use _run_pipeline_worker
- Rebuild gen_test_data.py as simple version (replace 3-profile v6 version)
- Add runtime/ db.sqlite3 logs/ .omo/ to .gitignore
- Delete old .omo/plans/ plan files (tianxiu-v4/v5/v6, globe-v6)
- 120/120 tests passing
2026-07-16 22:53:07 +08:00
PM-pinou 814f29645a chore: initial commit of 天璇 project state before cleanup 2026-07-16 22:41:26 +08:00