HJQ/tianxuan
2
1
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
178 Commits 5 Branches 0 Tags
master
Commit Graph

21 Commits

Author SHA1 Message Date
TianXuan Developer cd68d7c9af fix: gen_multi2 使用真实公网IP，GeoIP可解析 
根因: 服务器IP(203.0.113.x等RFC5737)和代理IP(10.0.x.x私有)无法被GeoIP解析
→ 500条边仅2条两端有坐标 → 图上不显示

修复:
- 服务器IP: 替换为 Akamai/Cloudflare/Google/AWS/Microsoft 公网IP
- 代理IP: 替换为 1.1.1.1/8.8.8.8 等已知DNS公网IP
- 移除硬编码SERVER_GEO,依赖MMDB自动解析

结果: 710/718节点有坐标, 496/500边两端可渲染 → 300条边显示

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-24 20:38:59 +08:00
TianXuan Developer 0a3c0a6eb3 fix: 修复gen_multi2过滤后无边的问题 
根因: gen_test_data 第135行 cert_cn 生成有bug
  f'*.{random.choice(SERVICES).split(".")[-2:]}'
  → 产生 "*.['example', 'com']" 垃圾cnam
  → 过滤匹配不到任何有效IP
  → 所有节点度=1,无边

修复: 检测并清除垃圾cnam/snam模式,仅保留有效值

测试结果(筛选cnam=api.example.com):
  节点718, 边500, 最大度201, 平均度18.5
  服务器入度: 152-201

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-24 20:30:15 +08:00
TianXuan Developer d6687e8aa0 fix: gen_multi2 完全继承 gen_test_data 格式 + 拓扑约束 
- 复用 generate_row() 保证所有列格式/覆盖率/空值率一致
- 重映射: snam/cnam 有值时 → :ipd 指向1-3个固定服务器IP
- 同步更新 GeoIP 字段匹配服务器位置
- 200文件 × 10000行 = 200万行
- 代理出度60K+, 服务器入度50K-202K

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-24 20:17:23 +08:00
TianXuan Developer 005a6b82bb fix: 重写gen_multi2, 继承gen_test_data精确格式 
- 复用 generate_row() 保证列格式/覆盖率/空值率完全一致
- 覆盖IP和服务字段为拓扑感知值
- cnam→1-3固定服务器IP, snam同理
- 代理出度870-1005, 服务器入度1615-2100
- 200文件×~100行, 列顺序/删减同gen_multi

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-24 20:08:49 +08:00
TianXuan Developer a4cd8cdf1d feat: 真实网络拓扑测试数据生成器 
gen_multi2.py:
- 服务器节点: 13个, 入度350-1700 (CDN/API/Mail/Auth等)
- 代理节点: 5个, 出度700+
- 客户端节点: 144个, 出度100-130
- cnam/snam 映射到特定服务器IP (1-3个)
- 12个地理区域, 匹配GeoIP坐标
- 输出: data/gen_multi2/ (200文件, 18148行)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-24 19:56:33 +08:00
TianXuan Developer e0a7400bd7 feat: 高精度离线GeoIP — MMDB支持 + 中文路径修复 + 分块自解压 
- geoip.py: 修复中文路径 str()→Path 编码问题 (maxminddb Latin-1 bug)
- geoip.py: 启动自动重组geoip_chunks分块 + gunzip解压
- scripts/setup_geoip.py: 手动重组/解压脚本
- .gitignore: 忽略解压后的 .mmdb 文件

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-23 23:42:38 +08:00
TianXuan Developer 66e3649152 feat: static SVG world map + large clickable points + edge detail 
- Generate high-quality SVG world map (static, no tiles)
- Points: radius 7, white border, high-contrast subnet colors
- Node click: show detail panel + highlight edges on map
- Highlighted edges: clickable, show edge detail panel
- Edge detail: Beijing time, avg bytes, src/dst IP clickable
- Map init: requestAnimationFrame ensures container has dimensions
- Cleanup: remove polygon-based continent drawing code

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-23 21:58:02 +08:00
PM-pinou 7d6d29475d fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True) 2026-07-22 23:39:37 +08:00
PM-pinou 7aa847ba3e v1.1.6: cleanup — schema sort fix, batch upload, temp dir to D:, update.bat wildcard fix 2026-07-22 19:10:39 +08:00
PM-pinou 8c90bad9f0 v1.1.6: batch upload BATCH=5 + temp dir to D: + schema sort + update.bat fix 2026-07-22 13:58:19 +08:00
PM-pinou 9ca5da1237 release: v1.1.6 — update.bat wildcard fix + schema column-order fix + gen_test format alignment + pipeline verify 2026-07-22 11:39:35 +08:00
PM-pinou 6a633016d2 v1.1.4: gen_test reads TlsDB coverage + schema fixes 
- gen_test_data.py: load_coverage() from TlsDB.csv, replaced all hardcoded _blank_or rates with coverage lookups
- gen_multi.py: coverage-based per-file column dropping
- collect_schema: force Utf8 for cross-file compatibility
- _background_process: pre-scan union columns, single CREATE TABLE, no ALTER TABLE
 2026-07-21 19:34:14 +08:00
PM-pinou 745b1481cd v1.1.3: entity_count fix + geoip database + multi-file test data + csv_glob display fix 
- Fix entity_count not populated after clustering pipeline
- Create comprehensive geoip_data.txt (803 IP ranges, 49 cities)
- Generate 100 multi_upload test files with missing columns
- Replace csv_glob path with SQLite table name in frontend
- Fix globe zoom-drag point-fixing (camera.z/12.0 scaling)
- Fix orphaned manual_run_analysis function
 2026-07-21 13:21:20 +08:00
PM-pinou 3617b7aabc release: v1.1.3 — 13 fixes across build, frontend, pipeline, and data handling 
Build: fix incremental package (update_info.json SHA256, __delete__, test exclude, files/ prefix), fix bump_version for 3-segment, PYTHONUTF8 in update.bat
Frontend: upload→manual redirect, LLM→cluster_overview redirect, filter UI before clustering, 地球→态势 rename, fix globe drag direction+inertia
Pipeline: LLM→auto clustering+PCA, column union handling (keep extra cols, no dropping), FFT spectral analysis tool
Quality: fix 24 bare except:pass patterns, enrich LLM context with TlsDB metadata, auto-save workflows with pinning, TlsDB-based column types
 2026-07-21 12:01:52 +08:00
PM-pinou 9314ac6942 chore: bump version to v1.1.2 
- New timeline UI: interleaved LLM reasoning + tool call display
- Fix: run_log truncation for large tool outputs
- Fix: lat/lon + values reduced to ~1% (was 99%)
- Fix: @csrf_exempt on upload/delete endpoints
- Fix: DATA_UPLOAD_MAX_NUMBER_FILES for 1998-file upload
- Fix: dtype alignment for multi-batch CSV processing
- New: tests/test_e2e_full.py with real DeepSeek LLM
- New: 1998-file test data generator
 2026-07-20 23:32:32 +08:00
PM-pinou efc6e3ab9d v8: fix server startup, CSV parse, upload flow, load_from_db types, integration test 2026-07-20 19:40:49 +08:00
PM-pinou 00774ea52b feat(scripts): add incremental update system with version tracking and diff utilities 
Add complete incremental update infrastructure:
- VERSION: project version tracking file
- scripts/version_utils.py: shared version parsing and comparison utilities
- scripts/build_update.py: build incremental update package from git diff
- scripts/apply_update.py: apply offline incremental update
- scripts/merge_config.py: safe config merge logic for update safety
- scripts/migrate_db_to_appdata.py: migrate DB from project dir to APPDATA

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
 2026-07-20 14:14:12 +08:00
PM-pinou 8fa36b774e v7: 19 issues fixed — SQLite storage, new clustering pipeline, display_id, globe rewrite, Chinese tools 
Key changes:
- New: data_loader.py SQLite persistence with drop_sqlite_table
- New: db_utils.py retry_on_lock decorator (3 retries, exponential backoff)
- New: tool_registry.py with 27 MCP tools (filter_and_cluster, compute_scores, etc.)
- New: tls_ref.py for TLS cipher/reference data
- New: import_tlsdb.py management command
- New: scripts/start_server.py for portable runtime
- New: migrations 0003-0007 for SQLite table, display_id, llm fields
- Changed: views.py unified pipeline worker, retry_run, display_id everywhere
- Changed: models.py with display_id auto-assignment, run_type, sqlite_table, llm_thinking, tool_calls_json
- Changed: urls.py added retry_run route
- Changed: session_store.py robust JSON persistence
- Changed: AGENTS.md v7 fix summary added
- Changed: templates — globe rewrite (inertia/polar flip), auto.html (thinking/tool accordions), base.html (toast/config), all pages use display_id
- Changed: run.bat PYTHONUTF8=1
- Deleted: entity_detector.py, entity_aggregator.py (replaced by filter_and_cluster clustering pipeline)
- Test: 92/92 unit tests passing
 2026-07-20 13:33:13 +08:00
PM-pinou 75af722f24 fix gen_test_data column count, cleanup temp data, add .gitignore for runtime/ 2026-07-16 22:56:23 +08:00
PM-pinou b58d01c7f2 cleanup: remove v6 garbage, restore original files, unify views.py workers 
- Delete 15 garbage scripts (wnl_to_user_csv, eval_clustering, debug_db2/3,
  debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_*,
  test_e2e_full, test_32b, test_chinese_path, verify_runtime)
- Delete data/wnl_converted.csv
- Delete analysis/value_normalizer.py + tests/test_value_normalizer.py
- Remove normalize_lf references from data_loader.py
- Remove _norm column lookups from entity_aggregator.py
- Fix settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB -> 100MB
- Fix upload.html: add per-file X delete button + removeFile()
- Refactor views.py: unify _run_analysis_worker/_run_llm_analysis into
  _run_pipeline_worker with callback pattern
- Wrap sync ORM call in tool_registry.py with sync_to_async
- Update run_pipeline.py to use _run_pipeline_worker
- Rebuild gen_test_data.py as simple version (replace 3-profile v6 version)
- Add runtime/ db.sqlite3 logs/ .omo/ to .gitignore
- Delete old .omo/plans/ plan files (tianxiu-v4/v5/v6, globe-v6)
- 120/120 tests passing
 2026-07-16 22:53:07 +08:00
PM-pinou 814f29645a chore: initial commit of 天璇 project state before cleanup 2026-07-16 22:41:26 +08:00