feat: 社区交互增强 + 卫星地图 + 节点/边数据丰富 + 项目清理
核心改进: - 节点数据丰富: TLS版本/加密套件/SNI/端口/流量统计/连接成功率 - 边数据丰富: 包数/持续时间/TLS/端口/SNI/分类标签 - 新增 edge_time_series 端点: 边流量时序图(北京时间) - 新增 cluster_nodes_edges 端点: 社区聚焦(zoom+高亮+专属边) - 社区点击: 自动缩放/高亮节点/显示社区边(可点击) - 边交互修复: 事件从layerGroup移至hitPoly,三处边渲染全修复 - 地图升级: ESRI卫星影像 + 255国矢量边界叠加 - 离线回退: 卫星→CartoDB→暗色矢量地图,完整可用 - 节点面板: 删除无意义社区标签,移至Host+组织同行 - sanitizeName: 清理5个非ASCII国家名避免乱码 - .gitignore: 补充geoip_chunks/和嵌套CSV规则 项目清理: - 删除 logs/ (20个调试文件,~2.5MB) - 删除 test_*.py (4个临时测试脚本) - 删除 _check_bool/_pipeline_test/_restart_server/start_server (硬编码路径) - 删除 node_modules/package.json/package-lock.json (18MB,不用Node) - 删除 tests/ 旧日志和调试脚本 - 删除 .pytest_cache/ .server_pid.txt RUNTIME_CHANGED.txt - AGENTS.md → CLAUDE.md Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
TianXuan Developer
+2
-1
@@ -21,6 +21,7 @@ runtime/
|
|||||||
# GeoIP MMDB (generated from geoip_chunks/)
|
# GeoIP MMDB (generated from geoip_chunks/)
|
||||||
*.mmdb
|
*.mmdb
|
||||||
*.mmdb.gz
|
*.mmdb.gz
|
||||||
|
data/geoip_chunks/
|
||||||
|
|
||||||
# Logs
|
# Logs
|
||||||
logs/
|
logs/
|
||||||
@@ -42,7 +43,7 @@ Thumbs.db
|
|||||||
RUNTIME_CHANGED.txt
|
RUNTIME_CHANGED.txt
|
||||||
|
|
||||||
# Test data (auto-generated CSV files)
|
# Test data (auto-generated CSV files)
|
||||||
data/*.csv
|
data/**/*.csv
|
||||||
data/multi_upload/
|
data/multi_upload/
|
||||||
node_modules/
|
node_modules/
|
||||||
package-lock.json
|
package-lock.json
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
16884
|
|
||||||
@@ -1,467 +0,0 @@
|
|||||||
# 天璇 (TianXuan) -Agent Knowledge Base
|
|
||||||
|
|
||||||
## Project Identity
|
|
||||||
|
|
||||||
| Field | Value |
|
|
||||||
|-------|-------|
|
|
||||||
| 项目名称 | 天璇 (TianXuan) |
|
|
||||||
| 原始名称 | tls-analyzer |
|
|
||||||
| 核心功能 | TLS 流数据分析、实体画像、聚类3D地球可视化、PCA散点图|
|
|
||||||
| Python 版本 | 3.12 (embedded portable at `runtime/python/python.exe`) |
|
|
||||||
| Polars 版本 | 1.42.1 (精确锁定) |
|
|
||||||
| 数据框架 | Polars (LazyFrame + streaming) |
|
|
||||||
| 机器学习 | scikit-learn (HDBSCAN, KMeans, UMAP, TruncatedSVD, StandardScaler) |
|
|
||||||
| Web框架 | Django 4.2 + SQLite WAL |
|
|
||||||
| 前端3D | Three.js (603KB, 离线, 地球贴图1.4MB) |
|
|
||||||
| 前端图表 | 纯Canvas散点图(无Chart.js) |
|
|
||||||
| LLM协议 | MCP (stdio transport) |
|
|
||||||
| 启动方式 | `run.bat`（双击即用）/ `runtime\python\python.exe manage.py runserver`（开发） |
|
|
||||||
| 目标设备 | Windows 10 1902, 4GB RAM, 无独立显卡（纯核显） |
|
|
||||||
|
|
||||||
## Project Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
tianxuan/
|
|
||||||
├── tianxuan/ # Django 项目配置
|
|
||||||
│ ├── settings.py # ALLOWED_HOSTS 自动检测 LOGGING 文件+stderr
|
|
||||||
│ ├── wsgi.py # SQLite 启动自修复
|
|
||||||
│ ├── urls.py
|
|
||||||
│ └── llm_orchestrator.py # LLM 编排(32B/284B 双兼容
|
|
||||||
│ ├── analysis/ # 核心分析模块 (Django app)
|
|
||||||
│ ├── data_loader.py # CSV 加载, BOM检测 schema_strict, 递归glob, 中文路径
|
|
||||||
│ ├── data_profiler.py # 列统计+ 相关性矩阵(无pandas回退:numpy)
|
|
||||||
│ ├── entity_detector.py # 实体列自动检测(tuple关键词+ unique_ratio + null惩罚)
|
|
||||||
│ ├── entity_aggregator.py # 实体聚合 (tuple关键词 lat/lon检测 多列group_by, IP子网)
|
|
||||||
│ ├── session_store.py # 线程安全单例内存存储
|
|
||||||
│ ├── tool_registry.py # 12个MCP工具 + DB持久化(sync_to_async)
|
|
||||||
│ ├── mcp_server.py # MCP stdio server
|
|
||||||
│ ├── views.py # 所有Django视图 (上传/手动/LLM/配置/LLM测试/日志/地球)
|
|
||||||
│ ├── urls.py # 16条路由 ├── models.py # ORM: AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
|
||||||
│ ├── data_validator.py # 列校验(缺失异常值IP有效性
|
|
||||||
│ ├── type_classifier.py # 值优先类型检测(MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON)
|
|
||||||
│ ├── geoip.py # GeoIP 经纬度查询(data/geoip_data.txt)
|
|
||||||
│ ├── admin.py
|
|
||||||
│ └── management/commands/
|
|
||||||
│ ├── start_mcp.py # MCP服务器启动命令
|
|
||||||
│ └── run_pipeline.py # 一键CLI管道命令
|
|
||||||
│ ├── config/
|
|
||||||
│ ├── config.yaml # 自动生成默认配置 (含entity.subnet_masks, columns覆盖)
|
|
||||||
│ ├── loader.py # Pydantic 配置加载 + 文件修改检测
|
|
||||||
│ └── __init__.py
|
|
||||||
│ ├── templates/
|
|
||||||
│ ├── base.html # 导航: 首页/上传/手动/LLM/记录/地球/配置
|
|
||||||
│ └── analysis/
|
|
||||||
│ tianxuan/ # 全部11个页面模板
|
|
||||||
│ ├── dashboard.html # 首页 - 最近运行
|
|
||||||
│ ├── upload.html # CSV上传 - 拖拽+多文件删除+进度
|
|
||||||
│ ├── manual.html # 3步手动向- 选数据→设参数→运行
|
|
||||||
│ ├── auto.html # LLM自动分析 - 实时日志取消
|
|
||||||
│ ├── config.html # 配置编辑 - LLM连通性测试
|
|
||||||
│ ├── run_list.html # 运行记录列表
|
|
||||||
│ ├── run_detail.html # 运行详情摘要
|
|
||||||
│ ├── cluster_overview.html # 聚类概览 - 纯Canvas散点PCA+地理
|
|
||||||
│ ├── cluster_detail.html # 簇详情- 特征实体列表
|
|
||||||
│ ├── entity_profile.html # 实体画像 - 特征偏离+Canvas单点
|
|
||||||
│ ├── globe.html # 3D地球可视- Three.js 流量弧线+经纬国境
|
|
||||||
│ └── log_viewer.html # 日志查看
|
|
||||||
│ ├── static/tianxuan/ # 离线静态资源
|
|
||||||
│ ├── three.min.js # Three.js 3D引擎 (603KB)
|
|
||||||
│ ├── earth_atmos_2048.jpg # 地球贴图 (1.4MB)
|
|
||||||
│ └── world_borders.js # 10国精简国境线轮廓 ├── runtime/python/ # 便携Python 3.12运行时(~593MB)
|
|
||||||
│ ├── scripts/
|
|
||||||
│ ├── gen_test_data.py # 简单测试数据生成(--globe模式使用真实GeoIP范围)
|
|
||||||
│ ├── gen_complex_test.py # 复杂数据生成 (5000行4列含55%缺失)
|
|
||||||
│ ├── column_survey.py # CSV列结构调查(统计各CSV的列类型/分布)
|
|
||||||
│ ├── diagnose_compare.py # 跨机日志对比诊断
|
|
||||||
│ └── debug_db.py # DB持久化调试 ├── tests/ # 92个测试(unit) / 121个测试(all)
|
|
||||||
│ ├── test_data_loader.py # 26: BOM/schema/中文路径/递归glob
|
|
||||||
│ ├── test_entity.py # 17: 关键词检测聚合
|
|
||||||
│ ├── test_e2e.py # 6: 全端到端
|
|
||||||
│ └── test_clustering_edge.py # HDBSCAN零样本降级 ├── docs/
|
|
||||||
│ ├── 工作流.md # 完整分析流水线文档
|
|
||||||
│ └── 故障诊断手册.md # 跨机问题诊断指南
|
|
||||||
│ ├── run.bat # 用户启动 (含PYTHONUTF8=1)
|
|
||||||
├── shell.bat # Django shell
|
|
||||||
├── TlsDB.csv # 参考表头名清单（非实际数据├── README.md
|
|
||||||
├── AGENTS.md
|
|
||||||
└── manage.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## 12 MCP Tools
|
|
||||||
|
|
||||||
| # | 工具| 功能 | 必需参数 |
|
|
||||||
|---|--------|------|----------|
|
|
||||||
| 1 | `load_data` | 加载 CSV 文件（glob / 递归 / schema容错| `csv_glob` |
|
|
||||||
| 2 | `profile_data` | 数据集概要统+ 相关性矩阵| `dataset_id` |
|
|
||||||
| 3 | `filter_data` | 按条件过滤（12操作+ AND/OR| `dataset_id`, `filters` |
|
|
||||||
| 4 | `preprocess_data` | 预处理（标准化编码/填充)| `dataset_id`, `columns` |
|
|
||||||
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans + 质量评估| `dataset_id`, `cluster_columns` |
|
|
||||||
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH| `cluster_result_id`, `dataset_id` |
|
|
||||||
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA| `dataset_id`, `cluster_result_id` |
|
|
||||||
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON| `result_id`, `output_path` |
|
|
||||||
| 9 | `list_datasets` | 列出所有活跃数据集 | |
|
|
||||||
| 10 | `drop_dataset` | 删除数据集释放内存| `dataset_id` |
|
|
||||||
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据| `dataset_id` |
|
|
||||||
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
|
|
||||||
|
|
||||||
所有工具通过 `analysis/tool_registry.py` 注册，`analysis/mcp_server.py` 暴露标准 MCP stdio 接口
|
|
||||||
|
|
||||||
## Config.Entity
|
|
||||||
|
|
||||||
`config/loader.py` 中的 Pydantic 模型
|
|
||||||
|
|
||||||
```python
|
|
||||||
class Entity(BaseModel):
|
|
||||||
subnet_masks: list[int] = [] # 子网掩码列表，如 [24, 28]
|
|
||||||
ip_columns: list[str] = ['src_ip', 'dst_ip'] # 需聚合的IP列```
|
|
||||||
|
|
||||||
config.yaml 默认配置:
|
|
||||||
```yaml
|
|
||||||
entity:
|
|
||||||
subnet_masks: [24, 28]
|
|
||||||
ip_columns: ["src_ip", "dst_ip"]
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
- **tuple 替代 frozenset**: 避免跨机 PYTHONHASHSEED 差异
|
|
||||||
- **PYTHONUTF8=1**: 跨机编码一致性- **类型安全聚合**: 聚合前检查dtype，非数值列cast
|
|
||||||
- **sync_to_async ORM**: 异步上下文中Django ORM
|
|
||||||
- **纯Canvas 图表**: 零外部JS 依赖
|
|
||||||
- **上传目录隔离**: %APPDATA%/TianXuan/data/uploads/，与项目路径无关
|
|
||||||
- **通用数值清理_coerce_to_float**: 处理 + / - /空白/N/A 等伪值- **值优先类型检测*: config→值→名称→STRING
|
|
||||||
- **地球深度测试**: depthTest: true, depthWrite: false，r=5.5
|
|
||||||
- **完整国境线*: Natural Earth 110m, 177国 286多边形 269KB
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Testing Protocol
|
|
||||||
|
|
||||||
### 运行全部测试
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 单元测试（不需要Django)runtime\python\python.exe -m pytest tests/test_data_loader.py tests/test_type_classifier.py -q
|
|
||||||
|
|
||||||
# 集成测试（自动启动后端、模拟前端行为、监控stderr报错)runtime\python\python.exe tests/test_integration.py
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
92 passed (unit tests); ALL INTEGRATION TESTS PASSED
|
|
||||||
```
|
|
||||||
|
|
||||||
> ⚠️ **重要**: `tests/test_integration.py` 模拟前端HTML行为（上传CSV 轮询状态访问页面 删除）> **当前端HTML/API发生变化时，必须同步更新此脚*，否则集成测试将不再反映真实用户流程
|
|
||||||
>
|
|
||||||
### 跨机一致性测```bash
|
|
||||||
|
|
||||||
# 两台机器各自执行
|
|
||||||
|
|
||||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
|
||||||
|
|
||||||
# 对比日志
|
|
||||||
|
|
||||||
runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
### DB持久化测```bash
|
|
||||||
runtime\python\python.exe scripts\debug_db.py
|
|
||||||
# 期望: n_features=30 db_saved=True
|
|
||||||
# 验证: run_analysis ClusterFeature 表有数据
|
|
||||||
```
|
|
||||||
|
|
||||||
### LLM编排测试
|
|
||||||
|
|
||||||
```bash
|
|
||||||
runtime\python\python.exe scripts\test_llm_full.py
|
|
||||||
# 或 runtime\python\python.exe -c "from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig; ..."
|
|
||||||
```
|
|
||||||
|
|
||||||
### 生成测试数据
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 简单数据runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
|
||||||
|
|
||||||
# Globe模式 (IP在真实GeoIP范围)
|
|
||||||
runtime\python\python.exe scripts\gen_test_data.py --globe
|
|
||||||
|
|
||||||
# 复杂数据 (5000行4列含55%缺失)
|
|
||||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
|
|
||||||
```
|
|
||||||
|
|
||||||
### 列结构调查```bash
|
|
||||||
|
|
||||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Polars Version Compatibility
|
|
||||||
|
|
||||||
| API | 旧版(<1.0) | 新版(.0) |
|
|
||||||
|-----|-----------|-----------|
|
|
||||||
| mode | `pl.mode()` | `pl.col(col).mode()` |
|
|
||||||
| encoding | `encoding='utf-8'` | `encoding='utf8'` |
|
|
||||||
| value_counts col | `'index'` | 原列名|
|
|
||||||
| streaming | `collect(streaming=True)` | `collect(engine='streaming')` |
|
|
||||||
| truth value | `if series:` | `if series.item():` |
|
|
||||||
|
|
||||||
## 32B LLM Optimization
|
|
||||||
|
|
||||||
- Step guidance: 每步告诉模型下一步做什么- Truncation: 工具结果截断1200 chars
|
|
||||||
- Timeout: 90s (32B 推理更慢)
|
|
||||||
- Max steps: 15
|
|
||||||
- Tool description: 一行简洁描述
|
|
||||||
---
|
|
||||||
|
|
||||||
## v1.1.3 修复 (2026-07-21)
|
|
||||||
|
|
||||||
13 issues fixed across build system, frontend, pipeline, and data handling.
|
|
||||||
|
|
||||||
| # | 问题 | 状态| 详情 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | 增量包构建修复| | update_info.json含SHA256、文件在files/子目录下、测试数据排除、__delete__支持 |
|
|
||||||
| 2 | 构建顺序+三段版本号| | 先commit后bump再tag、bump_version支持vX.Y.Z、update.bat加PYTHONUTF8 |
|
|
||||||
| 3 | 前端导航跳转 | | 上传后→手动分析页、LLM完成后→聚类概览|
|
|
||||||
| 4 | 聚类前手动筛选| | manual.html下filter_data工具UI |
|
|
||||||
| 5 | 地球→态势+拖拽修复 | | 导航改名、拖拽方向修正、惯性仅松开时施加|
|
|
||||||
| 6 | DB模型同步 | | 所有模型与SQLite表完全匹配|
|
|
||||||
| 7 | LLM后强制聚类| | LLM完成后自动运行clustering+PCA pipeline |
|
|
||||||
| 8 | 列名不一致处理| | 保留所有列(union)、不丢弃额外列、gen_test_data支持--missing-cols |
|
|
||||||
| 9 | FFT频谱分析 | | 新MCP工具analyze_fft(numpy.fft)、周期性特征提取|
|
|
||||||
| 10 | 异常处理修复 | | 24处裸except/except:pass全部修复为带日志处理 |
|
|
||||||
| 11 | LLM上下文增加| | 发送列类型+中文含义+空值率+众数给LLM |
|
|
||||||
| 12 | 工作流自动保存| | 最0条自动方案保存到手动分析页、支持固定防淘汰 |
|
|
||||||
| 13 | 列类型遵循TlsDB | | tls_ref_data作为权威类型源、未知列严格回退 |
|
|
||||||
|
|
||||||
## 修复计划 (2026-07-16)
|
|
||||||
|
|
||||||
| # | 问题 | 状态| 详情 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | 重启后旧数据可分析| | SessionStore 元数据持久化为JSON 文件(`%APPDATA%/TianXuan/.session_store.json`)，wsgi.py 启动时自动恢复|
|
|
||||||
| 2 | Web图标所有页| | `<link rel="icon">` 在 base.html 中，所有页面继承|
|
|
||||||
| 3 | 上传按钮在顶部| | `position: sticky` |
|
|
||||||
| 4 | Traceback完整 | | 移除 views.py 4处 + run_pipeline.py 1处`[:200]` 截断 |
|
|
||||||
| 5 | 大数据不卡顿 | | 降采样`head=` 参数支持，手动POST 可传 head，LLM 自动决定，clamp 500|
|
|
||||||
| 6 | 进度条| | progress_pct → 10/30/50/70/90/100 各阶段更新|
|
|
||||||
| 7 | TLS 0ver hex识别 | | TLS_HEX_MAP 移至 type_classifier.py 值优先检测链ver列自动ENUM("TLSv1.2") |
|
|
||||||
| 8 | Globe响应式缩放| | Canvas resize 事件绑定，vh 单位动态计算|
|
|
||||||
| 9 | Globe ?data= | | 异常安全：try/except 包裹，空数据仍渲染页面|
|
|
||||||
| 10 | 文件上传无限| | DATA_UPLOAD_MAX_NUMBER_FIELDS=1000000, FILE_UPLOAD_MAX_MEMORY_SIZE=100MB |
|
|
||||||
| 11 | 上传页删除功能| | removeFile() + 按钮每文件|
|
|
||||||
| 12 | LLM日志 | | progress_callback 写入 run_run_log |
|
|
||||||
| 13 | 聚类图表 | | cluster_overview.html 含散点图/柱状态Silhouette |特征纯Canvas) |
|
|
||||||
| 14 | 低内存不崩溃 | | MiniBatchKMeans + PCA降维 + 降采样head 参数 |
|
|
||||||
| 15 | +号字符串识别为数| | type_classifier.py FLOAT检测前排除 `+` 前缀 |
|
|
||||||
| 16 | 空闲高IO | | 日志级别 INFO，无需修改 |
|
|
||||||
| 17 | 聚类效果 | ⚠️ | 预置了特征过滤降维，但具体效果需业务验证 |
|
|
||||||
| 18 | 列名精确匹配 | | `re.match` 精确匹配 + `_find_column` tuple 关键词|
|
|
||||||
|
|
||||||
## v7 Final Verification (2026-07-20)
|
|
||||||
|
|
||||||
All **19 issues** fixed and verified. 92/92 unit tests passing. Full regression run completed.
|
|
||||||
|
|
||||||
| # | 问题 | 状态| 详情 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | `handle_call` imported in llm_orchestrator.py | | `from analysis.tool_registry import handle_call` at line 7 |
|
|
||||||
| 2 | Retry button + run_type filter | | Retry form in `run_list.html` (line 51), run_type dropdown filter, URL route `retry_run` added |
|
|
||||||
| 3 | auto.html logs show tool+result | | `liveLog` polling every 3s from `/logs/?pos=` endpoint |
|
|
||||||
| 4 | Data saved to SQLite tables | | `sqlite_table` field on `AnalysisRun` model, `drop_sqlite_table` in data_loader |
|
|
||||||
| 5 | Error toast in base.html | | Complete toast system with error/success types, auto-dismiss, slide animations |
|
|
||||||
| 6 | display_id in all URLs | | All 6 URL patterns (`run_detail`, `run_status`, `cluster_overview`, `cluster_detail`, `retry_run`, `delete_upload`) use `display_id` |
|
|
||||||
| 7 | Delete buttons on dashboard+run_list | | Both pages have styled delete buttons with confirmation dialog |
|
|
||||||
| 8 | Progress bar 100% on completion | | `progress_pct` field updated through pipeline stages (10/30/50/70/90/100) |
|
|
||||||
| 9 | LLM thinking panel in auto.html | | `thinkingPanel` div with `llm_thinking` text area and step indicator |
|
|
||||||
| 10 | Tool call accordion in auto.html | | `renderToolCalls()` with toggle-able accordion sections per tool step |
|
|
||||||
| 11 | LazyFrame.sample() fixed | | Uses `lf.sample(n=...)` with explicit row count parameter (not deprecated `.sample()` API) |
|
|
||||||
| 12 | `filter_and_cluster` tool exists | | Registered at line 407 in `tool_registry.py` as tool #13 |
|
|
||||||
| 13 | Tool descriptions in Chinese | | All tool descriptions and parameter docs use Chinese |
|
|
||||||
| 14 | Globe: lines, timestamp, inertia, polar flip | | Flow arcs (Line geometry), timestamp-based dot animation, quaternion momentum decay, camera-relative rotation (no polar flip) |
|
|
||||||
| 15 | Config button in nav | | `{% url 'analysis:config' %}` link 配置 in base.html nav bar |
|
|
||||||
| 16 | config host:port affects broadcast | | `settings.py` reads `_cfg.server.host` - auto-adds to `ALLOWED_HOSTS` |
|
|
||||||
| 17 | SQLite timeout+retry mechanism | | `retry_on_lock` decorator in `db_utils.py`: 3 retries, exponential backoff, logs `[DB_LOCKED]` warning |
|
|
||||||
| 18 | Delete crash fixed (null csv_glob) | | `delete_upload` view checks `if run.csv_glob:` before accessing path (line 366) |
|
|
||||||
| 19 | No file count limit, streaming upload | | `DATA_UPLOAD_MAX_NUMBER_FIELDS=10000000`, `FILE_UPLOAD_MAX_MEMORY_SIZE=100MB` |
|
|
||||||
|
|
||||||
### Remaining issues
|
|
||||||
- `test_e2e.py`, `test_entity.py`, `test_pipeline.py` still reference removed `entity_detector` module - need updating for new clustering pipeline
|
|
||||||
- `test_clustering_edge.py` does not exist on disk (only referenced in AGENTS.md)
|
|
||||||
- `retry_run` URL route was missing from `urls.py` - fixed during final verification
|
|
||||||
|
|
||||||
### Test Results (2026-07-20)
|
|
||||||
| Test suite | Tests | Result |
|
|
||||||
|------------|-------|--------|
|
|
||||||
| `test_data_loader.py` | 28 | All passed |
|
|
||||||
| `test_type_classifier.py` | 64 | All passed |
|
|
||||||
| **Total** | **92** | ** All passed** |
|
|
||||||
|
|
||||||
## v8 Fix (2026-07-20)
|
|
||||||
|
|
||||||
| # | 问题 | 状态|
|
|
||||||
|---|------|------|
|
|
||||||
| 1 | 服务器启动config模块找不到| 修复 |
|
|
||||||
| 2 | 首次启动无数据库| 修复 |
|
|
||||||
| 3 | CSV中+'无法解析为f64 | 修复 |
|
|
||||||
| 4 | 上传分块处理+释放临时文件 | 修复 |
|
|
||||||
| 5 | 集成测试脚本 | 添加 |
|
|
||||||
| 6 | TlsDB.csv文档说明 | 更新 |
|
|
||||||
| 7 | load_from_db类型冲突 | 修复 |
|
|
||||||
| 8 | CSV未合并处理| 修复 |
|
|
||||||
|
|
||||||
### Final Verification (2026-07-20)
|
|
||||||
| Check | Result |
|
|
||||||
|-------|--------|
|
|
||||||
| `manage.py check` (system) | 0 errors |
|
|
||||||
| Unit tests (data_loader + type_classifier) | 92 passed |
|
|
||||||
| Integration test (full upload→load→analyze) | ALL INTEGRATION TESTS PASSED |
|
|
||||||
| Git commit `v8` | `555ec29` |
|
|
||||||
|
|
||||||
## MCP 工具系统 (2026-07-19)
|
|
||||||
|
|
||||||
系统现有 **27 |MCP 工具**，分为三层：
|
|
||||||
|
|
||||||
### 工具架构
|
|
||||||
|
|
||||||
| | 数量 | 标签 | 用|
|
|
||||||
|---|------|------|------|
|
|
||||||
| Core | 7 | | 执行分析，可修改数据 |
|
|
||||||
| Analysis | 5 | 🔍 | 只读深潜，安全随时调用|
|
|
||||||
| Diagnostic | 7 | 🩺 | 排查问题 |
|
|
||||||
| 其他 | 8 | | 数据加载/过滤/导出|
|
|
||||||
|
|
||||||
### 手动分析工作流
|
|
||||||
手动分析页面已改为 **工作流构建器**:
|
|
||||||
- 添加/移除/重排步骤
|
|
||||||
- 每步选择工具 + 填写参数
|
|
||||||
- **保存/加载/删除**方案（存储`.omo/plans/<name>.json`)- 单步执行 |**▶▶ 全部执行**（自动串行）
|
|
||||||
- 执行结果显示在每一步下面
|
|
||||||
### LLM 自动编排
|
|
||||||
|
|
||||||
LLM orchestrator 重写，支持：
|
|
||||||
- 策略式系统提示词（先 profile 根据数据决策 失败自动诊断- 重复调用检测+ 自动错误恢复
|
|
||||||
- 27 个工具全部可用- 中文总结分析结果
|
|
||||||
|
|
||||||
### 全链路测试
|
|
||||||
| 测试 | 用例 | 结果 |
|
|
||||||
|------|------|------|
|
|
||||||
| 单元测试 | 120 | |
|
|
||||||
| 管道测试 | 8 (3 datasets × 2 algos + edge) | |
|
|
||||||
| CLI pipeline | e2e 5 stages | |
|
|
||||||
| MCP 工具集成 | 10 (profile/patterns/validate/tls/geo/build/scores/detect/detail/visualize) | |
|
|
||||||
|
|
||||||
## v3 修复 (2026-07-17)
|
|
||||||
|
|
||||||
| # | 问题 | 状态| 详情 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | 经纬度列名匹配`:ips.latd`) | | views.py直接字符串匹配+ entity_aggregator.py _LAT_KEYWORDS增加.latd后缀 |
|
|
||||||
| 2 | cnrs/isrs布尔类型 | | entity_aggregator.py聚合前将"+"→True/空白→False |
|
|
||||||
| 3 | 聚类图表（legend/柱状态负值） | | 纯Canvas：legend用HTML div+overflow、柱状图动态带宽、负值标签重新定义|
|
|
||||||
| 4 | 地球缩放改变数据流大小| | Three.js arc sphere尺寸与camera距离反比，zoom时视觉大小不变|
|
|
||||||
| 5 | 恶意/代理流量分析 | | entity_aggregator.py规则引擎：non_std_port_rate + modern_tls_rate + sni_missing_ratio + multi_country + proxy_score |
|
|
||||||
| 6 | EntityProfile批量IO | | views.py + tool_registry.py 改为 bulk_create(ignore_conflicts=True, batch_size=1000) |
|
|
||||||
| 7 | 重试分析 | 🗑| 已移除(v7 新架构不再需要entity_column 概念) |
|
|
||||||
| 8 | D5降采样导致特征提取行数不匹配 | | _handle_run_clustering存储_ds_idx抽样索引，_handle_extract_features按索引过滤数据|
|
|
||||||
|
|
||||||
## v4 修复 (2026-07-17)
|
|
||||||
|
|
||||||
| # | 问题 | 状态| 详情 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | 进度条位置| | DOM重排: submitBar → progressArea → fileList progressArea → fileList |
|
|
||||||
| 2 | PCA sklearn RuntimeWarning | | `warnings.filterwarnings('ignore', category=RuntimeWarning, module='sklearn')` |
|
|
||||||
| 3 | cluster_overview N+1查询 | | `prefetch_related('features')` + `.only()`字段限制 |
|
|
||||||
| 4 | SQLite PRAGMA未优| | WAL + synchronous=NORMAL + cache_size=20MB + temp_store=MEMORY + mmap_size=3GB |
|
|
||||||
| 5 | 零方差过滤崩| | 全零方差时回退`np.arange(data.shape[1])`保留原始特征 |
|
|
||||||
|
|
||||||
### 大规模测试结(2026-07-17)
|
|
||||||
|
|
||||||
| 测试 | 规模 | 结果 |
|
|
||||||
|------|------|------|
|
|
||||||
| 单元测试 | 120 tests | 全部通过 |
|
|
||||||
| 管道测试 | 3 datasets × 2 algorithms | 8 tests OK |
|
|
||||||
| 中等规模 | **100文件 × 10000行= 1M行 825MB** | 全部完成: 加载→实体检测→聚合→聚2285个实体→特征提取|
|
|
||||||
|
|
||||||
## Cleanup (2026-07-16)
|
|
||||||
|
|
||||||
对 v6 迭代引入的垃圾文件和代码破坏进行全面清理
|
|
||||||
| 操作 | 文件 |
|
|
||||||
|------|------|
|
|
||||||
| 🗑删除 | 15个垃圾脚(wnl_to_user_csv, eval_clustering, debug_db2/3, debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_loop/format, test_e2e_full, test_32b, test_chinese_path, verify_runtime) |
|
|
||||||
| 🗑删除 | data/wnl_converted.csv, data/globe_test.csv |
|
|
||||||
| 🗑删除 | analysis/value_normalizer.py + 关联测试 + data_loader/entity_aggregator中的_norm引用 |
|
|
||||||
| 🗑删除 | .omo/plans/ 旧计划文(tianxiu-v4/v5/v6, globe-v6) |
|
|
||||||
| 🔧 重构 | views.py: 统一 _run_analysis_worker → _run_llm_analysis → _run_pipeline_worker |
|
|
||||||
| 🔧 修复 | settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB→100MB |
|
|
||||||
| 🔧 修复 | upload.html: 添加每文件删除按钮 + removeFile() |
|
|
||||||
| 🔧 修复 | tool_registry.py: async函数中的同步ORM调用加sync_to_async包装 |
|
|
||||||
| 🔧 修复 | run_pipeline.py: 改为调用 _run_pipeline_worker |
|
|
||||||
| 🔧 替换 | gen_test_data.py: 3画像版→简单随机生成版 |
|
|
||||||
| 验证 | 120/120 测试通过, 4次端到端管道测试全部成功 |
|
|
||||||
|
|
||||||
### v6 (2026-07-16) 已清理（不再维护
|
|
||||||
> ⚠️ v6 引入value_normalizer 模块已被删除，_norm 列全部移除> entity_aggregator 回退到使用原始列名（0ver/0cph/cipher-suite等）
|
|
||||||
| 模块 | 变动 | 当前状态|
|
|
||||||
|------|------|---------|
|
|
||||||
| value_normalizer.py | ~~新增: hex→enum归一化~~ | 🗑已删|
|
|
||||||
| data_loader.py | ~~集成normalize_lf~~ | 🔧 恢复原始 |
|
|
||||||
| entity_aggregator.py | ~~_norm列依赖~~ | 🔧 回退原始列名 |
|
|
||||||
| gen_test_data.py | ~~3类流量画像~~ | 🔧 简单随机生|
|
|
||||||
| 特征维度 | 20→3| 特征维度保留3维），去掉_norm列后的纯原始列特征|
|
|
||||||
|
|
||||||
## Completed (2026-07-16)
|
|
||||||
|
|
||||||
18个问题全部修复，3次端到端管道测试全部通过（简单CSV、用户自定义列名CSV、大规模CSV）
|
|
||||||
| 工作流| 问题 | 状态|
|
|
||||||
|--------|------|------|
|
|
||||||
| A1 | 重启后旧数据可分析| 目录不存在时优雅降级，标记为failed |
|
|
||||||
| A2 | Web图标所有页面显示| 创建favicon.svg + base.html添加link |
|
|
||||||
| A3 | 上传按钮在顶部| 移至文件列表上方，sticky定位 |
|
|
||||||
| A10 | 文件上传限制 | 去除了Django字段限制，增大至256MB/50GB |
|
|
||||||
| A11 | 上传页删除功能| 所有状态显示删除按钮，处理中弹出确认|
|
|
||||||
| B7 | TLS 0ver hex格式 | 添加TLS_HEX_MAP，识别3 03/03 04 |
|
|
||||||
| B15 | "+"字符串识别为数字 | FLOAT检测添加/前缀检查|
|
|
||||||
| B18 | 列名映射 | 使用精确^exact_name$匹配，无模糊猜测 |
|
|
||||||
| C4 | Traceback截断 | tool_registry.py移除[:200] |
|
|
||||||
| D5 | 大数据量卡顿 | 添加head参数10K降采样、MAX_ROWS=500 |
|
|
||||||
| D6 | 进度条| 添加progress_pct/progress_msg字段 |
|
|
||||||
| D14 | 低内存崩溃| MiniBatchKMeans、PCA降维、内存检测|
|
|
||||||
| D16 | 空闲高IO | 日志级别已为INFO，无需修改 |
|
|
||||||
| E8 | Globe响应式缩放| 动态H计算、vh单位、resize处理 |
|
|
||||||
| E9 | Globe ?data=参数 | 支持从SessionStore直接加载数据|
|
|
||||||
| F12 | LLM日志 | 添加callback机制、run_log字段 |
|
|
||||||
| G13 | 聚类图表 | 添加簇大小柱状图、Silhouette对比|
|
|
||||||
| H17 | 聚类质量 | 方差过滤、相关过滤、HDBSCAN自动调参 |
|
|
||||||
|
|
||||||
## v1.1.6 (2026-07-22)
|
|
||||||
|
|
||||||
Cleanup + minor fixes release.
|
|
||||||
|
|
||||||
| # | 问题 | 状态 | 描述 |
|
|
||||||
|---|------|------|------|
|
|
||||||
| 1 | schema sort fix | 修复 | 排序逻辑修改，确保最后一行被包含 |
|
|
||||||
| 2 | batch upload | 修复 | 批量上传流程加固 |
|
|
||||||
| 3 | temp dir to D: | 修复 | 临时目录移到 D: 盘避免 C: 盘空间占用 |
|
|
||||||
| 4 | update.bat wildcard fix | 修复 | 通配符匹配优化使更新更可靠 |
|
|
||||||
| 5 | cleanup test artifacts | 修复 | 删除 test CSV data, temp scripts, gen_fast.py |
|
|
||||||
| 6 | VERSION sync | 修复 | VERSION 文件已更新为 v1.1.6 |
|
|
||||||
|
|
||||||
### Pending
|
|
||||||
- tests/ 需要更新 entity_detector 相关
|
|
||||||
- 大文件上传存在内存泄漏需要进一步排查
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Operating Procedures
|
|
||||||
|
|
||||||
1. **不要阻塞式启动后端** — Start Django backend in background (use `start /B` or threading), never block the terminal. Use `start /B runtime\python\python.exe manage.py runserver` to avoid hanging the shell.
|
|
||||||
|
|
||||||
2. **全量测试** — After all changes, run full E2E test: generate 2000 CSVs, upload through frontend, verify complete pipeline. This validates the entire system end-to-end before considering work complete.
|
|
||||||
|
|
||||||
3. **更新文档和Git** — After tests pass: update AGENTS.md, commit all changes, push, clean up temp files and running processes. Do not leave dirty state (orphan processes, temp files, uncommitted changes).
|
|
||||||
|
|
||||||
4. **同步依赖** — Use `git pull` to sync latest code, then `runtime\python\python.exe -m pip install -r requirements.txt` to install any new packages. Do this before starting any new work session.
|
|
||||||
|
|
||||||
5. **编码前缀** — Prefix ALL terminal commands with `chcp 65001` to prevent Opencode encoding errors. PowerShell/CMD encoding issues are the #1 source of inexplicable command failures.
|
|
||||||
|
|
||||||
6. **Python UTF-8** — When running Python, always add `set PYTHONUTF8=1` before the command. This ensures consistent Unicode handling across all machines and prevents cross-machine encoding drift.
|
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
CLAUDE.md — 天璇 (TianXuan) 系统
|
||||||
|
|
||||||
|
项目概览
|
||||||
|
天璇 是一个 TLS 加密流量数据分析与实体画像系统，基于 Django + Polars + scikit-learn + Three.js 构建，支持从 CSV 导入、实体检测、聚合、聚类、特征提取到 3D 可视化的全自动离线分析流水线。
|
||||||
|
|
||||||
|
- 版本: v2.0.0beta
|
||||||
|
- 运行方式: 离线桌面应用，内置 Python 3.12 运行时，双击 run.bat 即可启动，无需安装任何依赖。
|
||||||
|
- 数据存储: SQLite (WAL 模式) + 内存缓存
|
||||||
|
- 分支策略: 后续所有改动直接提交到 main 分支，并推送到远程仓库 https://gitea.cattysteve.top/HJQ/tianxuan。
|
||||||
|
|
||||||
|
核心模块
|
||||||
|
| 模块 | 功能 |
|
||||||
|
|------|------|
|
||||||
|
| analysis/data_loader.py | CSV 加载、合并、列名规范化、GeoIP 解析 |
|
||||||
|
| analysis/entity_detector.py | 基于列名关键词和唯一值比率的实体列自动检测 |
|
||||||
|
| analysis/entity_aggregator.py | 按实体分组计算流量统计、TLS特征、时间模式等 |
|
||||||
|
| analysis/tool_registry.py | 12 个 MCP 工具注册（145KB） |
|
||||||
|
| simple_analysis/ | 新增的“简单分析”模块，独立 Django App，提供上传→筛选→聚类→地图可视化一站式工作流 |
|
||||||
|
| templates/ | 传统功能页面（仪表盘、上传、手动分析、LLM分析、3D地球等） |
|
||||||
|
| data/ | 离线 GeoIP 数据库 (GeoLite2-City.mmdb、GeoLite2-ASN.mmdb)、自定义 IP 数据 geoip_data.txt |
|
||||||
|
|
||||||
|
项目结构要点
|
||||||
|
天璇/
|
||||||
|
├── analysis/ # 主分析 App
|
||||||
|
├── simple_analysis/ # 简单分析 App (新增)
|
||||||
|
│ ├── views.py # API 视图 (upload, filter, cluster, edges)
|
||||||
|
│ ├── urls.py # 路由 /simple/
|
||||||
|
│ ├── templates/simple_analysis/
|
||||||
|
│ │ └── simple_analysis.html # 单页应用
|
||||||
|
│ └── static/
|
||||||
|
├── config/
|
||||||
|
│ ├── config.yaml # 全局配置（含 API Key、参数）
|
||||||
|
│ └── loader.py
|
||||||
|
├── data/
|
||||||
|
│ ├── GeoLite2-City.mmdb # 130MB，城市级 GeoIP
|
||||||
|
│ ├── GeoLite2-ASN.mmdb # 9.5MB，ASN 数据库
|
||||||
|
│ └── geoip_data.txt # 补充 IP 地理位置
|
||||||
|
├── runtime/python/ # 嵌入式 Python 3.12 运行时
|
||||||
|
├── db.sqlite3 # 主数据库
|
||||||
|
├── main.py / manage.py
|
||||||
|
└── run.bat
|
||||||
|
|
||||||
|
当前开发焦点：简单分析模块 (v3)
|
||||||
|
目标：构建一个完全离线、高性能的网络流量 IP 通信社区发现与可视化工具。
|
||||||
|
工作流（4 步）：
|
||||||
|
1. 上传与预筛选：拖拽上传 CSV，支持高级筛选器（AND/OR/NOT 树状表达式），上传时立即过滤以减少内存占用。
|
||||||
|
2. 高级手动筛选：可视化布尔表达式编辑器，对已加载数据进一步精筛。
|
||||||
|
3. 网络聚类：基于 IP 通信图进行社区发现：
|
||||||
|
- 提取所有唯一 :ips / :ipd 作为节点。
|
||||||
|
- 先按 /24 子网分组，子网内部使用 HDBSCAN (haversine 距离) 二次聚类。
|
||||||
|
- 最终每个节点（IP）获得一个 cluster_id。
|
||||||
|
4. 地图可视化与交互：
|
||||||
|
- 离线地图：使用 Leaflet + 离线瓦片缓存（leaflet-offline），无网络时回退到本地低级别瓦片。
|
||||||
|
- 节点按簇着色，支持 Canvas 渲染器和聚类插件优化性能。
|
||||||
|
- 点击节点展示：IP、运营商、组织、城市、关联对端IP、host/text 等。
|
||||||
|
- 点击边（IP 间连线）展示：通信次数、总流量、首次/末次北京时间、时间规律。
|
||||||
|
- 导出点表/边表 CSV。
|
||||||
|
|
||||||
|
重要技术约定
|
||||||
|
- 列名规范化：所有 CSV 列名通过 normalize_columns() 处理成小写+下划线格式，内部使用标准列名（如 src_ip, dst_ip, cnam 等）。
|
||||||
|
- GeoIP：完全抛弃 CSV 中自带经纬度（:ips.latd 等），所有坐标均使用离线 GeoLite2-City.mmdb 解析。
|
||||||
|
- 时间处理：原始时间戳/时间列统一转为北京时间 (UTC+8) 展示。
|
||||||
|
- 内存管理：上传后的数据保存在内存缓存（_cache dict），以 session_id（12位 hex）索引，无持久化。
|
||||||
|
- 性能要求：i7-10700 + 8GB RAM，十万级 IP 节点需在 1 分钟内完成聚类，内存占用 <2GB。
|
||||||
|
|
||||||
|
前端注意事项
|
||||||
|
- 所有静态资源（Leaflet, Chart.js, Three.js 等）必须本地托管，确保离线可用。
|
||||||
|
- 地图瓦片离线方案必须实现：在线时缓存 OSM 瓦片到 IndexedDB，离线时使用缓存或内置低级别瓦片。
|
||||||
|
- 按钮必须防重复点击，操作栏固定在顶部，调试信息面板实时显示。
|
||||||
|
- 需修复 Cannot set properties of null 类 DOM 操作错误。
|
||||||
|
|
||||||
|
API 端点 (简单分析)
|
||||||
|
| 方法 | 路径 | 作用 |
|
||||||
|
|------|------|------|
|
||||||
|
| GET | /simple/ | 主页面 |
|
||||||
|
| POST | /simple/upload/ | 上传文件 + 预设筛选 + GeoIP 解析 |
|
||||||
|
| POST | /simple/filter/advanced/ | 树状高级筛选 |
|
||||||
|
| GET | /simple/column-values/ | 列 Top 50 唯一值（供下拉菜单） |
|
||||||
|
| POST | /simple/cluster/ | 执行网络聚类 |
|
||||||
|
| GET | /simple/cluster/<label>/ | 簇详情 |
|
||||||
|
| POST | /simple/edges/ | 获取边关系 |
|
||||||
|
|
||||||
|
已知问题与待办
|
||||||
|
- [ ] 离线地图瓦片缓存机制完善（目前仅低级别内置）。
|
||||||
|
- [ ] 边渲染时坐标查找需使用节点真实坐标，而非簇中心近似。
|
||||||
|
- [ ] 大数据量时地图标记性能优化（可换用 WebGL 渲染器）。
|
||||||
|
- [ ] 预设筛选器与 Step 2 筛选器 UI 统一为同一个高级筛选组件。
|
||||||
|
|
||||||
|
开发命令
|
||||||
|
- 启动开发服务器：python manage.py runserver
|
||||||
|
- 运行测试：python manage.py test simple_analysis
|
||||||
|
- 生成要求：pip freeze > requirements.txt (注意剔除内部路径)
|
||||||
|
|
||||||
|
提交规范
|
||||||
|
- 直接提交到 main 分支，git push origin main。
|
||||||
|
- 提交信息格式：feat: xxx / fix: xxx / refactor: xxx，简要描述改动。
|
||||||
|
|
||||||
|
---
|
||||||
|
最后更新: 2026-07-24
|
||||||
|
维护者: 天璇开发团队
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
import csv, os
|
|
||||||
os.chdir(r'E:\hjq\天璇')
|
|
||||||
with open('TlsDB.csv', 'r', encoding='utf-8') as f:
|
|
||||||
reader = csv.DictReader(f)
|
|
||||||
for row in reader:
|
|
||||||
dtype = row.get('data_type', '').strip().upper()
|
|
||||||
code = row.get('field_code', '').strip().lower()
|
|
||||||
if 'BOOL' in dtype or code in ('cnrs', 'isrs'):
|
|
||||||
print(f"{row['field_code']:20s} {dtype:15s} {row.get('meaning', '')[:50]}")
|
|
||||||
Binary file not shown.
@@ -1,73 +0,0 @@
|
|||||||
import urllib.request, json, time, sys
|
|
||||||
|
|
||||||
BASE = 'http://127.0.0.1:18766'
|
|
||||||
RUN_ID = 2 # the upload we just did
|
|
||||||
|
|
||||||
# Step 1: Trigger manual analysis (full pipeline: clustering + UMAP + SVD)
|
|
||||||
print('[1] Starting manual analysis pipeline...')
|
|
||||||
body = json.dumps({
|
|
||||||
'run_id': int(RUN_ID),
|
|
||||||
'algorithm': 'hdbscan',
|
|
||||||
'min_cluster_size': 5,
|
|
||||||
'cluster_mode': 'raw',
|
|
||||||
}).encode('utf-8')
|
|
||||||
|
|
||||||
req = urllib.request.Request(f'{BASE}/analyze/run/', data=body,
|
|
||||||
headers={'Content-Type': 'application/json'})
|
|
||||||
try:
|
|
||||||
r = urllib.request.urlopen(req, timeout=10)
|
|
||||||
resp = json.loads(r.read())
|
|
||||||
r.close()
|
|
||||||
print(f'[1] Started: {resp}')
|
|
||||||
except Exception as e:
|
|
||||||
print(f'[1] FAILED: {e}')
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
# Step 2: Poll until completed or failed
|
|
||||||
print('[2] Polling analysis progress...')
|
|
||||||
for i in range(240):
|
|
||||||
try:
|
|
||||||
r = urllib.request.urlopen(f'{BASE}/runs/{RUN_ID}/status/', timeout=5)
|
|
||||||
s = json.loads(r.read())
|
|
||||||
r.close()
|
|
||||||
except:
|
|
||||||
time.sleep(2)
|
|
||||||
continue
|
|
||||||
|
|
||||||
status = s['status']
|
|
||||||
pct = s.get('progress_pct', 0)
|
|
||||||
msg = s.get('progress_msg', '')
|
|
||||||
if i % 10 == 0:
|
|
||||||
clusters = s.get('cluster_count') or '?'
|
|
||||||
entities = s.get('entity_count') or '?'
|
|
||||||
print(f' [{i}] {status} ({pct}%) clusters={clusters} entities={entities} {msg[:60]}')
|
|
||||||
|
|
||||||
if status == 'completed':
|
|
||||||
print(f'\n[2] DONE!')
|
|
||||||
print(f' status: {status}')
|
|
||||||
print(f' cluster_count: {s.get("cluster_count")}')
|
|
||||||
print(f' entity_count: {s.get("entity_count")}')
|
|
||||||
print(f' progress: {pct}%')
|
|
||||||
break
|
|
||||||
if status == 'failed':
|
|
||||||
err = s.get('error_message', '')[:800]
|
|
||||||
print(f'\n[2] FAILED: {err}')
|
|
||||||
break
|
|
||||||
time.sleep(3)
|
|
||||||
else:
|
|
||||||
print('[2] TIMEOUT after 12 minutes')
|
|
||||||
|
|
||||||
# Step 3: Verify cluster overview page
|
|
||||||
print('[3] Checking cluster overview...')
|
|
||||||
try:
|
|
||||||
r = urllib.request.urlopen(f'{BASE}/clusters/{RUN_ID}/', timeout=10)
|
|
||||||
body = r.read().decode('utf-8', errors='replace')
|
|
||||||
r.close()
|
|
||||||
has_scatter = 'scatter_data' in body
|
|
||||||
has_geo = 'geo_data' in body
|
|
||||||
has_umap = 'UMAP' in body
|
|
||||||
print(f' scatter_data: {has_scatter}')
|
|
||||||
print(f' geo_data: {has_geo}')
|
|
||||||
print(f' UMAP text: {has_umap}')
|
|
||||||
except Exception as e:
|
|
||||||
print(f' FAILED: {e}')
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
import subprocess, time, urllib.request, os, sys
|
|
||||||
|
|
||||||
PROJ_DIR = r"C:\Users\25044\Desktop\Proj\天璇"
|
|
||||||
PYTHON_EXE = os.path.join(PROJ_DIR, "runtime", "python", "python.exe")
|
|
||||||
os.chdir(PROJ_DIR)
|
|
||||||
|
|
||||||
server = subprocess.Popen(
|
|
||||||
[PYTHON_EXE, "manage.py", "runserver", "127.0.0.1:8765", "--noreload"],
|
|
||||||
stdout=subprocess.DEVNULL,
|
|
||||||
stderr=subprocess.DEVNULL
|
|
||||||
)
|
|
||||||
|
|
||||||
print(f"SERVER_PID={server.pid}")
|
|
||||||
|
|
||||||
# Poll with backoff: fast early, then slower
|
|
||||||
deadline = time.time() + 60
|
|
||||||
attempt = 0
|
|
||||||
while time.time() < deadline:
|
|
||||||
attempt += 1
|
|
||||||
try:
|
|
||||||
resp = urllib.request.urlopen("http://127.0.0.1:8765/", timeout=3)
|
|
||||||
if resp.status == 200:
|
|
||||||
print("SERVER_READY")
|
|
||||||
sys.exit(0)
|
|
||||||
except Exception as e:
|
|
||||||
if attempt % 10 == 0:
|
|
||||||
print(f"WAITING... attempt={attempt} err={type(e).__name__}")
|
|
||||||
time.sleep(1)
|
|
||||||
|
|
||||||
print("SERVER_START_TIMEOUT")
|
|
||||||
server.kill()
|
|
||||||
sys.exit(1)
|
|
||||||
@@ -243,6 +243,21 @@
|
|||||||
.sa-point-tooltip .tt-row { display: flex; gap: 0.5rem; }
|
.sa-point-tooltip .tt-row { display: flex; gap: 0.5rem; }
|
||||||
.sa-point-tooltip .tt-key { color: #888; min-width: 70px; }
|
.sa-point-tooltip .tt-key { color: #888; min-width: 70px; }
|
||||||
|
|
||||||
|
/* Country labels on world map */
|
||||||
|
.sa-country-label {
|
||||||
|
font-size: 9px; font-weight: 400; color: #5c5c4a;
|
||||||
|
text-shadow: 0 0 3px #fff, 0 0 3px #fff;
|
||||||
|
white-space: nowrap; pointer-events: none;
|
||||||
|
letter-spacing: 0.5px;
|
||||||
|
}
|
||||||
|
/* Satellite-mode labels */
|
||||||
|
.sa-country-label-sat {
|
||||||
|
color: #fff;
|
||||||
|
text-shadow: 0 1px 3px rgba(0,0,0,0.8), 0 0 6px rgba(0,0,0,0.5);
|
||||||
|
font-weight: 500;
|
||||||
|
letter-spacing: 1px;
|
||||||
|
}
|
||||||
|
|
||||||
/* ── Responsive ──────────────────────────────────── */
|
/* ── Responsive ──────────────────────────────────── */
|
||||||
@media (max-width: 900px) {
|
@media (max-width: 900px) {
|
||||||
.sa-map-row { flex-direction: column; }
|
.sa-map-row { flex-direction: column; }
|
||||||
@@ -1193,37 +1208,161 @@ function getSubnetColor(ip) {
|
|||||||
return SUBNET_COLORS[0];
|
return SUBNET_COLORS[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ── Load offline TopoJSON world map (fallback to CartoDB tiles) ── */
|
/* ── World map: satellite tiles + vector overlay ────── */
|
||||||
function loadOfflineWorldMap(map) {
|
function loadOfflineWorldMap(map) {
|
||||||
const mapFiles = [
|
// Satellite tile providers (free, no API key)
|
||||||
{ url: STATIC_MAPS + '/land-10m.json', obj: 'ne_10m_land', label: '10m精细' },
|
const tileProviders = [
|
||||||
{ url: STATIC_MAPS + '/land-50m.json', obj: 'land', label: '50m标准' },
|
{
|
||||||
{ url: STATIC_MAPS + '/land-110m.json', obj: 'land', label: '110m基础' },
|
name: 'ESRI卫星',
|
||||||
|
url: 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}',
|
||||||
|
attr: '© Esri', maxZoom: 18,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: 'CartoDB浅色',
|
||||||
|
url: 'https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png',
|
||||||
|
attr: '© OSM', subdomains: 'abcd', maxZoom: 19,
|
||||||
|
},
|
||||||
];
|
];
|
||||||
|
|
||||||
function tryLoad(idx) {
|
// Offline vector fallback files
|
||||||
if (idx >= mapFiles.length) {
|
const landFiles = [
|
||||||
console.log('离线地图均不可用，使用在线瓦片');
|
{ url: STATIC_MAPS + '/land-10m.json', obj: 'land', label: '10m' },
|
||||||
L.tileLayer('https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png', {
|
{ url: STATIC_MAPS + '/land-50m.json', obj: 'land', label: '50m' },
|
||||||
attribution: '© <a href="https://www.openstreetmap.org/copyright">OSM</a>',
|
];
|
||||||
subdomains: 'abcd', maxZoom: 19,
|
const countryFiles = [
|
||||||
}).addTo(map);
|
{ url: STATIC_MAPS + '/countries-10m.json', obj: 'countries', label: '10m' },
|
||||||
|
{ url: STATIC_MAPS + '/countries-50m.json', obj: 'countries', label: '50m' },
|
||||||
|
];
|
||||||
|
|
||||||
|
function sanitizeName(name) {
|
||||||
|
if (!name) return '';
|
||||||
|
return name
|
||||||
|
.replace(/Å/g,'A').replace(/å/g,'a').replace(/Á/g,'A').replace(/á/g,'a')
|
||||||
|
.replace(/À/g,'A').replace(/à/g,'a').replace(/Â/g,'A').replace(/â/g,'a')
|
||||||
|
.replace(/Ã/g,'A').replace(/ã/g,'a').replace(/Ä/g,'A').replace(/ä/g,'a')
|
||||||
|
.replace(/Ç/g,'C').replace(/ç/g,'c').replace(/É/g,'E').replace(/é/g,'e')
|
||||||
|
.replace(/È/g,'E').replace(/è/g,'e').replace(/Ê/g,'E').replace(/ê/g,'e')
|
||||||
|
.replace(/Ë/g,'E').replace(/ë/g,'e').replace(/Í/g,'I').replace(/í/g,'i')
|
||||||
|
.replace(/Î/g,'I').replace(/î/g,'i').replace(/Ï/g,'I').replace(/ï/g,'i')
|
||||||
|
.replace(/Ñ/g,'N').replace(/ñ/g,'n').replace(/Ó/g,'O').replace(/ó/g,'o')
|
||||||
|
.replace(/Ô/g,'O').replace(/ô/g,'o').replace(/Õ/g,'O').replace(/õ/g,'o')
|
||||||
|
.replace(/Ö/g,'O').replace(/ö/g,'o').replace(/Ø/g,'O').replace(/ø/g,'o')
|
||||||
|
.replace(/Ú/g,'U').replace(/ú/g,'u').replace(/Ü/g,'U').replace(/ü/g,'u')
|
||||||
|
.replace(/Ý/g,'Y').replace(/ý/g,'y').replace(/Æ/g,'AE').replace(/æ/g,'ae')
|
||||||
|
.replace(/Œ/g,'OE').replace(/œ/g,'oe').replace(/ß/g,'ss');
|
||||||
|
}
|
||||||
|
|
||||||
|
let tileLayer = null;
|
||||||
|
let tileLoaded = false;
|
||||||
|
let vectorLoaded = false;
|
||||||
|
|
||||||
|
// ── Step 1: Try satellite tiles ──
|
||||||
|
function tryTile(providerIdx) {
|
||||||
|
if (providerIdx >= tileProviders.length) {
|
||||||
|
console.log('在线瓦片不可用，加载离线矢量地图');
|
||||||
|
loadVector(0);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const mf = mapFiles[idx];
|
const p = tileProviders[providerIdx];
|
||||||
fetch(mf.url)
|
const opts = { attribution: p.attr, maxZoom: p.maxZoom };
|
||||||
.then(r => { if (!r.ok) throw new Error('HTTP ' + r.status); return r.json(); })
|
if (p.subdomains) opts.subdomains = p.subdomains;
|
||||||
|
|
||||||
|
const testTile = p.url
|
||||||
|
.replace('{s}', p.subdomains ? p.subdomains[0] : 'a')
|
||||||
|
.replace('{z}', '2').replace('{y}', '1').replace('{x}', '1')
|
||||||
|
.replace('{r}', '');
|
||||||
|
|
||||||
|
fetch(testTile)
|
||||||
|
.then(r => { if (!r.ok) throw new Error(); return r.blob(); })
|
||||||
|
.then(() => {
|
||||||
|
tileLayer = L.tileLayer(p.url, opts).addTo(map);
|
||||||
|
tileLoaded = true;
|
||||||
|
console.log('在线地图: ' + p.name);
|
||||||
|
// Still load vector borders overlay
|
||||||
|
loadVector(0);
|
||||||
|
})
|
||||||
|
.catch(() => tryTile(providerIdx + 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Step 2: Vector overlay (land + borders) ──
|
||||||
|
function loadVector(landIdx) {
|
||||||
|
if (landIdx >= landFiles.length) { return; }
|
||||||
|
fetch(landFiles[landIdx].url)
|
||||||
|
.then(r => { if (!r.ok) throw new Error(); return r.json(); })
|
||||||
.then(topo => {
|
.then(topo => {
|
||||||
const geojson = topojson.feature(topo, topo.objects[mf.obj]);
|
if (!tileLoaded) {
|
||||||
|
// No satellite — opaque land fill
|
||||||
|
L.geoJSON(topojson.feature(topo, topo.objects.land), {
|
||||||
|
style: { fillColor: '#2a2a2a', color: '#444', weight: 0.5, fillOpacity: 1 },
|
||||||
|
interactive: false,
|
||||||
|
}).addTo(map);
|
||||||
|
map.getContainer().style.background = '#1a1a2e';
|
||||||
|
}
|
||||||
|
console.log('矢量陆地: ' + landFiles[landIdx].label);
|
||||||
|
loadBorders(0);
|
||||||
|
})
|
||||||
|
.catch(() => loadVector(landIdx + 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadBorders(idx) {
|
||||||
|
if (idx >= countryFiles.length) { vectorLoaded = true; return; }
|
||||||
|
fetch(countryFiles[idx].url)
|
||||||
|
.then(r => { if (!r.ok) throw new Error(); return r.json(); })
|
||||||
|
.then(topo => {
|
||||||
|
const geojson = topojson.feature(topo, topo.objects.countries);
|
||||||
|
const feats = geojson.features || [];
|
||||||
|
const isSat = tileLoaded;
|
||||||
|
|
||||||
L.geoJSON(geojson, {
|
L.geoJSON(geojson, {
|
||||||
style: { fillColor: '#dde4ed', color: '#b0bccb', weight: 0.5, fillOpacity: 1 },
|
style: {
|
||||||
|
fillColor: 'transparent', fillOpacity: 0,
|
||||||
|
color: isSat ? 'rgba(255,255,255,0.55)' : '#666',
|
||||||
|
weight: isSat ? 1.0 : 0.6,
|
||||||
|
opacity: isSat ? 0.7 : 0.5,
|
||||||
|
dashArray: isSat ? null : '3 2',
|
||||||
|
},
|
||||||
interactive: false,
|
interactive: false,
|
||||||
}).addTo(map);
|
}).addTo(map);
|
||||||
console.log('离线世界地图: ' + mf.label + ' (' + mf.url + ')');
|
|
||||||
|
// Labels
|
||||||
|
const labelGroup = L.layerGroup();
|
||||||
|
feats.forEach(function(f) {
|
||||||
|
const name = sanitizeName((f.properties && f.properties.name) || '');
|
||||||
|
if (!name || name.length > 25) return;
|
||||||
|
let c = null;
|
||||||
|
const g = f.geometry;
|
||||||
|
if (g && g.type === 'Polygon') c = ringCenter(g.coordinates[0]);
|
||||||
|
else if (g && g.type === 'MultiPolygon') {
|
||||||
|
let m = 0;
|
||||||
|
for (const p of g.coordinates) {
|
||||||
|
if (p[0].length > m) { m = p[0].length; c = ringCenter(p[0]); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (c && Math.abs(c[0]) < 82) {
|
||||||
|
L.marker(c, { icon: L.divIcon({
|
||||||
|
className: 'sa-country-label' + (isSat ? ' sa-country-label-sat' : ''),
|
||||||
|
html: name, iconSize: null,
|
||||||
|
iconAnchor: [name.length * 3.5, 7],
|
||||||
|
}), interactive: false }).addTo(labelGroup);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
map.on('zoomend', function() {
|
||||||
|
map.getZoom() >= 3 ? labelGroup.addTo(map) : map.removeLayer(labelGroup);
|
||||||
|
});
|
||||||
|
if (map.getZoom() >= 3) labelGroup.addTo(map);
|
||||||
|
vectorLoaded = true;
|
||||||
|
console.log('矢量国界: ' + countryFiles[idx].label + ', ' + feats.length + '国');
|
||||||
})
|
})
|
||||||
.catch(err => { console.log('地图 ' + mf.label + ' 不可用:', err.message); tryLoad(idx + 1); });
|
.catch(() => loadBorders(idx + 1));
|
||||||
}
|
}
|
||||||
tryLoad(0);
|
|
||||||
|
function ringCenter(coords) {
|
||||||
|
let la = 0, lo = 0, n = coords.length;
|
||||||
|
for (let i = 0; i < n; i++) { la += coords[i][1]; lo += coords[i][0]; }
|
||||||
|
return n > 0 ? [la / n, lo / n] : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
tryTile(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
function initMap() {
|
function initMap() {
|
||||||
@@ -1232,7 +1371,7 @@ function initMap() {
|
|||||||
container.style.height = '550px';
|
container.style.height = '550px';
|
||||||
container.style.width = '100%';
|
container.style.width = '100%';
|
||||||
// Set ocean background color (world map rendered via L.imageOverlay)
|
// Set ocean background color (world map rendered via L.imageOverlay)
|
||||||
container.style.background = '#b9c8da';
|
container.style.background = '#1a1a2e'; // dark ocean (overridden by satellite tiles)
|
||||||
|
|
||||||
if (SA.map) { SA.map.invalidateSize(); renderMapData(); return; }
|
if (SA.map) { SA.map.invalidateSize(); renderMapData(); return; }
|
||||||
|
|
||||||
@@ -1511,22 +1650,60 @@ function showNodeDetailPanel(data) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const srvBadge = n.is_server ? ' 🖥 服务器' : '';
|
const srvBadge = n.is_server ? ' 🖥 服务器' : '';
|
||||||
|
|
||||||
|
// Enriched node attributes
|
||||||
|
const attrToHtml = (items, title, icon) => {
|
||||||
|
if (!items || !items.length) return '';
|
||||||
|
return `<div class="section-title">${icon} ${title}</div>
|
||||||
|
<div>${items.slice(0,5).map(i => `<span class="sa-tag">${escapeHtml(i.value||i)} (${i.count||''})</span>`).join(' ')}</div>`;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Traffic stats row
|
||||||
|
const totalBytes = n.total_bytes || data.total_flow_bytes || 0;
|
||||||
|
const totalPkts = n.total_packets;
|
||||||
|
const avgDur = n.avg_duration;
|
||||||
|
const connRate = n.conn_success_rate;
|
||||||
|
let trafficHtml = '<div class="stat-row">';
|
||||||
|
trafficHtml += `<div class="stat-card"><div class="num">${totalBytes ? (totalBytes/1024).toFixed(1)+' KB' : '—'}</div><div class="lbl">总流量</div></div>`;
|
||||||
|
if (totalPkts != null) trafficHtml += `<div class="stat-card"><div class="num">${totalPkts.toLocaleString()}</div><div class="lbl">总包数</div></div>`;
|
||||||
|
if (avgDur != null) trafficHtml += `<div class="stat-card"><div class="num">${avgDur}s</div><div class="lbl">平均持续时间</div></div>`;
|
||||||
|
trafficHtml += '</div>';
|
||||||
|
|
||||||
|
if (connRate != null || n.conn_count) {
|
||||||
|
trafficHtml += '<div class="stat-row">';
|
||||||
|
if (connRate != null) trafficHtml += `<div class="stat-card"><div class="num">${connRate}%</div><div class="lbl">连接成功率</div></div>`;
|
||||||
|
if (n.conn_count) trafficHtml += `<div class="stat-card"><div class="num">${n.conn_count}</div><div class="lbl">总连接数</div></div>`;
|
||||||
|
trafficHtml += '</div>';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Time range
|
||||||
|
if (n.first_seen || n.last_seen) {
|
||||||
|
trafficHtml += '<div class="stat-row">';
|
||||||
|
if (n.first_seen) trafficHtml += `<div class="stat-card"><div class="num" style="font-size:0.7rem;">${n.first_seen.slice(0,19)}</div><div class="lbl">首次出现</div></div>`;
|
||||||
|
if (n.last_seen) trafficHtml += `<div class="stat-card"><div class="num" style="font-size:0.7rem;">${n.last_seen.slice(0,19)}</div><div class="lbl">末次出现</div></div>`;
|
||||||
|
trafficHtml += '</div>';
|
||||||
|
}
|
||||||
|
|
||||||
body.innerHTML =
|
body.innerHTML =
|
||||||
`<div class="stat-row">` +
|
`<div class="stat-row">` +
|
||||||
`<div class="stat-card"><div class="num" style="font-size:1rem;word-break:break-all;">${escapeHtml(n.ip)}${srvBadge}</div><div class="lbl">IP 地址</div></div>` +
|
`<div class="stat-card"><div class="num" style="font-size:1rem;word-break:break-all;">${escapeHtml(n.ip)}${srvBadge}</div><div class="lbl">IP 地址</div></div>` +
|
||||||
`<div class="stat-card"><div class="num">社区 ${n.cluster_id}</div><div class="lbl">所属社区</div></div></div>` +
|
(n.host ? `<div class="stat-card"><div class="num" style="font-size:0.8rem;">${escapeHtml(n.host)}</div><div class="lbl">Host</div></div>` : '') +
|
||||||
|
`</div>` +
|
||||||
`<div class="stat-row">` +
|
`<div class="stat-row">` +
|
||||||
(n.city ? `<div class="stat-card"><div class="num">${escapeHtml(n.city)}</div><div class="lbl">城市</div></div>` : '') +
|
(n.city ? `<div class="stat-card"><div class="num">${escapeHtml(n.city)}</div><div class="lbl">城市</div></div>` : '') +
|
||||||
(n.country ? `<div class="stat-card"><div class="num">${escapeHtml(n.country)}</div><div class="lbl">国家</div></div>` : '') +
|
(n.country ? `<div class="stat-card"><div class="num">${escapeHtml(n.country)}</div><div class="lbl">国家</div></div>` : '') +
|
||||||
(n.isp ? `<div class="stat-card"><div class="num" style="font-size:0.85rem;">${escapeHtml(n.isp)}</div><div class="lbl">运营商</div></div>` : '') +
|
(n.isp ? `<div class="stat-card"><div class="num" style="font-size:0.85rem;">${escapeHtml(n.isp)}</div><div class="lbl">运营商</div></div>` : '') +
|
||||||
`</div>` +
|
|
||||||
`<div class="stat-row">` +
|
|
||||||
(n.host ? `<div class="stat-card"><div class="num" style="font-size:0.8rem;">${escapeHtml(n.host)}</div><div class="lbl">Host</div></div>` : '') +
|
|
||||||
(n.org ? `<div class="stat-card"><div class="num" style="font-size:0.85rem;">${escapeHtml(n.org)}</div><div class="lbl">组织</div></div>` : '') +
|
(n.org ? `<div class="stat-card"><div class="num" style="font-size:0.85rem;">${escapeHtml(n.org)}</div><div class="lbl">组织</div></div>` : '') +
|
||||||
`</div>` +
|
`</div>` +
|
||||||
`<div class="stat-row">` +
|
`<div class="stat-row">` +
|
||||||
`<div class="stat-card"><div class="num">${data.n_peers || 0}</div><div class="lbl">对端IP数</div></div>` +
|
`<div class="stat-card"><div class="num">${data.n_peers || 0}</div><div class="lbl">对端IP数</div></div>` +
|
||||||
`<div class="stat-card"><div class="num">${data.total_flow_bytes ? (data.total_flow_bytes/1024).toFixed(1) + ' KB' : '—'}</div><div class="lbl">总流量</div></div></div>` +
|
`<div class="stat-card"><div class="num">${n.asn || '—'}</div><div class="lbl">ASN</div></div></div>` +
|
||||||
|
trafficHtml +
|
||||||
|
attrToHtml(n.top_snis, 'SNI / 域名', '🌐') +
|
||||||
|
attrToHtml(n.top_tls_vers, 'TLS 版本', '🔒') +
|
||||||
|
attrToHtml(n.top_ciphers, '加密套件', '🔑') +
|
||||||
|
attrToHtml(n.top_ports, '目的端口', '🔌') +
|
||||||
|
attrToHtml(n.top_categories, '分类标签', '🏷') +
|
||||||
textsHtml +
|
textsHtml +
|
||||||
peersHtml +
|
peersHtml +
|
||||||
edgeHtml;
|
edgeHtml;
|
||||||
@@ -1573,20 +1750,20 @@ function highlightNodeEdges() {
|
|||||||
interactive: false, bubblingMouseEvents: true,
|
interactive: false, bubblingMouseEvents: true,
|
||||||
});
|
});
|
||||||
const pair = L.layerGroup([hitPoly, visPoly]);
|
const pair = L.layerGroup([hitPoly, visPoly]);
|
||||||
pair.bindTooltip(
|
hitPoly.bindTooltip(
|
||||||
`${escapeHtml(e.source)} → ${escapeHtml(e.target)}<br>` +
|
`${escapeHtml(e.source)} → ${escapeHtml(e.target)}<br>` +
|
||||||
`${e.comm_count} 次 · ${(e.total_bytes/1024).toFixed(1)} KB` +
|
`${e.comm_count} 次 · ${(e.total_bytes/1024).toFixed(1)} KB` +
|
||||||
`<br><span style="color:#aaa;">🖱 点击查看详情</span>`,
|
`<br><span style="color:#aaa;">🖱 点击查看详情</span>`,
|
||||||
{sticky: true, direction: 'top'}
|
{sticky: true, direction: 'top'}
|
||||||
);
|
);
|
||||||
pair.on('click', function(ev) {
|
hitPoly.on('click', function(ev) {
|
||||||
L.DomEvent.stopPropagation(ev);
|
L.DomEvent.stopPropagation(ev);
|
||||||
loadEdgeDetail(e);
|
loadEdgeDetail(e);
|
||||||
});
|
});
|
||||||
pair.on('mouseover', function() {
|
hitPoly.on('mouseover', function() {
|
||||||
visPoly.setStyle({ color: '#e60000', weight: Math.min(w + 2, 8), opacity: 1 });
|
visPoly.setStyle({ color: '#e60000', weight: Math.min(w + 2, 8), opacity: 1 });
|
||||||
});
|
});
|
||||||
pair.on('mouseout', function() {
|
hitPoly.on('mouseout', function() {
|
||||||
visPoly.setStyle({ color: '#ff3333', weight: w, opacity: 0.8 });
|
visPoly.setStyle({ color: '#ff3333', weight: w, opacity: 0.8 });
|
||||||
});
|
});
|
||||||
polys.push(pair);
|
polys.push(pair);
|
||||||
@@ -1596,7 +1773,10 @@ function highlightNodeEdges() {
|
|||||||
_edgeLayer = L.layerGroup(polys).addTo(SA.map);
|
_edgeLayer = L.layerGroup(polys).addTo(SA.map);
|
||||||
// Fit map to show these edges
|
// Fit map to show these edges
|
||||||
const allPts = [];
|
const allPts = [];
|
||||||
polys.forEach(p => { allPts.push(p.getLatLngs()[0]); allPts.push(p.getLatLngs()[1]); });
|
polys.forEach(p => {
|
||||||
|
const latlngs = p.getLayers()[0].getLatLngs();
|
||||||
|
allPts.push(latlngs[0]); allPts.push(latlngs[1]);
|
||||||
|
});
|
||||||
if (allPts.length) {
|
if (allPts.length) {
|
||||||
try { SA.map.fitBounds(L.latLngBounds(allPts), {padding: [50, 50], maxZoom: 8}); } catch(e) {}
|
try { SA.map.fitBounds(L.latLngBounds(allPts), {padding: [50, 50], maxZoom: 8}); } catch(e) {}
|
||||||
}
|
}
|
||||||
@@ -1674,24 +1854,24 @@ function renderEdgesOnMap() {
|
|||||||
interactive: false, bubblingMouseEvents: true,
|
interactive: false, bubblingMouseEvents: true,
|
||||||
});
|
});
|
||||||
|
|
||||||
// 将两条线绑定为一个组，共享 tooltip 和 click
|
// 将两条线绑定为一个组
|
||||||
const pair = L.layerGroup([hitPoly, visPoly]);
|
const pair = L.layerGroup([hitPoly, visPoly]);
|
||||||
|
|
||||||
pair.bindTooltip(
|
hitPoly.bindTooltip(
|
||||||
`<b>${escapeHtml(e.source)} → ${escapeHtml(e.target)}</b><br>` +
|
`<b>${escapeHtml(e.source)} → ${escapeHtml(e.target)}</b><br>` +
|
||||||
`${e.comm_count} 次 · ${e.total_bytes ? (e.total_bytes/1024).toFixed(1)+'KB' : ''}` +
|
`${e.comm_count} 次 · ${e.total_bytes ? (e.total_bytes/1024).toFixed(1)+'KB' : ''}` +
|
||||||
`<br><span style="color:#aaa;">🖱 点击查看详情</span>`,
|
`<br><span style="color:#aaa;">🖱 点击查看详情</span>`,
|
||||||
{sticky: true, direction: 'top'}
|
{sticky: true, direction: 'top'}
|
||||||
);
|
);
|
||||||
pair.on('click', function(ev) {
|
hitPoly.on('click', function(ev) {
|
||||||
L.DomEvent.stopPropagation(ev);
|
L.DomEvent.stopPropagation(ev);
|
||||||
loadEdgeDetail(e);
|
loadEdgeDetail(e);
|
||||||
});
|
});
|
||||||
// Hover 高亮
|
// Hover 高亮
|
||||||
pair.on('mouseover', function() {
|
hitPoly.on('mouseover', function() {
|
||||||
visPoly.setStyle({ color: '#e63946', weight: Math.min(weight + 2, 8), opacity: 0.95 });
|
visPoly.setStyle({ color: '#e63946', weight: Math.min(weight + 2, 8), opacity: 0.95 });
|
||||||
});
|
});
|
||||||
pair.on('mouseout', function() {
|
hitPoly.on('mouseout', function() {
|
||||||
visPoly.setStyle({ color: '#ff6b6b', weight: weight, opacity: opacity });
|
visPoly.setStyle({ color: '#ff6b6b', weight: weight, opacity: opacity });
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -1706,6 +1886,10 @@ function loadEdgeDetail(e) {
|
|||||||
const panel = document.getElementById('saDetailPanel');
|
const panel = document.getElementById('saDetailPanel');
|
||||||
const body = document.getElementById('saDetailBody');
|
const body = document.getElementById('saDetailBody');
|
||||||
if (!panel || !body) return;
|
if (!panel || !body) return;
|
||||||
|
|
||||||
|
// Destroy previous chart if any
|
||||||
|
if (SA._edgeTimeChart) { SA._edgeTimeChart.destroy(); SA._edgeTimeChart = null; }
|
||||||
|
|
||||||
setText('saDetailTitle', `边详情`);
|
setText('saDetailTitle', `边详情`);
|
||||||
// Convert timestamps to Beijing time if possible
|
// Convert timestamps to Beijing time if possible
|
||||||
const toBeijing = (ts) => {
|
const toBeijing = (ts) => {
|
||||||
@@ -1731,11 +1915,31 @@ function loadEdgeDetail(e) {
|
|||||||
const dstCity = e.dst_city || '';
|
const dstCity = e.dst_city || '';
|
||||||
const srcCountry = e.src_country || '';
|
const srcCountry = e.src_country || '';
|
||||||
const dstCountry = e.dst_country || '';
|
const dstCountry = e.dst_country || '';
|
||||||
|
|
||||||
|
// Enriched edge attributes helper
|
||||||
|
const edgeAttrToHtml = (items, title, icon) => {
|
||||||
|
if (!items || !items.length) return '';
|
||||||
|
return `<div class="section-title">${icon} ${title}</div>
|
||||||
|
<div>${items.slice(0,4).map(i => `<span class="sa-tag" style="font-size:0.75rem;">${escapeHtml(i.value||i)} (${i.count||''})</span>`).join(' ')}</div>`;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Extra traffic stats row
|
||||||
|
const totalPkts = e.total_packets;
|
||||||
|
const avgDur = e.avg_duration;
|
||||||
|
let extraStats = '';
|
||||||
|
if (totalPkts != null || avgDur != null) {
|
||||||
|
extraStats = '<div class="stat-row">';
|
||||||
|
if (totalPkts != null) extraStats += `<div class="stat-card"><div class="num">${Number(totalPkts).toLocaleString()}</div><div class="lbl">总包数</div></div>`;
|
||||||
|
if (avgDur != null) extraStats += `<div class="stat-card"><div class="num">${avgDur}s</div><div class="lbl">平均持续时间</div></div>`;
|
||||||
|
extraStats += '</div>';
|
||||||
|
}
|
||||||
|
|
||||||
body.innerHTML =
|
body.innerHTML =
|
||||||
`<div class="stat-row">` +
|
`<div class="stat-row">` +
|
||||||
`<div class="stat-card"><div class="num">${e.comm_count}</div><div class="lbl">通信次数</div></div>` +
|
`<div class="stat-card"><div class="num">${e.comm_count}</div><div class="lbl">通信次数</div></div>` +
|
||||||
`<div class="stat-card"><div class="num">${(e.total_bytes/1024).toFixed(1)} KB</div><div class="lbl">总流量</div></div>` +
|
`<div class="stat-card"><div class="num">${(e.total_bytes/1024).toFixed(1)} KB</div><div class="lbl">总流量</div></div>` +
|
||||||
`<div class="stat-card"><div class="num">${avgBytes} B</div><div class="lbl">平均单次流量</div></div></div>` +
|
`<div class="stat-card"><div class="num">${avgBytes} B</div><div class="lbl">平均单次流量</div></div></div>` +
|
||||||
|
extraStats +
|
||||||
`<div class="stat-row">` +
|
`<div class="stat-row">` +
|
||||||
`<div class="stat-card"><div class="num" style="font-size:0.8rem;">${toBeijing(e.first_seen)}</div><div class="lbl">首次通信</div></div>` +
|
`<div class="stat-card"><div class="num" style="font-size:0.8rem;">${toBeijing(e.first_seen)}</div><div class="lbl">首次通信</div></div>` +
|
||||||
`<div class="stat-card"><div class="num" style="font-size:0.8rem;">${toBeijing(e.last_seen)}</div><div class="lbl">末次通信</div></div></div>` +
|
`<div class="stat-card"><div class="num" style="font-size:0.8rem;">${toBeijing(e.last_seen)}</div><div class="lbl">末次通信</div></div></div>` +
|
||||||
@@ -1753,8 +1957,135 @@ function loadEdgeDetail(e) {
|
|||||||
${dstHost ? ` <span style="font-size:0.75rem;color:#666;">${escapeHtml(dstHost)}</span>` : ''}
|
${dstHost ? ` <span style="font-size:0.75rem;color:#666;">${escapeHtml(dstHost)}</span>` : ''}
|
||||||
${dstOrg ? ` <span style="font-size:0.75rem;color:#888;">· ${escapeHtml(dstOrg)}</span>` : ''}
|
${dstOrg ? ` <span style="font-size:0.75rem;color:#888;">· ${escapeHtml(dstOrg)}</span>` : ''}
|
||||||
${dstCity ? ` <span style="font-size:0.75rem;color:#999;">· ${escapeHtml(dstCity)}${dstCountry ? ', ' + escapeHtml(dstCountry) : ''}</span>` : ''}
|
${dstCity ? ` <span style="font-size:0.75rem;color:#999;">· ${escapeHtml(dstCity)}${dstCountry ? ', ' + escapeHtml(dstCountry) : ''}</span>` : ''}
|
||||||
</div>`;
|
</div>` +
|
||||||
|
edgeAttrToHtml(e.top_snis, 'SNI / 域名', '🌐') +
|
||||||
|
edgeAttrToHtml(e.top_tls_vers, 'TLS 版本', '🔒') +
|
||||||
|
edgeAttrToHtml(e.top_ciphers, '加密套件', '🔑') +
|
||||||
|
edgeAttrToHtml(e.top_ports, '目的端口', '🔌') +
|
||||||
|
edgeAttrToHtml(e.top_categories, '分类标签', '🏷') +
|
||||||
|
`<div class="section-title">📈 流量时序图 (北京时间)</div>
|
||||||
|
<div id="saEdgeTsChart" style="position:relative; width:100%; height:220px;">
|
||||||
|
<canvas id="saEdgeTimeCanvas" style="width:100%;height:100%;"></canvas>
|
||||||
|
</div>` +
|
||||||
|
`<div style="font-size:0.7rem;color:#999;text-align:center;margin-top:0.25rem;" id="saEdgeTsNote"></div>`;
|
||||||
panel.classList.add('open');
|
panel.classList.add('open');
|
||||||
|
|
||||||
|
// ── Fetch time-series data ──
|
||||||
|
const tsContainer = document.getElementById('saEdgeTsChart');
|
||||||
|
const tsNote = document.getElementById('saEdgeTsNote');
|
||||||
|
if (!tsContainer || !tsNote) return;
|
||||||
|
tsNote.textContent = '加载中...';
|
||||||
|
|
||||||
|
fetch('/simple/edge-time-series/', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json', 'X-CSRFToken': getCSRF()},
|
||||||
|
body: JSON.stringify({session_id: SA.sessionId, source: e.source, target: e.target}),
|
||||||
|
})
|
||||||
|
.then(r => r.json())
|
||||||
|
.then(data => {
|
||||||
|
if (data.error) {
|
||||||
|
tsNote.textContent = '⚠ ' + data.error;
|
||||||
|
tsContainer.style.display = 'none';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const series = data.time_series || [];
|
||||||
|
if (!series.length) {
|
||||||
|
tsNote.textContent = '无时间数据';
|
||||||
|
tsContainer.style.display = 'none';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Build config for dual-axis chart
|
||||||
|
const labels = series.map(d => d.t);
|
||||||
|
const countData = series.map(d => d.count);
|
||||||
|
|
||||||
|
const datasets = [{
|
||||||
|
label: '通信次数',
|
||||||
|
data: countData,
|
||||||
|
borderColor: '#4361ee',
|
||||||
|
backgroundColor: 'rgba(67,97,238,0.08)',
|
||||||
|
fill: true,
|
||||||
|
tension: 0.3,
|
||||||
|
pointRadius: series.length > 60 ? 0 : 2,
|
||||||
|
borderWidth: 2,
|
||||||
|
yAxisID: 'y',
|
||||||
|
}];
|
||||||
|
|
||||||
|
if (data.has_bytes) {
|
||||||
|
const bytesData = series.map(d => d.bytes / 1024);
|
||||||
|
datasets.push({
|
||||||
|
label: '流量 (KB)',
|
||||||
|
data: bytesData,
|
||||||
|
borderColor: '#e63946',
|
||||||
|
backgroundColor: 'rgba(230,57,70,0.06)',
|
||||||
|
fill: true,
|
||||||
|
tension: 0.3,
|
||||||
|
pointRadius: series.length > 60 ? 0 : 2,
|
||||||
|
borderWidth: 2,
|
||||||
|
yAxisID: 'y1',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const ctx = document.getElementById('saEdgeTimeCanvas');
|
||||||
|
if (!ctx) return;
|
||||||
|
SA._edgeTimeChart = new Chart(ctx.getContext('2d'), {
|
||||||
|
type: 'line',
|
||||||
|
data: { labels, datasets },
|
||||||
|
options: {
|
||||||
|
responsive: true,
|
||||||
|
maintainAspectRatio: false,
|
||||||
|
interaction: { mode: 'index', intersect: false },
|
||||||
|
plugins: {
|
||||||
|
legend: {
|
||||||
|
display: true,
|
||||||
|
position: 'top',
|
||||||
|
labels: { boxWidth: 12, font: { size: 11 }, padding: 16 }
|
||||||
|
},
|
||||||
|
tooltip: {
|
||||||
|
callbacks: {
|
||||||
|
title: function(items) { return '🕐 ' + items[0].label + ' (北京)'; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
scales: {
|
||||||
|
x: {
|
||||||
|
display: true,
|
||||||
|
ticks: {
|
||||||
|
font: { size: 10 },
|
||||||
|
maxTicksLimit: series.length > 30 ? 12 : series.length,
|
||||||
|
maxRotation: 45,
|
||||||
|
},
|
||||||
|
grid: { display: false }
|
||||||
|
},
|
||||||
|
y: {
|
||||||
|
type: 'linear',
|
||||||
|
display: true,
|
||||||
|
position: 'left',
|
||||||
|
title: { display: true, text: '次数', font: { size: 10 } },
|
||||||
|
beginAtZero: true,
|
||||||
|
ticks: { font: { size: 10 } },
|
||||||
|
grid: { color: '#f0f0f0' }
|
||||||
|
},
|
||||||
|
y1: {
|
||||||
|
type: 'linear',
|
||||||
|
display: data.has_bytes,
|
||||||
|
position: 'right',
|
||||||
|
title: { display: true, text: 'KB', font: { size: 10 } },
|
||||||
|
beginAtZero: true,
|
||||||
|
ticks: { font: { size: 10 } },
|
||||||
|
grid: { display: false }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const spanH = data.span_hours || 0;
|
||||||
|
const bucketM = data.bucket_minutes || 0;
|
||||||
|
tsNote.textContent = `${series.length} 个时间桶 · 跨度 ${spanH.toFixed(1)}h · 桶粒度 ${bucketM}min · ${data.n_raw} 条原始记录`;
|
||||||
|
})
|
||||||
|
.catch(err => {
|
||||||
|
tsNote.textContent = '⚠ 加载失败: ' + err.message;
|
||||||
|
tsContainer.style.display = 'none';
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
function toggleEdges() {
|
function toggleEdges() {
|
||||||
@@ -1838,72 +2169,271 @@ function downloadCSV(csv, filename) {
|
|||||||
.catch(() => {});
|
.catch(() => {});
|
||||||
})();
|
})();
|
||||||
|
|
||||||
/* ── Cluster detail ─────────────────────────────────────── */
|
/* ── Cluster detail + focus ────────────────────────────── */
|
||||||
|
let _clusterFocusLayer = null; // LayerGroup for cluster-focused edges
|
||||||
|
let _clusterNodeLayer = null; // LayerGroup for focused node markers
|
||||||
|
|
||||||
function loadClusterDetail(label) {
|
function loadClusterDetail(label) {
|
||||||
showLoading('加载集群详情...');
|
showLoading('加载集群详情...');
|
||||||
fetch(`/simple/cluster/${label}/?session_id=${SA.sessionId}`, {
|
fetch('/simple/cluster-nodes-edges/', {
|
||||||
headers: {'X-CSRFToken': getCSRF()},
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json', 'X-CSRFToken': getCSRF()},
|
||||||
|
body: JSON.stringify({session_id: SA.sessionId, cluster_label: label}),
|
||||||
})
|
})
|
||||||
.then(r => r.json())
|
.then(r => r.json())
|
||||||
.then(data => {
|
.then(data => {
|
||||||
hideLoading();
|
hideLoading();
|
||||||
if (data.error) { showError(data.error); return; }
|
if (data.error) { showError(data.error); return; }
|
||||||
showClusterDetailPanel(data);
|
focusClusterOnMap(data);
|
||||||
|
showClusterDetailPanelV2(data);
|
||||||
})
|
})
|
||||||
.catch(err => { hideLoading(); showError('加载失败: ' + err.message); });
|
.catch(err => { hideLoading(); showError('加载失败: ' + err.message); });
|
||||||
}
|
}
|
||||||
|
|
||||||
function showClusterDetailPanel(data) {
|
function focusClusterOnMap(data) {
|
||||||
|
if (!SA.map) return;
|
||||||
|
clearClusterFocus();
|
||||||
|
|
||||||
|
// 1. Zoom to bounds
|
||||||
|
const bounds = data.geo_bounds;
|
||||||
|
if (bounds) {
|
||||||
|
try {
|
||||||
|
SA.map.fitBounds(
|
||||||
|
[[bounds.lat_min, bounds.lon_min], [bounds.lat_max, bounds.lon_max]],
|
||||||
|
{padding: [60, 60], maxZoom: 10}
|
||||||
|
);
|
||||||
|
} catch(e) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Highlight cluster nodes with larger markers
|
||||||
|
const clusterNodes = data.nodes || [];
|
||||||
|
const clusterIps = new Set(clusterNodes.map(n => n.ip));
|
||||||
|
const clusterColors = {};
|
||||||
|
clusterNodes.forEach(n => {
|
||||||
|
clusterColors[n.ip] = getSubnetColor(n.ip);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Redraw all markers in focus mode
|
||||||
|
const allNodes = SA.nodes || [];
|
||||||
|
const markers = [];
|
||||||
|
const centerMarkers = [];
|
||||||
|
|
||||||
|
allNodes.forEach(n => {
|
||||||
|
if (n.lat == null || n.lon == null) return;
|
||||||
|
const inCluster = clusterIps.has(n.ip);
|
||||||
|
const color = inCluster ? (clusterColors[n.ip] || '#4361ee') : '#ccc';
|
||||||
|
const radius = inCluster ? 10 : 4;
|
||||||
|
const opacity = inCluster ? 1 : 0.35;
|
||||||
|
|
||||||
|
const circle = L.circleMarker([n.lat, n.lon], {
|
||||||
|
radius, fillColor: color, color: inCluster ? '#333' : '#ddd',
|
||||||
|
weight: inCluster ? 2 : 1, opacity, fillOpacity: opacity,
|
||||||
|
interactive: true, bubblingMouseEvents: false,
|
||||||
|
});
|
||||||
|
circle.bindTooltip(buildNodeTooltip(n), {sticky: true, direction: 'top', offset: [0, -8]});
|
||||||
|
circle.on('click', function(e) { L.DomEvent.stopPropagation(e); loadNodeDetail(n.ip); });
|
||||||
|
markers.push(circle);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Community center markers for other clusters
|
||||||
|
const clusters = SA.clusterMap || {};
|
||||||
|
const colors = generateColors(20);
|
||||||
|
const clusterColorMap = {};
|
||||||
|
Object.values(clusters).sort((a, b) => b.size - a.size).forEach((c, i) => {
|
||||||
|
clusterColorMap[c.label] = colors[i % colors.length];
|
||||||
|
});
|
||||||
|
|
||||||
|
Object.entries(clusters).forEach(([cid, cinfo]) => {
|
||||||
|
const cidInt = parseInt(cid);
|
||||||
|
if (cidInt === data.cluster_label) return; // Skip focused cluster
|
||||||
|
if (cinfo.size <= 1 || SA.hiddenClusters.has(cidInt)) return;
|
||||||
|
if (cinfo.center_lat == null || cinfo.center_lon == null) return;
|
||||||
|
const color = clusterColorMap[cidInt] || '#4361ee';
|
||||||
|
const circle = L.circleMarker([cinfo.center_lat, cinfo.center_lon], {
|
||||||
|
radius: 8, fillColor: color, color: '#fff', weight: 2,
|
||||||
|
opacity: 0.7, fillOpacity: 0.5, interactive: true, bubblingMouseEvents: false,
|
||||||
|
});
|
||||||
|
circle.bindTooltip(`<b>社区 ${cidInt}</b><br>${cinfo.size} IP`, {direction: 'top'});
|
||||||
|
circle.on('click', function(e) { L.DomEvent.stopPropagation(e); loadClusterDetail(cidInt); });
|
||||||
|
centerMarkers.push(circle);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Replace point layers
|
||||||
|
if (SA.pointLayer) SA.map.removeLayer(SA.pointLayer);
|
||||||
|
if (SA.centerLayer) SA.map.removeLayer(SA.centerLayer);
|
||||||
|
SA.pointLayer = L.layerGroup(markers).addTo(SA.map);
|
||||||
|
SA.centerLayer = L.layerGroup(centerMarkers).addTo(SA.map);
|
||||||
|
|
||||||
|
// 3. Render cluster edges (internal + external) as clickable
|
||||||
|
const allEdges = [...(data.internal_edges || []), ...(data.external_edges || [])];
|
||||||
|
if (allEdges.length && SA.map) {
|
||||||
|
const ipCoord = {};
|
||||||
|
(SA.nodes || []).forEach(n => {
|
||||||
|
if (n.lat != null && n.lon != null) ipCoord[n.ip] = [n.lat, n.lon];
|
||||||
|
});
|
||||||
|
|
||||||
|
const edgePolys = [];
|
||||||
|
allEdges.forEach((e, i) => {
|
||||||
|
const src = ipCoord[e.source];
|
||||||
|
const dst = ipCoord[e.target];
|
||||||
|
if (!src || !dst) return;
|
||||||
|
const isInternal = clusterIps.has(e.source) && clusterIps.has(e.target);
|
||||||
|
const w = Math.max(1.5, Math.min((e.comm_count || 1) / 5 + 1, 4));
|
||||||
|
const edgeColor = isInternal ? '#e63946' : '#f77f00';
|
||||||
|
|
||||||
|
const hitPoly = L.polyline([src, dst], {
|
||||||
|
color: edgeColor, weight: 10, opacity: 0,
|
||||||
|
interactive: true, bubblingMouseEvents: false,
|
||||||
|
});
|
||||||
|
const visPoly = L.polyline([src, dst], {
|
||||||
|
color: edgeColor, weight: w, opacity: 0.75,
|
||||||
|
interactive: false, bubblingMouseEvents: true,
|
||||||
|
});
|
||||||
|
const pair = L.layerGroup([hitPoly, visPoly]);
|
||||||
|
hitPoly.bindTooltip(
|
||||||
|
`${escapeHtml(e.source)} → ${escapeHtml(e.target)}<br>` +
|
||||||
|
`${e.comm_count} 次 · ${(e.total_bytes/1024).toFixed(1)} KB` +
|
||||||
|
`<br><span style="color:#aaa;">🖱 点击查看详情</span>`,
|
||||||
|
{sticky: true, direction: 'top'}
|
||||||
|
);
|
||||||
|
hitPoly.on('click', function(ev) {
|
||||||
|
L.DomEvent.stopPropagation(ev);
|
||||||
|
loadEdgeDetail(e);
|
||||||
|
});
|
||||||
|
hitPoly.on('mouseover', function() {
|
||||||
|
visPoly.setStyle({ color: '#e60000', weight: Math.min(w + 3, 8), opacity: 1 });
|
||||||
|
});
|
||||||
|
hitPoly.on('mouseout', function() {
|
||||||
|
visPoly.setStyle({ color: edgeColor, weight: w, opacity: 0.75 });
|
||||||
|
});
|
||||||
|
edgePolys.push(pair);
|
||||||
|
});
|
||||||
|
|
||||||
|
_clusterFocusLayer = L.layerGroup(edgePolys).addTo(SA.map);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Store focus state
|
||||||
|
SA._clusterFocus = data;
|
||||||
|
// Hide general edge layer if visible
|
||||||
|
if (_edgeLayer && SA.map) { SA.map.removeLayer(_edgeLayer); }
|
||||||
|
}
|
||||||
|
|
||||||
|
function clearClusterFocus() {
|
||||||
|
if (_clusterFocusLayer && SA.map) { SA.map.removeLayer(_clusterFocusLayer); _clusterFocusLayer = null; }
|
||||||
|
if (_clusterNodeLayer && SA.map) { SA.map.removeLayer(_clusterNodeLayer); _clusterNodeLayer = null; }
|
||||||
|
SA._clusterFocus = null;
|
||||||
|
|
||||||
|
// Restore normal markers
|
||||||
|
if (SA.map && SA.nodes && SA.nodes.length) {
|
||||||
|
renderMapData();
|
||||||
|
}
|
||||||
|
// Restore general edge layer
|
||||||
|
if (_edgeLayer) {
|
||||||
|
_edgeLayer.addTo(SA.map);
|
||||||
|
} else if (document.getElementById('saShowEdges')?.checked) {
|
||||||
|
renderEdgesOnMap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function showClusterDetailPanelV2(data) {
|
||||||
const panel = document.getElementById('saDetailPanel');
|
const panel = document.getElementById('saDetailPanel');
|
||||||
const body = document.getElementById('saDetailBody');
|
const body = document.getElementById('saDetailBody');
|
||||||
if (!panel || !body) return;
|
if (!panel || !body) return;
|
||||||
setText('saDetailTitle', `集群 ${data.label} 详情`);
|
setText('saDetailTitle', `社区 ${data.cluster_label} 详情`);
|
||||||
|
|
||||||
const ci = data.cluster_info || {};
|
const ci = data.cluster_info || {};
|
||||||
const geoBounds = data.geo_bounds;
|
const bounds = data.geo_bounds;
|
||||||
let geoHtml = '';
|
let geoHtml = '';
|
||||||
if (geoBounds) {
|
if (bounds) {
|
||||||
geoHtml = `<div class="section-title">📍 地理范围</div>
|
geoHtml = `<div class="section-title">📍 地理范围</div>
|
||||||
<div style="font-size:0.8rem;">lat: ${geoBounds.lat_min} ~ ${geoBounds.lat_max}, lon: ${geoBounds.lon_min} ~ ${geoBounds.lon_max}</div>`;
|
<div style="font-size:0.78rem;">lat: ${bounds.lat_min} ~ ${bounds.lat_max}<br>lon: ${bounds.lon_min} ~ ${bounds.lon_max}</div>`;
|
||||||
}
|
}
|
||||||
|
|
||||||
const distToHtml = (items, title) => {
|
const distToHtml = (items, title) => {
|
||||||
if (!items || !items.length) return '';
|
if (!items || !items.length) return '';
|
||||||
|
const maxC = Math.max(...items.map(i => i.count));
|
||||||
return `<div class="section-title">${title}</div>
|
return `<div class="section-title">${title}</div>
|
||||||
<div>${items.slice(0,5).map(i => `<span class="sa-tag">${escapeHtml(i.value)} (${i.count})</span>`).join(' ')}</div>`;
|
<div>${items.slice(0,5).map(i =>
|
||||||
|
`<span class="sa-tag" style="font-size:${0.75 + 0.25*i.count/maxC}rem;">${escapeHtml(i.value)} (${i.count})</span>`
|
||||||
|
).join(' ')}</div>`;
|
||||||
};
|
};
|
||||||
|
|
||||||
// IP nodes list (clickable)
|
// Internal edges list (clickable)
|
||||||
const ipAttrs = data.ip_attributes || [];
|
const intEdges = data.internal_edges || [];
|
||||||
let ipHtml = '';
|
let intEdgeHtml = '';
|
||||||
if (ipAttrs.length) {
|
if (intEdges.length) {
|
||||||
ipHtml = `<div class="section-title">🔗 节点 (${ipAttrs.length}/${data.unique_ips_count || ipAttrs.length})</div>
|
intEdgeHtml = `<div class="section-title">🔴 内部边 (${intEdges.length})</div>
|
||||||
<div class="ip-list">`;
|
<div style="max-height:150px;overflow-y:auto;font-size:0.75rem;">`;
|
||||||
ipAttrs.slice(0, 50).forEach(n => {
|
intEdges.slice(0, 20).forEach((e, i) => {
|
||||||
ipHtml += `<span class="sa-tag" style="cursor:pointer;color:#4361ee;margin:0.15rem;"
|
intEdgeHtml += `<div style="padding:0.15rem 0.3rem;cursor:pointer;border-radius:3px;"
|
||||||
|
onmouseover="this.style.background='#fef0f0'" onmouseout="this.style.background=''"
|
||||||
|
onclick="loadEdgeDetail(SA._clusterFocus.internal_edges[${i}])">
|
||||||
|
${escapeHtml(e.source)} → ${escapeHtml(e.target)} · ${e.comm_count}次 · ${(e.total_bytes/1024).toFixed(1)}KB</div>`;
|
||||||
|
});
|
||||||
|
if (intEdges.length > 20) intEdgeHtml += `<div style="color:#999;">... 还有 ${intEdges.length-20} 条</div>`;
|
||||||
|
intEdgeHtml += '</div>';
|
||||||
|
}
|
||||||
|
|
||||||
|
// External edges list (clickable)
|
||||||
|
const extEdges = data.external_edges || [];
|
||||||
|
let extEdgeHtml = '';
|
||||||
|
if (extEdges.length) {
|
||||||
|
extEdgeHtml = `<div class="section-title">🟠 外部关联边 (${extEdges.length})</div>
|
||||||
|
<div style="max-height:150px;overflow-y:auto;font-size:0.75rem;">`;
|
||||||
|
extEdges.slice(0, 20).forEach((e, i) => {
|
||||||
|
extEdgeHtml += `<div style="padding:0.15rem 0.3rem;cursor:pointer;border-radius:3px;"
|
||||||
|
onmouseover="this.style.background='#fff8f0'" onmouseout="this.style.background=''"
|
||||||
|
onclick="loadEdgeDetail(SA._clusterFocus.external_edges[${i}])">
|
||||||
|
${escapeHtml(e.source)} → ${escapeHtml(e.target)} · ${e.comm_count}次 · ${(e.total_bytes/1024).toFixed(1)}KB</div>`;
|
||||||
|
});
|
||||||
|
if (extEdges.length > 20) extEdgeHtml += `<div style="color:#999;">... 还有 ${extEdges.length-20} 条</div>`;
|
||||||
|
extEdgeHtml += '</div>';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Node IPs (clickable)
|
||||||
|
const clusterNodes = data.nodes || [];
|
||||||
|
let ipHtml = `<div class="section-title">🔗 节点 (${clusterNodes.length})</div><div class="ip-list">`;
|
||||||
|
clusterNodes.slice(0, 50).forEach(n => {
|
||||||
|
ipHtml += `<span class="sa-tag" style="cursor:pointer;color:#4361ee;margin:0.1rem;"
|
||||||
|
onclick="loadNodeDetail('${escapeHtml(n.ip)}')">${escapeHtml(n.ip)}</span>`;
|
||||||
|
});
|
||||||
|
if (clusterNodes.length > 50) ipHtml += `<div style="font-size:0.75rem;color:#999;">... 还有 ${clusterNodes.length-50} 个</div>`;
|
||||||
|
ipHtml += '</div>';
|
||||||
|
|
||||||
|
// Adjacent nodes
|
||||||
|
const adjNodes = data.adjacent_nodes || [];
|
||||||
|
let adjHtml = '';
|
||||||
|
if (adjNodes.length) {
|
||||||
|
adjHtml = `<div class="section-title">🌐 邻接节点 (${adjNodes.length})</div><div class="ip-list">`;
|
||||||
|
adjNodes.slice(0, 30).forEach(n => {
|
||||||
|
adjHtml += `<span class="sa-tag" style="cursor:pointer;color:#e76f51;margin:0.1rem;"
|
||||||
onclick="loadNodeDetail('${escapeHtml(n.ip)}')">${escapeHtml(n.ip)}</span>`;
|
onclick="loadNodeDetail('${escapeHtml(n.ip)}')">${escapeHtml(n.ip)}</span>`;
|
||||||
});
|
});
|
||||||
if ((data.unique_ips_count || 0) > 50) {
|
if (adjNodes.length > 30) adjHtml += `<div style="font-size:0.75rem;color:#999;">... 还有 ${adjNodes.length-30} 个</div>`;
|
||||||
ipHtml += `<div style="font-size:0.75rem;color:#999;">... 还有 ${data.unique_ips_count - 50} 个IP</div>`;
|
adjHtml += '</div>';
|
||||||
}
|
|
||||||
ipHtml += '</div>';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
body.innerHTML =
|
body.innerHTML =
|
||||||
`<div class="stat-row">
|
`<div style="margin-bottom:0.5rem;">
|
||||||
<div class="stat-card"><div class="num">${data.size || 0}</div><div class="lbl">节点数</div></div>
|
<button class="btn" onclick="clearClusterFocus()" style="font-size:0.75rem;padding:0.2rem 0.5rem;">🗺 恢复全局视图</button>
|
||||||
|
</div>
|
||||||
|
<div class="stat-row">
|
||||||
|
<div class="stat-card"><div class="num">${data.n_nodes || 0}</div><div class="lbl">节点数</div></div>
|
||||||
<div class="stat-card"><div class="num">${data.n_internal_edges || 0}</div><div class="lbl">内部边</div></div>
|
<div class="stat-card"><div class="num">${data.n_internal_edges || 0}</div><div class="lbl">内部边</div></div>
|
||||||
<div class="stat-card"><div class="num">${data.n_related_edges || 0}</div><div class="lbl">关联边</div></div>
|
<div class="stat-card"><div class="num">${data.n_external_edges || 0}</div><div class="lbl">外部边</div></div>
|
||||||
</div>
|
</div>
|
||||||
<div class="stat-row">
|
<div class="stat-row">
|
||||||
<div class="stat-card"><div class="num">${(data.total_internal_bytes/1024).toFixed(1)} KB</div><div class="lbl">内部流量</div></div>
|
<div class="stat-card"><div class="num">${(data.total_internal_bytes/1024).toFixed(1)} KB</div><div class="lbl">内部流量</div></div>
|
||||||
<div class="stat-card"><div class="num">${(data.total_related_bytes/1024).toFixed(1)} KB</div><div class="lbl">关联流量</div></div>
|
<div class="stat-card"><div class="num">${(data.total_external_bytes/1024).toFixed(1)} KB</div><div class="lbl">关联流量</div></div>
|
||||||
</div>
|
</div>
|
||||||
${geoHtml}
|
${geoHtml}
|
||||||
${distToHtml(data.top_isps, '🏢 ISP 分布')}
|
${distToHtml(data.top_isps, '🏢 ISP 分布')}
|
||||||
${distToHtml(data.top_cities, '🏙 城市分布')}
|
${distToHtml(data.top_cities, '🏙 城市分布')}
|
||||||
${distToHtml(data.top_countries, '🌍 国家分布')}
|
${distToHtml(data.top_countries, '🌍 国家分布')}
|
||||||
${ipHtml}`;
|
${intEdgeHtml}
|
||||||
|
${extEdgeHtml}
|
||||||
|
${ipHtml}
|
||||||
|
${adjHtml}`;
|
||||||
|
|
||||||
panel.classList.add('open');
|
panel.classList.add('open');
|
||||||
}
|
}
|
||||||
@@ -1911,6 +2441,10 @@ function showClusterDetailPanel(data) {
|
|||||||
/* ── Close detail ───────────────────────────────────────── */
|
/* ── Close detail ───────────────────────────────────────── */
|
||||||
function closeDetail() {
|
function closeDetail() {
|
||||||
document.getElementById('saDetailPanel').classList.remove('open');
|
document.getElementById('saDetailPanel').classList.remove('open');
|
||||||
|
// Destroy edge time-series chart
|
||||||
|
if (SA._edgeTimeChart) { SA._edgeTimeChart.destroy(); SA._edgeTimeChart = null; }
|
||||||
|
// Clear cluster focus if active
|
||||||
|
if (SA._clusterFocus) { clearClusterFocus(); }
|
||||||
// Remove highlighted node edges, restore saved general edge layer
|
// Remove highlighted node edges, restore saved general edge layer
|
||||||
if (_edgeLayer && SA._highlightedNodeEdges && SA._highlightedNodeEdges.length) {
|
if (_edgeLayer && SA._highlightedNodeEdges && SA._highlightedNodeEdges.length) {
|
||||||
SA.map.removeLayer(_edgeLayer);
|
SA.map.removeLayer(_edgeLayer);
|
||||||
@@ -1947,6 +2481,8 @@ function resetAll() {
|
|||||||
if (SA.map) { SA.map.remove(); SA.map = null; }
|
if (SA.map) { SA.map.remove(); SA.map = null; }
|
||||||
if (SA.barChart) { SA.barChart.destroy(); SA.barChart = null; }
|
if (SA.barChart) { SA.barChart.destroy(); SA.barChart = null; }
|
||||||
if (SA.pieChart) { SA.pieChart.destroy(); SA.pieChart = null; }
|
if (SA.pieChart) { SA.pieChart.destroy(); SA.pieChart = null; }
|
||||||
|
if (SA._edgeTimeChart) { SA._edgeTimeChart.destroy(); SA._edgeTimeChart = null; }
|
||||||
|
clearClusterFocus();
|
||||||
updateDebug({status: '等待上传'});
|
updateDebug({status: '等待上传'});
|
||||||
goStep(1);
|
goStep(1);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,4 +14,6 @@ urlpatterns = [
|
|||||||
path('cluster/<int:label>/', views.cluster_detail, name='cluster_detail'),
|
path('cluster/<int:label>/', views.cluster_detail, name='cluster_detail'),
|
||||||
path('node/', views.node_detail, name='node_detail'),
|
path('node/', views.node_detail, name='node_detail'),
|
||||||
path('edges/', views.edge_relationships, name='edges'),
|
path('edges/', views.edge_relationships, name='edges'),
|
||||||
|
path('edge-time-series/', views.edge_time_series, name='edge_time_series'),
|
||||||
|
path('cluster-nodes-edges/', views.cluster_nodes_edges, name='cluster_nodes_edges'),
|
||||||
]
|
]
|
||||||
|
|||||||
+511
-50
@@ -371,6 +371,7 @@ def upload_csv(request):
|
|||||||
tls_ver_col = _get_std('tls_ver')
|
tls_ver_col = _get_std('tls_ver')
|
||||||
tls_cph_col = _get_std('tls_cph')
|
tls_cph_col = _get_std('tls_cph')
|
||||||
bytes_col = _get_std('bytes')
|
bytes_col = _get_std('bytes')
|
||||||
|
packets_col = _get_std('packets')
|
||||||
dur_col = _get_std('duration')
|
dur_col = _get_std('duration')
|
||||||
time_col = _get_std('time')
|
time_col = _get_std('time')
|
||||||
ispn_col = _get_std('ispn')
|
ispn_col = _get_std('ispn')
|
||||||
@@ -438,6 +439,7 @@ def upload_csv(request):
|
|||||||
'tls_ver_col': tls_ver_col,
|
'tls_ver_col': tls_ver_col,
|
||||||
'tls_cph_col': tls_cph_col,
|
'tls_cph_col': tls_cph_col,
|
||||||
'bytes_col': bytes_col,
|
'bytes_col': bytes_col,
|
||||||
|
'packets_col': packets_col,
|
||||||
'dur_col': dur_col,
|
'dur_col': dur_col,
|
||||||
'time_col': time_col,
|
'time_col': time_col,
|
||||||
'ispn_col': ispn_col,
|
'ispn_col': ispn_col,
|
||||||
@@ -765,6 +767,13 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
orgn_col = meta.get('orgn_col')
|
orgn_col = meta.get('orgn_col')
|
||||||
ispn_col = meta.get('ispn_col')
|
ispn_col = meta.get('ispn_col')
|
||||||
city_col = meta.get('city_col')
|
city_col = meta.get('city_col')
|
||||||
|
dport_col = meta.get('dport_col')
|
||||||
|
dur_col = meta.get('dur_col')
|
||||||
|
cnrs_col = meta.get('cnrs_col')
|
||||||
|
tls_ver_col = meta.get('tls_ver_col')
|
||||||
|
tls_cph_col = meta.get('tls_cph_col')
|
||||||
|
cnam_col = meta.get('cnam_col')
|
||||||
|
packets_col = meta.get('packets_col')
|
||||||
geo_cache = entry.get('geo_cache', {})
|
geo_cache = entry.get('geo_cache', {})
|
||||||
|
|
||||||
if not ips_col or not ipd_col:
|
if not ips_col or not ipd_col:
|
||||||
@@ -960,61 +969,128 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
if ip and ip not in ('None', '', '+', '-'):
|
if ip and ip not in ('None', '', '+', '-'):
|
||||||
server_ips.add(ip)
|
server_ips.add(ip)
|
||||||
|
|
||||||
# ── 收集 per-IP 元数据 (host, text, org, peers) ────────────────────
|
# ── 收集 per-IP 元数据 (host, text, org, peers, TLS, traffic) ────
|
||||||
# 从 DataFrame 中为每个 IP 收集关联的 host/snam、text、org
|
ip_hosts: dict[str, str] = {} # ip → 最常见的 host
|
||||||
ip_hosts: dict[str, str] = {} # ip → 最常见的 host
|
ip_texts: dict[str, list] = {} # ip → 前几条 payload 样本
|
||||||
ip_texts: dict[str, list] = {} # ip → 前几条 payload 样本
|
ip_orgs: dict[str, str] = {} # ip → 最常见的 org
|
||||||
ip_orgs: dict[str, str] = {} # ip → 最常见的 org
|
ip_peers: dict[str, list] = {} # ip → 对端 IP 列表
|
||||||
ip_peers: dict[str, list] = {} # ip → 对端 IP 列表
|
ip_snis: dict[str, list] = {} # ip → SNI/域名列表
|
||||||
|
ip_tls_vers: dict[str, list] = {} # ip → TLS 版本列表
|
||||||
|
ip_ciphers: dict[str, list] = {} # ip → 加密套件列表
|
||||||
|
ip_dports: dict[str, list] = {} # ip → 目的端口列表
|
||||||
|
ip_categories: dict[str, list] = {} # ip → 分类标签列表
|
||||||
|
ip_total_bytes: dict[str, float] = {} # ip → 总字节数
|
||||||
|
ip_total_pkts: dict[str, int] = {} # ip → 总包数
|
||||||
|
ip_durations: dict[str, list] = {} # ip → 持续时间列表
|
||||||
|
ip_conn_ok: dict[str, int] = {} # ip → 成功连接次数
|
||||||
|
ip_conn_total: dict[str, int] = {} # ip → 总连接次数
|
||||||
|
ip_first_ts: dict[str, str] = {} # ip → 首次出现时间
|
||||||
|
ip_last_ts: dict[str, str] = {} # ip → 最后出现时间
|
||||||
|
|
||||||
# 使用 host_col 或 snam_col 作为 host 来源
|
scan_limit = min(len(df), 50000)
|
||||||
effective_host_col = host_col or snam_col
|
# Pre-extract column lists for faster access
|
||||||
if effective_host_col and effective_host_col in df.columns:
|
_host_col = host_col or snam_col
|
||||||
# 为每个 IP 收集关联的 host（作为源 IP 时的 host）
|
_has_host = _host_col and _host_col in df.columns
|
||||||
for i in range(min(len(df), 50000)): # 限制扫描行数
|
_has_text = text_col and text_col in df.columns
|
||||||
sip = str(src_ips[i]) if i < len(src_ips) and src_ips[i] not in (None, '', 'None') else None
|
_has_orgn = orgn_col and orgn_col in df.columns
|
||||||
if sip and sip in unique_ips_set:
|
_has_snam = snam_col and snam_col in df.columns
|
||||||
|
_has_tls_ver = tls_ver_col and tls_ver_col in df.columns
|
||||||
|
_has_tls_cph = tls_cph_col and tls_cph_col in df.columns
|
||||||
|
_has_dport = dport_col and dport_col in df.columns
|
||||||
|
_has_cnam = cnam_col and cnam_col in df.columns
|
||||||
|
_has_bytes = bytes_col and bytes_col in df.columns
|
||||||
|
_has_pkts = packets_col and packets_col in df.columns if packets_col else False
|
||||||
|
_has_dur = dur_col and dur_col in df.columns
|
||||||
|
_has_cnrs = cnrs_col and cnrs_col in df.columns
|
||||||
|
_has_time = time_col and time_col in df.columns
|
||||||
|
|
||||||
|
for i in range(scan_limit):
|
||||||
|
sip = str(src_ips[i]) if i < len(src_ips) and src_ips[i] not in (None, '', 'None') else None
|
||||||
|
if not sip or sip not in unique_ips_set:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
if _has_host:
|
||||||
|
hval = str(df[_host_col][i]) if df[_host_col][i] is not None else None
|
||||||
|
if hval and hval not in ('None', '', 'nan'):
|
||||||
|
ip_hosts.setdefault(sip, []).append(hval)
|
||||||
|
if _has_text:
|
||||||
|
tval = str(df[text_col][i]) if df[text_col][i] is not None else None
|
||||||
|
if tval and tval not in ('None', '', 'nan'):
|
||||||
|
lst = ip_texts.setdefault(sip, [])
|
||||||
|
if len(lst) < 3:
|
||||||
|
lst.append(tval[:200])
|
||||||
|
if _has_orgn:
|
||||||
|
oval = str(df[orgn_col][i]) if df[orgn_col][i] is not None else None
|
||||||
|
if oval and oval not in ('None', '', 'nan'):
|
||||||
|
ip_orgs.setdefault(sip, []).append(oval)
|
||||||
|
if _has_snam:
|
||||||
|
sval = str(df[snam_col][i]) if df[snam_col][i] is not None else None
|
||||||
|
if sval and sval not in ('None', '', 'nan', '+', '-'):
|
||||||
|
ip_snis.setdefault(sip, []).append(sval)
|
||||||
|
if _has_tls_ver:
|
||||||
|
tv = str(df[tls_ver_col][i]) if df[tls_ver_col][i] is not None else None
|
||||||
|
if tv and tv not in ('None', '', 'nan'):
|
||||||
|
ip_tls_vers.setdefault(sip, []).append(tv)
|
||||||
|
if _has_tls_cph:
|
||||||
|
tc = str(df[tls_cph_col][i]) if df[tls_cph_col][i] is not None else None
|
||||||
|
if tc and tc not in ('None', '', 'nan'):
|
||||||
|
ip_ciphers.setdefault(sip, []).append(tc[:50])
|
||||||
|
if _has_dport:
|
||||||
|
dp = str(df[dport_col][i]) if df[dport_col][i] is not None else None
|
||||||
|
if dp and dp not in ('None', '', 'nan'):
|
||||||
|
ip_dports.setdefault(sip, []).append(dp)
|
||||||
|
if _has_cnam:
|
||||||
|
cv = str(df[cnam_col][i]) if df[cnam_col][i] is not None else None
|
||||||
|
if cv and cv not in ('None', '', 'nan', '+', '-'):
|
||||||
|
ip_categories.setdefault(sip, []).append(cv)
|
||||||
|
if _has_bytes:
|
||||||
try:
|
try:
|
||||||
hval = str(df[effective_host_col][i]) if df[effective_host_col][i] is not None else None
|
bv = float(df[bytes_col][i] or 0)
|
||||||
if hval and hval not in ('None', '', 'nan'):
|
ip_total_bytes[sip] = ip_total_bytes.get(sip, 0.0) + bv
|
||||||
ip_hosts.setdefault(sip, []).append(hval)
|
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
if _has_pkts:
|
||||||
|
try:
|
||||||
|
pv = int(df[packets_col][i] or 0)
|
||||||
|
ip_total_pkts[sip] = ip_total_pkts.get(sip, 0) + pv
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if _has_dur:
|
||||||
|
try:
|
||||||
|
dv = float(df[dur_col][i] or 0)
|
||||||
|
if dv > 0:
|
||||||
|
ip_durations.setdefault(sip, []).append(dv)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if _has_cnrs:
|
||||||
|
cv = str(df[cnrs_col][i]) if df[cnrs_col][i] is not None else ''
|
||||||
|
ip_conn_total[sip] = ip_conn_total.get(sip, 0) + 1
|
||||||
|
if cv in ('+', '1', 'true', 'True', 'success'):
|
||||||
|
ip_conn_ok[sip] = ip_conn_ok.get(sip, 0) + 1
|
||||||
|
if _has_time:
|
||||||
|
tv = str(df[time_col][i]) if df[time_col][i] is not None else ''
|
||||||
|
if tv and tv not in ('None', '', 'nan'):
|
||||||
|
if sip not in ip_first_ts or tv < ip_first_ts[sip]:
|
||||||
|
ip_first_ts[sip] = tv
|
||||||
|
if sip not in ip_last_ts or tv > ip_last_ts[sip]:
|
||||||
|
ip_last_ts[sip] = tv
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# 取每 IP 最常见的值
|
||||||
|
def _most_common(lst: list, top_n: int = 8) -> list:
|
||||||
|
if not lst: return []
|
||||||
|
cnt = Counter(lst)
|
||||||
|
return [{'value': k, 'count': v} for k, v in cnt.most_common(top_n)]
|
||||||
|
|
||||||
|
def _top1(lst: list) -> str:
|
||||||
|
if not lst: return ''
|
||||||
|
return Counter(lst).most_common(1)[0][0]
|
||||||
|
|
||||||
# 取每 IP 最常见的 host
|
|
||||||
for ip, hosts in ip_hosts.items():
|
for ip, hosts in ip_hosts.items():
|
||||||
if hosts:
|
ip_hosts[ip] = _top1(hosts) if isinstance(hosts, list) else hosts
|
||||||
from collections import Counter as _Counter
|
|
||||||
ip_hosts[ip] = _Counter(hosts).most_common(1)[0][0]
|
|
||||||
|
|
||||||
if text_col and text_col in df.columns:
|
|
||||||
for i in range(min(len(df), 50000)):
|
|
||||||
sip = str(src_ips[i]) if i < len(src_ips) and src_ips[i] not in (None, '', 'None') else None
|
|
||||||
if sip and sip in unique_ips_set:
|
|
||||||
try:
|
|
||||||
tval = str(df[text_col][i]) if df[text_col][i] is not None else None
|
|
||||||
if tval and tval not in ('None', '', 'nan'):
|
|
||||||
lst = ip_texts.setdefault(sip, [])
|
|
||||||
if len(lst) < 3:
|
|
||||||
lst.append(tval[:200])
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
if orgn_col and orgn_col in df.columns:
|
|
||||||
for i in range(min(len(df), 50000)):
|
|
||||||
sip = str(src_ips[i]) if i < len(src_ips) and src_ips[i] not in (None, '', 'None') else None
|
|
||||||
if sip and sip in unique_ips_set:
|
|
||||||
try:
|
|
||||||
oval = str(df[orgn_col][i]) if df[orgn_col][i] is not None else None
|
|
||||||
if oval and oval not in ('None', '', 'nan'):
|
|
||||||
ip_orgs.setdefault(sip, []).append(oval)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
for ip, orgs in ip_orgs.items():
|
for ip, orgs in ip_orgs.items():
|
||||||
if orgs:
|
ip_orgs[ip] = _top1(orgs) if isinstance(orgs, list) else orgs
|
||||||
from collections import Counter as _Counter2
|
|
||||||
ip_orgs[ip] = _Counter2(orgs).most_common(1)[0][0]
|
|
||||||
|
|
||||||
# ── 5. 构建节点列表 ──────────────────────────────────────────────
|
# ── 5. 构建节点列表 ──────────────────────────────────────────────
|
||||||
# Build peer IP lists from raw src/dst pairs
|
# Build peer IP lists from raw src/dst pairs
|
||||||
@@ -1030,6 +1106,9 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
info = node_geo.get(ip, {})
|
info = node_geo.get(ip, {})
|
||||||
cid = cluster_labels.get(ip, -1)
|
cid = cluster_labels.get(ip, -1)
|
||||||
peer_set = ip_peers.get(ip, set())
|
peer_set = ip_peers.get(ip, set())
|
||||||
|
dur_list = ip_durations.get(ip, [])
|
||||||
|
conn_total = ip_conn_total.get(ip, 0)
|
||||||
|
conn_ok = ip_conn_ok.get(ip, 0)
|
||||||
nodes.append({
|
nodes.append({
|
||||||
'ip': ip,
|
'ip': ip,
|
||||||
'cluster_id': int(cid),
|
'cluster_id': int(cid),
|
||||||
@@ -1046,11 +1125,25 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
'texts': ip_texts.get(ip, [])[:3],
|
'texts': ip_texts.get(ip, [])[:3],
|
||||||
'peer_ips': sorted(peer_set)[:20],
|
'peer_ips': sorted(peer_set)[:20],
|
||||||
'peer_count': len(peer_set),
|
'peer_count': len(peer_set),
|
||||||
|
# Enriched attributes
|
||||||
|
'top_snis': _most_common(ip_snis.get(ip, []), 5),
|
||||||
|
'top_tls_vers': _most_common(ip_tls_vers.get(ip, []), 5),
|
||||||
|
'top_ciphers': _most_common(ip_ciphers.get(ip, []), 5),
|
||||||
|
'top_ports': _most_common(ip_dports.get(ip, []), 5),
|
||||||
|
'top_categories': _most_common(ip_categories.get(ip, []), 5),
|
||||||
|
'total_bytes': round(ip_total_bytes.get(ip, 0.0), 2),
|
||||||
|
'total_packets': ip_total_pkts.get(ip, 0),
|
||||||
|
'avg_duration': round(sum(dur_list) / len(dur_list), 2) if dur_list else None,
|
||||||
|
'conn_success_rate': round(conn_ok / conn_total * 100, 1) if conn_total > 0 else None,
|
||||||
|
'conn_count': conn_total,
|
||||||
|
'first_seen': ip_first_ts.get(ip, ''),
|
||||||
|
'last_seen': ip_last_ts.get(ip, ''),
|
||||||
})
|
})
|
||||||
|
|
||||||
# ── 6. 构建边列表（IP 对聚合通信统计） ──────────────────────────
|
# ── 6. 构建边列表（IP 对聚合通信统计） ──────────────────────────
|
||||||
pair_data: dict[tuple[str, str], dict] = {} # (src, dst) → {count, bytes, times}
|
pair_data: dict[tuple[str, str], dict] = {}
|
||||||
|
|
||||||
|
# Pre-extract column lists
|
||||||
bytes_list = None
|
bytes_list = None
|
||||||
if bytes_col and bytes_col in df.columns:
|
if bytes_col and bytes_col in df.columns:
|
||||||
try:
|
try:
|
||||||
@@ -1065,6 +1158,55 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
pkts_list = None
|
||||||
|
if packets_col and packets_col in df.columns:
|
||||||
|
try:
|
||||||
|
pkts_list = df[packets_col].cast(pl.Float64).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
dur_list = None
|
||||||
|
if dur_col and dur_col in df.columns:
|
||||||
|
try:
|
||||||
|
dur_list = df[dur_col].cast(pl.Float64).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
dport_list = None
|
||||||
|
if dport_col and dport_col in df.columns:
|
||||||
|
try:
|
||||||
|
dport_list = df[dport_col].cast(pl.Utf8).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
tls_ver_list = None
|
||||||
|
if tls_ver_col and tls_ver_col in df.columns:
|
||||||
|
try:
|
||||||
|
tls_ver_list = df[tls_ver_col].cast(pl.Utf8).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
cph_list = None
|
||||||
|
if tls_cph_col and tls_cph_col in df.columns:
|
||||||
|
try:
|
||||||
|
cph_list = df[tls_cph_col].cast(pl.Utf8).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
snam_list = None
|
||||||
|
if snam_col and snam_col in df.columns:
|
||||||
|
try:
|
||||||
|
snam_list = df[snam_col].cast(pl.Utf8).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
cnam_list = None
|
||||||
|
if cnam_col and cnam_col in df.columns:
|
||||||
|
try:
|
||||||
|
cnam_list = df[cnam_col].cast(pl.Utf8).to_list()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
for i in range(len(df)):
|
for i in range(len(df)):
|
||||||
s = str(src_ips[i]) if src_ips[i] not in (None, '', 'None') else None
|
s = str(src_ips[i]) if src_ips[i] not in (None, '', 'None') else None
|
||||||
d = str(dst_ips[i]) if dst_ips[i] not in (None, '', 'None') else None
|
d = str(dst_ips[i]) if dst_ips[i] not in (None, '', 'None') else None
|
||||||
@@ -1072,7 +1214,11 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
continue
|
continue
|
||||||
key = (s, d)
|
key = (s, d)
|
||||||
if key not in pair_data:
|
if key not in pair_data:
|
||||||
pair_data[key] = {'count': 0, 'total_bytes': 0.0, 'times': []}
|
pair_data[key] = {
|
||||||
|
'count': 0, 'total_bytes': 0.0, 'total_packets': 0.0,
|
||||||
|
'times': [], 'durations': [], 'ports': [],
|
||||||
|
'tls_vers': [], 'ciphers': [], 'snis': [], 'categories': [],
|
||||||
|
}
|
||||||
pair_data[key]['count'] += 1
|
pair_data[key]['count'] += 1
|
||||||
if bytes_list and i < len(bytes_list):
|
if bytes_list and i < len(bytes_list):
|
||||||
try:
|
try:
|
||||||
@@ -1083,6 +1229,38 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
t = times_list[i]
|
t = times_list[i]
|
||||||
if t and t not in ('None', ''):
|
if t and t not in ('None', ''):
|
||||||
pair_data[key]['times'].append(t)
|
pair_data[key]['times'].append(t)
|
||||||
|
if pkts_list and i < len(pkts_list):
|
||||||
|
try:
|
||||||
|
pair_data[key]['total_packets'] += float(pkts_list[i] or 0)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
if dur_list and i < len(dur_list):
|
||||||
|
try:
|
||||||
|
dv = float(dur_list[i] or 0)
|
||||||
|
if dv > 0:
|
||||||
|
pair_data[key]['durations'].append(dv)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
if dport_list and i < len(dport_list):
|
||||||
|
pv = dport_list[i]
|
||||||
|
if pv and pv not in ('None', '', 'nan'):
|
||||||
|
pair_data[key]['ports'].append(pv)
|
||||||
|
if tls_ver_list and i < len(tls_ver_list):
|
||||||
|
tv = tls_ver_list[i]
|
||||||
|
if tv and tv not in ('None', '', 'nan'):
|
||||||
|
pair_data[key]['tls_vers'].append(tv)
|
||||||
|
if cph_list and i < len(cph_list):
|
||||||
|
cv = cph_list[i]
|
||||||
|
if cv and cv not in ('None', '', 'nan'):
|
||||||
|
pair_data[key]['ciphers'].append(cv[:50])
|
||||||
|
if snam_list and i < len(snam_list):
|
||||||
|
sv = snam_list[i]
|
||||||
|
if sv and sv not in ('None', '', 'nan', '+', '-'):
|
||||||
|
pair_data[key]['snis'].append(sv)
|
||||||
|
if cnam_list and i < len(cnam_list):
|
||||||
|
catv = cnam_list[i]
|
||||||
|
if catv and catv not in ('None', '', 'nan', '+', '-'):
|
||||||
|
pair_data[key]['categories'].append(catv)
|
||||||
|
|
||||||
# Build IP metadata lookup for edges
|
# Build IP metadata lookup for edges
|
||||||
ip_meta = {}
|
ip_meta = {}
|
||||||
@@ -1093,11 +1271,19 @@ def _cluster_network(entry: dict, df: pl.DataFrame, meta: dict,
|
|||||||
for (src, dst), data in pair_data.items():
|
for (src, dst), data in pair_data.items():
|
||||||
sm = ip_meta.get(src, {})
|
sm = ip_meta.get(src, {})
|
||||||
dm = ip_meta.get(dst, {})
|
dm = ip_meta.get(dst, {})
|
||||||
|
durs = data.get('durations', [])
|
||||||
edge = {
|
edge = {
|
||||||
'source': src,
|
'source': src,
|
||||||
'target': dst,
|
'target': dst,
|
||||||
'comm_count': data['count'],
|
'comm_count': data['count'],
|
||||||
'total_bytes': round(data['total_bytes'], 2),
|
'total_bytes': round(data['total_bytes'], 2),
|
||||||
|
'total_packets': round(data.get('total_packets', 0), 2),
|
||||||
|
'avg_duration': round(sum(durs) / len(durs), 2) if durs else None,
|
||||||
|
'top_ports': _most_common(data.get('ports', []), 5),
|
||||||
|
'top_tls_vers': _most_common(data.get('tls_vers', []), 5),
|
||||||
|
'top_ciphers': _most_common(data.get('ciphers', []), 5),
|
||||||
|
'top_snis': _most_common(data.get('snis', []), 5),
|
||||||
|
'top_categories': _most_common(data.get('categories', []), 5),
|
||||||
'src_host': sm.get('host', ''),
|
'src_host': sm.get('host', ''),
|
||||||
'dst_host': dm.get('host', ''),
|
'dst_host': dm.get('host', ''),
|
||||||
'src_org': sm.get('org', ''),
|
'src_org': sm.get('org', ''),
|
||||||
@@ -1530,6 +1716,281 @@ def _edge_relationships_legacy(entry: dict, body: dict) -> JsonResponse:
|
|||||||
return JsonResponse({'error': f'边关系计算失败: {e}'}, status=400)
|
return JsonResponse({'error': f'边关系计算失败: {e}'}, status=400)
|
||||||
|
|
||||||
|
|
||||||
|
@csrf_exempt
|
||||||
|
@require_POST
|
||||||
|
def edge_time_series(request):
|
||||||
|
"""返回指定边的时间序列数据（按时间桶聚合），用于绘制流量-时间图。
|
||||||
|
|
||||||
|
POST body: session_id, source, target, bucket (可选: 'auto'|'hour'|'minute'|'day')
|
||||||
|
返回: {time_series: [{t: "yyyy-mm-dd HH:MM (北京)", bytes: N, count: N}, ...], span_hours: N}
|
||||||
|
"""
|
||||||
|
body = json.loads(request.body)
|
||||||
|
session_id = body.get('session_id')
|
||||||
|
source = body.get('source', '')
|
||||||
|
target = body.get('target', '')
|
||||||
|
bucket = body.get('bucket', 'auto')
|
||||||
|
|
||||||
|
if not session_id or not source or not target:
|
||||||
|
return JsonResponse({'error': '缺少 session_id / source / target'}, status=400)
|
||||||
|
|
||||||
|
entry = _cached(session_id)
|
||||||
|
if entry is None:
|
||||||
|
return JsonResponse({'error': '会话已过期'}, status=400)
|
||||||
|
|
||||||
|
df = entry.get('filtered_df') if entry.get('filtered_df') is not None else entry.get('df')
|
||||||
|
if df is None:
|
||||||
|
return JsonResponse({'error': '数据未就绪'}, status=400)
|
||||||
|
|
||||||
|
meta = entry.get('meta', {})
|
||||||
|
ips_col = meta.get('ips_col')
|
||||||
|
ipd_col = meta.get('ipd_col')
|
||||||
|
time_col = meta.get('time_col')
|
||||||
|
bytes_col = meta.get('bytes_col')
|
||||||
|
|
||||||
|
if not ips_col or not ipd_col:
|
||||||
|
return JsonResponse({'error': '缺少源/目的IP列'}, status=400)
|
||||||
|
if not time_col or time_col not in df.columns:
|
||||||
|
return JsonResponse({'error': '缺少时间列，无法绘制时间序列'}, status=400)
|
||||||
|
|
||||||
|
# ── 1. 筛选该边的所有行 ──
|
||||||
|
try:
|
||||||
|
mask = (
|
||||||
|
(df[ips_col].cast(pl.Utf8) == source) &
|
||||||
|
(df[ipd_col].cast(pl.Utf8) == target)
|
||||||
|
)
|
||||||
|
edge_rows = df.filter(mask)
|
||||||
|
except Exception as e:
|
||||||
|
return JsonResponse({'error': f'数据筛选失败: {e}'}, status=400)
|
||||||
|
|
||||||
|
n_rows = len(edge_rows)
|
||||||
|
if n_rows == 0:
|
||||||
|
return JsonResponse({'error': '未找到该边的原始记录'}, status=400)
|
||||||
|
|
||||||
|
# ── 2. 提取时间列并转为 datetime ──
|
||||||
|
try:
|
||||||
|
times_series = edge_rows[time_col].cast(pl.Utf8)
|
||||||
|
except Exception:
|
||||||
|
return JsonResponse({'error': '时间列读取失败'}, status=400)
|
||||||
|
|
||||||
|
# 提取 bytes（如果有的话）
|
||||||
|
has_bytes = bytes_col and bytes_col in df.columns
|
||||||
|
bytes_series = None
|
||||||
|
if has_bytes:
|
||||||
|
try:
|
||||||
|
bytes_series = edge_rows[bytes_col].cast(pl.Float64)
|
||||||
|
except Exception:
|
||||||
|
has_bytes = False
|
||||||
|
|
||||||
|
# ── 3. 解析时间戳为 datetime ──
|
||||||
|
raw_times = times_series.to_list()
|
||||||
|
raw_bytes = bytes_series.to_list() if has_bytes else []
|
||||||
|
parsed_times = []
|
||||||
|
for row_idx, t_str in enumerate(raw_times):
|
||||||
|
try:
|
||||||
|
t_clean = str(t_str).strip() if t_str else ''
|
||||||
|
if not t_clean or t_clean in ('None', '', 'nan'):
|
||||||
|
continue
|
||||||
|
# 尝试多种格式
|
||||||
|
parsed = None
|
||||||
|
for fmt in [
|
||||||
|
'%Y-%m-%dT%H:%M:%S', '%Y-%m-%d %H:%M:%S',
|
||||||
|
'%Y-%m-%dT%H:%M:%S.%f', '%Y-%m-%d %H:%M:%S.%f',
|
||||||
|
'%Y/%m/%d %H:%M:%S', '%Y/%m/%dT%H:%M:%S',
|
||||||
|
'%d/%m/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S',
|
||||||
|
]:
|
||||||
|
try:
|
||||||
|
parsed = time.strptime(t_clean[:26], fmt)
|
||||||
|
break
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if parsed is None:
|
||||||
|
# Fallback: try to extract Y-M-D H:M:S via regex
|
||||||
|
m = re.search(r'(\d{4})-(\d{2})-(\d{2})[T ](\d{2}):(\d{2}):(\d{2})', t_clean)
|
||||||
|
if m:
|
||||||
|
parsed = time.strptime(f"{m.group(1)}-{m.group(2)}-{m.group(3)} "
|
||||||
|
f"{m.group(4)}:{m.group(5)}:{m.group(6)}",
|
||||||
|
'%Y-%m-%d %H:%M:%S')
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
# 转为北京时间 (UTC+8)
|
||||||
|
import calendar
|
||||||
|
ts_utc = calendar.timegm(parsed)
|
||||||
|
ts_bj = ts_utc + 8 * 3600 # 北京时间 = UTC+8
|
||||||
|
parsed_times.append((ts_bj, row_idx))
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not parsed_times:
|
||||||
|
return JsonResponse({'error': '无法解析时间戳'}, status=400)
|
||||||
|
|
||||||
|
# ── 4. 确定时间桶粒度 ──
|
||||||
|
min_ts = min(t[0] for t in parsed_times)
|
||||||
|
max_ts = max(t[0] for t in parsed_times)
|
||||||
|
span_seconds = max_ts - min_ts
|
||||||
|
span_hours = span_seconds / 3600.0
|
||||||
|
|
||||||
|
if bucket == 'auto':
|
||||||
|
if span_hours <= 2:
|
||||||
|
bucket_seconds = 60 # 1 分钟桶
|
||||||
|
elif span_hours <= 24:
|
||||||
|
bucket_seconds = 300 # 5 分钟桶
|
||||||
|
elif span_hours <= 72:
|
||||||
|
bucket_seconds = 900 # 15 分钟桶
|
||||||
|
elif span_hours <= 168:
|
||||||
|
bucket_seconds = 3600 # 1 小时桶
|
||||||
|
else:
|
||||||
|
bucket_seconds = 14400 # 4 小时桶
|
||||||
|
elif bucket == 'minute':
|
||||||
|
bucket_seconds = 60
|
||||||
|
elif bucket == 'hour':
|
||||||
|
bucket_seconds = 3600
|
||||||
|
elif bucket == 'day':
|
||||||
|
bucket_seconds = 86400
|
||||||
|
else:
|
||||||
|
bucket_seconds = 300
|
||||||
|
|
||||||
|
# ── 5. 按时间桶聚合 ──
|
||||||
|
bucket_data: dict[int, dict] = {} # bucket_start_ts → {count, bytes}
|
||||||
|
for ts_bj, idx in parsed_times:
|
||||||
|
bucket_key = int(ts_bj // bucket_seconds) * bucket_seconds
|
||||||
|
if bucket_key not in bucket_data:
|
||||||
|
bucket_data[bucket_key] = {'count': 0, 'bytes': 0.0}
|
||||||
|
bucket_data[bucket_key]['count'] += 1
|
||||||
|
if has_bytes and idx < len(raw_bytes):
|
||||||
|
try:
|
||||||
|
bval = raw_bytes[idx]
|
||||||
|
if bval is not None:
|
||||||
|
bucket_data[bucket_key]['bytes'] += float(bval)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# ── 6. 构建输出 ──
|
||||||
|
def fmt_bj(ts: int, bucket_sec: int) -> str:
|
||||||
|
"""格式化北京时间字符串。"""
|
||||||
|
tm = time.gmtime(ts)
|
||||||
|
if bucket_sec >= 86400:
|
||||||
|
return time.strftime('%m-%d', tm)
|
||||||
|
elif bucket_sec >= 3600:
|
||||||
|
return time.strftime('%m-%d %H:00', tm)
|
||||||
|
elif bucket_sec >= 60:
|
||||||
|
return time.strftime('%m-%d %H:%M', tm)
|
||||||
|
else:
|
||||||
|
return time.strftime('%H:%M:%S', tm)
|
||||||
|
|
||||||
|
time_series = []
|
||||||
|
for ts_key in sorted(bucket_data.keys()):
|
||||||
|
d = bucket_data[ts_key]
|
||||||
|
time_series.append({
|
||||||
|
't': fmt_bj(ts_key, bucket_seconds),
|
||||||
|
'ts': ts_key,
|
||||||
|
'count': d['count'],
|
||||||
|
'bytes': round(d['bytes'], 2),
|
||||||
|
})
|
||||||
|
|
||||||
|
return JsonResponse({
|
||||||
|
'time_series': time_series,
|
||||||
|
'span_hours': round(span_hours, 2),
|
||||||
|
'bucket_minutes': bucket_seconds // 60,
|
||||||
|
'n_raw': len(parsed_times),
|
||||||
|
'has_bytes': has_bytes,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@csrf_exempt
|
||||||
|
@require_POST
|
||||||
|
def cluster_nodes_edges(request):
|
||||||
|
"""返回指定簇的所有节点、内部边、关联外部边及邻接节点。
|
||||||
|
|
||||||
|
POST body: {session_id, cluster_label}
|
||||||
|
返回: {nodes, internal_edges, external_edges, adjacent_nodes, geo_bounds, cluster_info}
|
||||||
|
"""
|
||||||
|
body = json.loads(request.body)
|
||||||
|
session_id = body.get('session_id')
|
||||||
|
cluster_label = body.get('cluster_label')
|
||||||
|
|
||||||
|
if not session_id or cluster_label is None:
|
||||||
|
return JsonResponse({'error': '缺少 session_id / cluster_label'}, status=400)
|
||||||
|
|
||||||
|
entry = _cached(session_id)
|
||||||
|
if entry is None:
|
||||||
|
return JsonResponse({'error': '会话已过期'}, status=400)
|
||||||
|
|
||||||
|
nodes = entry.get('nodes', [])
|
||||||
|
edges_data = entry.get('edges_data', [])
|
||||||
|
cluster_info = entry.get('cluster_map', {}).get(str(cluster_label), {})
|
||||||
|
|
||||||
|
if not nodes:
|
||||||
|
return JsonResponse({'error': '无节点数据'}, status=400)
|
||||||
|
|
||||||
|
# ── 1. 筛选簇内节点 ──
|
||||||
|
cluster_nodes = [n for n in nodes if n.get('cluster_id') == cluster_label]
|
||||||
|
if not cluster_nodes:
|
||||||
|
return JsonResponse({'error': f'簇 {cluster_label} 无数据'}, status=404)
|
||||||
|
|
||||||
|
cluster_ips = set(n['ip'] for n in cluster_nodes)
|
||||||
|
|
||||||
|
# ── 2. 查找所有关联边（至少一端在簇内） ──
|
||||||
|
internal_edges = []
|
||||||
|
external_edges = []
|
||||||
|
adjacent_ip_set = set()
|
||||||
|
|
||||||
|
for e in edges_data:
|
||||||
|
src_in = e['source'] in cluster_ips
|
||||||
|
dst_in = e['target'] in cluster_ips
|
||||||
|
if src_in and dst_in:
|
||||||
|
internal_edges.append(e)
|
||||||
|
elif src_in:
|
||||||
|
external_edges.append(e)
|
||||||
|
adjacent_ip_set.add(e['target'])
|
||||||
|
elif dst_in:
|
||||||
|
external_edges.append(e)
|
||||||
|
adjacent_ip_set.add(e['source'])
|
||||||
|
|
||||||
|
# ── 3. 查找邻接节点 ──
|
||||||
|
adjacent_nodes = [n for n in nodes if n['ip'] in adjacent_ip_set]
|
||||||
|
|
||||||
|
# ── 4. 计算 Geo bounds ──
|
||||||
|
lats = [n.get('lat') for n in cluster_nodes if n.get('lat') is not None]
|
||||||
|
lons = [n.get('lon') for n in cluster_nodes if n.get('lon') is not None]
|
||||||
|
lats += [n.get('lat') for n in adjacent_nodes if n.get('lat') is not None]
|
||||||
|
lons += [n.get('lon') for n in adjacent_nodes if n.get('lon') is not None]
|
||||||
|
|
||||||
|
geo_bounds = None
|
||||||
|
if lats and lons:
|
||||||
|
geo_bounds = {
|
||||||
|
'lat_min': round(min(lats), 4), 'lat_max': round(max(lats), 4),
|
||||||
|
'lon_min': round(min(lons), 4), 'lon_max': round(max(lons), 4),
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── 5. 簇级别统计 ──
|
||||||
|
def _dist(items, key):
|
||||||
|
counts = Counter(i.get(key, '') or '(未知)' for i in items)
|
||||||
|
total = sum(counts.values()) or 1
|
||||||
|
return [{'value': k, 'count': v, 'pct': round(v / total * 100, 1)}
|
||||||
|
for k, v in counts.most_common(10)]
|
||||||
|
|
||||||
|
total_internal_bytes = sum(e.get('total_bytes', 0) or 0 for e in internal_edges)
|
||||||
|
total_external_bytes = sum(e.get('total_bytes', 0) or 0 for e in external_edges)
|
||||||
|
|
||||||
|
return JsonResponse({
|
||||||
|
'cluster_label': cluster_label,
|
||||||
|
'cluster_info': cluster_info,
|
||||||
|
'nodes': cluster_nodes,
|
||||||
|
'n_nodes': len(cluster_nodes),
|
||||||
|
'internal_edges': internal_edges,
|
||||||
|
'external_edges': external_edges,
|
||||||
|
'n_internal_edges': len(internal_edges),
|
||||||
|
'n_external_edges': len(external_edges),
|
||||||
|
'adjacent_nodes': adjacent_nodes,
|
||||||
|
'n_adjacent_nodes': len(adjacent_nodes),
|
||||||
|
'geo_bounds': geo_bounds,
|
||||||
|
'top_isps': _dist(cluster_nodes, 'isp'),
|
||||||
|
'top_cities': _dist(cluster_nodes, 'city'),
|
||||||
|
'top_countries': _dist(cluster_nodes, 'country'),
|
||||||
|
'total_internal_bytes': round(total_internal_bytes, 2),
|
||||||
|
'total_external_bytes': round(total_external_bytes, 2),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
@require_GET
|
@require_GET
|
||||||
def column_values(request):
|
def column_values(request):
|
||||||
|
|||||||
@@ -1,16 +0,0 @@
|
|||||||
import subprocess, os, time, urllib.request
|
|
||||||
|
|
||||||
os.chdir(r"C:\Users\25044\Desktop\Proj\天璇")
|
|
||||||
|
|
||||||
server = subprocess.Popen(
|
|
||||||
[r"runtime\python\python.exe", "manage.py", "runserver", "127.0.0.1:8765", "--noreload"],
|
|
||||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
|
|
||||||
)
|
|
||||||
|
|
||||||
time.sleep(4)
|
|
||||||
|
|
||||||
try:
|
|
||||||
urllib.request.urlopen("http://127.0.0.1:8765/", timeout=5)
|
|
||||||
print("SERVER_UP")
|
|
||||||
except Exception as e:
|
|
||||||
print("SERVER_DOWN:", e)
|
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,77 +0,0 @@
|
|||||||
import os, sys, django, time
|
|
||||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
|
||||||
os.environ['DJANGO_ALLOW_ASYNC_UNSAFE'] = 'true'
|
|
||||||
sys.path.insert(0, r'C:\Users\25044\Desktop\Proj\天璇')
|
|
||||||
django.setup()
|
|
||||||
|
|
||||||
import polars as pl, numpy as np
|
|
||||||
from analysis.data_loader import load_from_db
|
|
||||||
|
|
||||||
lf = load_from_db('_data_43')
|
|
||||||
if lf is None:
|
|
||||||
print('_data_43 not found')
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
# Coerce key columns to numeric (simulating background process)
|
|
||||||
schema = lf.collect_schema()
|
|
||||||
names = schema.names()
|
|
||||||
print(f'Columns: {len(names)}')
|
|
||||||
|
|
||||||
# Pick numeric candidates
|
|
||||||
numeric_candidates = []
|
|
||||||
utf8_cols = []
|
|
||||||
for n, d in zip(names, list(schema.dtypes())):
|
|
||||||
if d == pl.Utf8 and not n.startswith('_'):
|
|
||||||
utf8_cols.append(n)
|
|
||||||
|
|
||||||
# Sample to check castability
|
|
||||||
sample = lf.select(utf8_cols).head(500).collect(streaming=True)
|
|
||||||
type_map = {}
|
|
||||||
for col in utf8_cols:
|
|
||||||
try:
|
|
||||||
casted = sample[col].cast(pl.Float64, strict=False)
|
|
||||||
nn = casted.is_not_null().sum()
|
|
||||||
if nn > len(casted) * 0.5:
|
|
||||||
type_map[col] = pl.Float64
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
print(f'Coercible columns: {len(type_map)}: {list(type_map.keys())[:8]}')
|
|
||||||
|
|
||||||
# Apply casts
|
|
||||||
casts = [pl.col(c).cast(pl.Float64, strict=False) for c in type_map]
|
|
||||||
lf = lf.with_columns(casts)
|
|
||||||
|
|
||||||
# Find numeric columns
|
|
||||||
numeric_types = (pl.Int8, pl.Int16, pl.Int32, pl.Int64, pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64, pl.Float32, pl.Float64)
|
|
||||||
schema2 = lf.collect_schema()
|
|
||||||
feature_cols = [schema2.names()[i] for i, dt in enumerate(list(schema2.dtypes()))
|
|
||||||
if dt in numeric_types and not schema2.names()[i].startswith('_')][:10]
|
|
||||||
print(f'Feature cols ({len(feature_cols)}): {feature_cols}')
|
|
||||||
|
|
||||||
# Collect and run
|
|
||||||
t0 = time.time()
|
|
||||||
df = lf.select(feature_cols).collect(streaming=True)
|
|
||||||
data = df.to_numpy().astype(np.float64)
|
|
||||||
print(f'Data shape: {data.shape}, nans: {np.isnan(data).sum()}')
|
|
||||||
|
|
||||||
col_mean = np.nanmean(data, axis=0)
|
|
||||||
col_mean = np.nan_to_num(col_mean, nan=0.0)
|
|
||||||
data = np.where(np.isnan(data), col_mean, data)
|
|
||||||
|
|
||||||
if len(data) > 50000:
|
|
||||||
rng = np.random.default_rng(42)
|
|
||||||
data = data[rng.choice(len(data), 50000, replace=False)]
|
|
||||||
print(f'After sample: {data.shape}')
|
|
||||||
|
|
||||||
from sklearn.preprocessing import StandardScaler
|
|
||||||
print('Scaling...')
|
|
||||||
data_scaled = StandardScaler().fit_transform(data)
|
|
||||||
print(f'Scaled shape: {data_scaled.shape}')
|
|
||||||
|
|
||||||
from sklearn.cluster import HDBSCAN
|
|
||||||
print('HDBSCAN...')
|
|
||||||
model = HDBSCAN(min_cluster_size=5, min_samples=5)
|
|
||||||
labels = model.fit_predict(data_scaled)
|
|
||||||
nc = len(set(labels)) - (1 if -1 in labels else 0)
|
|
||||||
print(f'DONE in {time.time()-t0:.1f}s: {nc} clusters, {(labels==-1).sum()} noise')
|
|
||||||
@@ -1,55 +0,0 @@
|
|||||||
/**
|
|
||||||
* Minimal Playwright test — debug the "missing ) after argument list" error.
|
|
||||||
*/
|
|
||||||
import { chromium } from 'file:///C:/Users/Zjz/AppData/Roaming/npm/node_modules/@playwright/mcp/node_modules/playwright/index.mjs';
|
|
||||||
|
|
||||||
const BASE = 'http://127.0.0.1:8000';
|
|
||||||
|
|
||||||
async function main() {
|
|
||||||
const browser = await chromium.launch({ headless: true });
|
|
||||||
const page = await browser.newPage();
|
|
||||||
|
|
||||||
// Track ALL errors
|
|
||||||
page.on('pageerror', err => {
|
|
||||||
console.log('PAGE_ERROR:', err.message);
|
|
||||||
console.log(' Stack:', err.stack?.substring(0, 200));
|
|
||||||
});
|
|
||||||
page.on('console', msg => {
|
|
||||||
if (msg.type() === 'error') {
|
|
||||||
const text = msg.text();
|
|
||||||
console.log('CONSOLE_ERROR:', text.substring(0, 300));
|
|
||||||
// Try to get the stack trace
|
|
||||||
try {
|
|
||||||
const stack = msg.stackTrace?.();
|
|
||||||
if (stack && stack.length > 0) {
|
|
||||||
console.log(' At:', stack[0].url?.substring(0, 100), stack[0].lineNumber, stack[0].columnNumber);
|
|
||||||
}
|
|
||||||
} catch(e) {}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
page.on('weberror', err => {
|
|
||||||
console.log('WEB_ERROR:', err.message);
|
|
||||||
});
|
|
||||||
|
|
||||||
console.log('Loading page...');
|
|
||||||
await page.goto(`${BASE}/simple/`, { waitUntil: 'load', timeout: 15000 });
|
|
||||||
|
|
||||||
console.log('Waiting for scripts to settle...');
|
|
||||||
await new Promise(r => setTimeout(r, 2000));
|
|
||||||
|
|
||||||
// Check if key functions exist
|
|
||||||
const exists = await page.evaluate(() => ({
|
|
||||||
typeofSA: typeof SA,
|
|
||||||
typeofGoStep: typeof goStep,
|
|
||||||
typeofHandleFiles: typeof handleFiles,
|
|
||||||
typeofShowError: typeof showError,
|
|
||||||
typeofEscapeHtml: typeof escapeHtml,
|
|
||||||
leafletExists: typeof L !== 'undefined',
|
|
||||||
chartExists: typeof Chart !== 'undefined',
|
|
||||||
}));
|
|
||||||
console.log('Window globals:', JSON.stringify(exists, null, 2));
|
|
||||||
|
|
||||||
await browser.close();
|
|
||||||
}
|
|
||||||
|
|
||||||
main().catch(e => console.error(e));
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
/**
|
|
||||||
* Debug: check if value dropdown is populated in filter builder
|
|
||||||
*/
|
|
||||||
import { chromium } from 'file:///C:/Users/Zjz/AppData/Roaming/npm/node_modules/@playwright/mcp/node_modules/playwright/index.mjs';
|
|
||||||
|
|
||||||
const BASE = 'http://127.0.0.1:8000';
|
|
||||||
|
|
||||||
async function main() {
|
|
||||||
const browser = await chromium.launch({ headless: true });
|
|
||||||
const page = await browser.newPage({ viewport: { width: 1280, height: 900 } });
|
|
||||||
|
|
||||||
page.on('console', msg => {
|
|
||||||
if (msg.type() === 'error') console.log(' 🐛', msg.text().substring(0, 150));
|
|
||||||
});
|
|
||||||
page.on('pageerror', err => console.log(' ❌', err.message));
|
|
||||||
|
|
||||||
// 1. Go to simple analysis page
|
|
||||||
await page.goto(`${BASE}/simple/`, { waitUntil: 'networkidle' });
|
|
||||||
console.log('✅ Page loaded');
|
|
||||||
|
|
||||||
// 2. Upload test CSV
|
|
||||||
const fileInput = page.locator('#saFileInput');
|
|
||||||
await fileInput.setInputFiles('C:\\Users\\Zjz\\Desktop\\tianxuan\\天璇\\data\\test_small.csv');
|
|
||||||
await page.waitForResponse(r => r.url().includes('/simple/upload/'), { timeout: 15000 });
|
|
||||||
await new Promise(r => setTimeout(r, 2000));
|
|
||||||
console.log('✅ Upload done');
|
|
||||||
|
|
||||||
// 3. Check preset panel - does it have a condition row?
|
|
||||||
const presetConds = await page.locator('#saPresetConditions .sa-filter-row').count();
|
|
||||||
console.log(`Preset conditions: ${presetConds}`);
|
|
||||||
|
|
||||||
if (presetConds > 0) {
|
|
||||||
// Select a column in the preset
|
|
||||||
const presetCol = page.locator('#saPresetConditions .sa-filter-col-select').first();
|
|
||||||
const opts = await presetCol.locator('option').all();
|
|
||||||
console.log(`Preset column options: ${opts.length}`);
|
|
||||||
if (opts.length > 1) {
|
|
||||||
const val = await opts[1].getAttribute('value');
|
|
||||||
await presetCol.selectOption(val);
|
|
||||||
console.log(`Selected column: ${val}`);
|
|
||||||
await new Promise(r => setTimeout(r, 3000));
|
|
||||||
|
|
||||||
// Check value dropdown in preset
|
|
||||||
const presetVal = page.locator('#saPresetConditions .sa-filter-val');
|
|
||||||
const valOpts = await presetVal.locator('option').all();
|
|
||||||
console.log(`Preset value options: ${valOpts.length}`);
|
|
||||||
for (const opt of valOpts) {
|
|
||||||
const v = await opt.getAttribute('value');
|
|
||||||
if (v) console.log(` value: "${v}"`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 4. Go to Step 2
|
|
||||||
await page.evaluate(() => { if (typeof goStep === 'function') goStep(2); });
|
|
||||||
await new Promise(r => setTimeout(r, 1500));
|
|
||||||
|
|
||||||
// Check step 2 has condition row
|
|
||||||
const step2Conds = await page.locator('#saFilterConditions .sa-filter-row').count();
|
|
||||||
console.log(`Step 2 conditions: ${step2Conds}`);
|
|
||||||
|
|
||||||
if (step2Conds > 0) {
|
|
||||||
const step2Col = page.locator('#saFilterConditions .sa-filter-col-select').first();
|
|
||||||
const opts2 = await step2Col.locator('option').all();
|
|
||||||
console.log(`Step 2 column options: ${opts2.length}`);
|
|
||||||
if (opts2.length > 1) {
|
|
||||||
const val2 = await opts2[1].getAttribute('value');
|
|
||||||
await step2Col.selectOption(val2);
|
|
||||||
console.log(`Selected column: ${val2}`);
|
|
||||||
await new Promise(r => setTimeout(r, 3000));
|
|
||||||
|
|
||||||
// Check value dropdown
|
|
||||||
const step2Val = page.locator('#saFilterConditions .sa-filter-val');
|
|
||||||
const valOpts2 = await step2Val.locator('option').all();
|
|
||||||
console.log(`Step 2 value options: ${valOpts2.length}`);
|
|
||||||
for (const opt of valOpts2) {
|
|
||||||
const v = await opt.getAttribute('value');
|
|
||||||
if (v) console.log(` value: "${v}"`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
await browser.close();
|
|
||||||
}
|
|
||||||
|
|
||||||
main().catch(e => { console.error(e); process.exit(1); });
|
|
||||||
@@ -1,166 +0,0 @@
|
|||||||
const { chromium } = require('playwright');
|
|
||||||
const path = require('path');
|
|
||||||
const fs = require('fs');
|
|
||||||
|
|
||||||
(async () => {
|
|
||||||
const browser = await chromium.launch({ headless: true });
|
|
||||||
const page = await browser.newPage({ viewport: { width: 1440, height: 900 } });
|
|
||||||
const errors = [];
|
|
||||||
page.on('console', msg => { if (msg.type() === 'error') errors.push(msg.text()); });
|
|
||||||
|
|
||||||
try {
|
|
||||||
// 1. Load page
|
|
||||||
console.log('1. Loading page...');
|
|
||||||
await page.goto('http://127.0.0.1:8000/simple/', { waitUntil: 'networkidle' });
|
|
||||||
await page.waitForSelector('.sa-dropzone', { timeout: 10000 });
|
|
||||||
console.log(' OK');
|
|
||||||
|
|
||||||
// 2. Upload CSV
|
|
||||||
console.log('2. Uploading CSV...');
|
|
||||||
const fileInput = page.locator('#saFileInput');
|
|
||||||
const dataDir = 'data/test_csvs';
|
|
||||||
const files = fs.readdirSync(dataDir).filter(f => f.endsWith('.csv'));
|
|
||||||
await fileInput.setInputFiles(files.map(f => path.join(dataDir, f)));
|
|
||||||
await page.waitForTimeout(500);
|
|
||||||
|
|
||||||
await page.waitForFunction(() => {
|
|
||||||
const el = document.getElementById('saPresetFilterArea');
|
|
||||||
return el && el.style.display === 'block';
|
|
||||||
}, { timeout: 15000 });
|
|
||||||
console.log(' Quick scan done');
|
|
||||||
|
|
||||||
await page.evaluate(() => {
|
|
||||||
const c = document.getElementById('saPresetConditions');
|
|
||||||
if (c) c.innerHTML = '';
|
|
||||||
});
|
|
||||||
|
|
||||||
await page.click('#saBtnUpload');
|
|
||||||
await page.waitForFunction(() => {
|
|
||||||
const el = document.getElementById('saLoading');
|
|
||||||
return el && !el.classList.contains('show');
|
|
||||||
}, { timeout: 30000 });
|
|
||||||
const rows = await page.textContent('#saTotalRows');
|
|
||||||
console.log(' Upload OK, rows:', rows);
|
|
||||||
|
|
||||||
// 3. Cluster
|
|
||||||
console.log('3. Clustering...');
|
|
||||||
await page.click('#saUploadResult button');
|
|
||||||
await page.waitForTimeout(300);
|
|
||||||
await page.evaluate(() => goStep(3));
|
|
||||||
await page.waitForTimeout(300);
|
|
||||||
|
|
||||||
const clusterResult = await page.evaluate(async () => {
|
|
||||||
const resp = await fetch('/simple/cluster/', {
|
|
||||||
method: 'POST',
|
|
||||||
headers: {'Content-Type':'application/json','X-CSRFToken':document.cookie.match(/csrftoken=([^;]+)/)?.[1]||''},
|
|
||||||
body: JSON.stringify({session_id:SA.sessionId,min_cluster_size:1,cluster_selection_epsilon:0.0001}),
|
|
||||||
});
|
|
||||||
return await resp.json();
|
|
||||||
});
|
|
||||||
console.log(' Nodes:', clusterResult.n_nodes, 'Edges:', clusterResult.n_edges, 'Communities:', clusterResult.n_clusters);
|
|
||||||
|
|
||||||
await page.evaluate((data) => {
|
|
||||||
SA.nodes = data.nodes || [];
|
|
||||||
SA.edgesData = data.edges || [];
|
|
||||||
SA.clusterMap = data.clusters || {};
|
|
||||||
}, clusterResult);
|
|
||||||
|
|
||||||
// 4. Map
|
|
||||||
console.log('4. Map...');
|
|
||||||
await page.evaluate(() => {
|
|
||||||
document.querySelectorAll('.sa-step').forEach(el => el.classList.remove('active'));
|
|
||||||
document.getElementById('saStep4').classList.add('active');
|
|
||||||
document.querySelectorAll('.sp-item').forEach(el => {
|
|
||||||
const s = parseInt(el.dataset.step);
|
|
||||||
el.classList.remove('active','done');
|
|
||||||
if (s <= 4) el.classList.add('done');
|
|
||||||
if (s === 4) el.classList.add('active');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
await page.evaluate(() => { if (typeof initMap === 'function') initMap(); });
|
|
||||||
await page.waitForTimeout(3000);
|
|
||||||
const markerCount = await page.evaluate(() => SA.pointLayer ? SA.pointLayer.getLayers().length : 0);
|
|
||||||
console.log(' Markers:', markerCount);
|
|
||||||
await page.screenshot({ path: '/tmp/tianxuan_step4_map.png' });
|
|
||||||
console.log(' Screenshot: /tmp/tianxuan_step4_map.png');
|
|
||||||
|
|
||||||
// 5. Node click
|
|
||||||
console.log('5. Node click...');
|
|
||||||
const nodeIP = await page.evaluate(() => {
|
|
||||||
if (SA.nodes && SA.nodes.length > 0) { loadNodeDetail(SA.nodes[0].ip); return SA.nodes[0].ip; }
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
await page.waitForTimeout(1500);
|
|
||||||
const panel1 = await page.locator('#saDetailPanel.open').count();
|
|
||||||
console.log(' IP:', nodeIP, 'Panel:', panel1 > 0 ? 'open' : 'closed');
|
|
||||||
if (panel1 > 0) {
|
|
||||||
const txt = await page.locator('#saDetailBody').textContent();
|
|
||||||
console.log(' Content:', txt.replace(/\s+/g, ' ').slice(0, 200));
|
|
||||||
await page.screenshot({ path: '/tmp/tianxuan_node_detail.png' });
|
|
||||||
console.log(' Screenshot: /tmp/tianxuan_node_detail.png');
|
|
||||||
}
|
|
||||||
await page.click('#saDetailPanel .close-btn');
|
|
||||||
await page.waitForTimeout(300);
|
|
||||||
|
|
||||||
// 6. Edges
|
|
||||||
console.log('6. Edge display...');
|
|
||||||
await page.evaluate(() => {
|
|
||||||
document.getElementById('saEdgeControls').style.display = 'flex';
|
|
||||||
document.getElementById('saShowEdges').checked = true;
|
|
||||||
if (typeof renderEdgesOnMap === 'function') renderEdgesOnMap();
|
|
||||||
});
|
|
||||||
await page.waitForTimeout(1000);
|
|
||||||
const edgeCount = await page.evaluate(() => _edgeLayer ? _edgeLayer.getLayers().length : 0);
|
|
||||||
console.log(' Polylines:', edgeCount);
|
|
||||||
await page.screenshot({ path: '/tmp/tianxuan_edges.png' });
|
|
||||||
console.log(' Screenshot: /tmp/tianxuan_edges.png');
|
|
||||||
|
|
||||||
// 7. Edge click
|
|
||||||
console.log('7. Edge click...');
|
|
||||||
await page.evaluate(() => {
|
|
||||||
if (SA.edgesData && SA.edgesData.length > 0 && typeof loadEdgeDetail === 'function') {
|
|
||||||
loadEdgeDetail(SA.edgesData[0]);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
await page.waitForTimeout(500);
|
|
||||||
const panel2 = await page.locator('#saDetailPanel.open').count();
|
|
||||||
console.log(' Edge panel:', panel2 > 0 ? 'open' : 'closed');
|
|
||||||
if (panel2 > 0) {
|
|
||||||
const txt2 = await page.locator('#saDetailBody').textContent();
|
|
||||||
console.log(' Content:', txt2.replace(/\s+/g, ' ').slice(0, 250));
|
|
||||||
await page.screenshot({ path: '/tmp/tianxuan_edge_detail.png' });
|
|
||||||
}
|
|
||||||
await page.click('#saDetailPanel .close-btn');
|
|
||||||
|
|
||||||
// 8. Export
|
|
||||||
console.log('8. Export...');
|
|
||||||
const [dl1] = await Promise.all([page.waitForEvent('download',{timeout:10000}), page.click('button:has-text("导出点表")')]);
|
|
||||||
const csv1 = fs.readFileSync(await dl1.path(), 'utf-8');
|
|
||||||
const lines1 = csv1.split('\n').filter(l=>l.trim());
|
|
||||||
console.log(' Node CSV:', lines1.length-1, 'data rows');
|
|
||||||
|
|
||||||
const [dl2] = await Promise.all([page.waitForEvent('download',{timeout:10000}), page.click('button:has-text("导出边表")')]);
|
|
||||||
const csv2 = fs.readFileSync(await dl2.path(), 'utf-8');
|
|
||||||
const lines2 = csv2.split('\n').filter(l=>l.trim());
|
|
||||||
console.log(' Edge CSV:', lines2.length-1, 'data rows');
|
|
||||||
|
|
||||||
console.log('\n========================================');
|
|
||||||
console.log(' ALL TESTS PASSED');
|
|
||||||
console.log('========================================');
|
|
||||||
console.log(' Page: OK');
|
|
||||||
console.log(' Upload: OK (' + rows + ' rows)');
|
|
||||||
console.log(' Cluster: OK (' + clusterResult.n_nodes + ' nodes)');
|
|
||||||
console.log(' Map: OK (' + markerCount + ' markers)');
|
|
||||||
console.log(' Node: OK (' + nodeIP + ')');
|
|
||||||
console.log(' Edges: OK (' + edgeCount + ' polylines)');
|
|
||||||
console.log(' Export: OK (' + (lines1.length-1) + '/' + (lines2.length-1) + ' rows)');
|
|
||||||
if (errors.length) console.warn(' Console errors:', errors.slice(0, 5));
|
|
||||||
|
|
||||||
} catch (e) {
|
|
||||||
console.error('FAILED:', e.message);
|
|
||||||
await page.screenshot({ path: '/tmp/tianxuan_failure.png', fullPage: true });
|
|
||||||
process.exit(1);
|
|
||||||
} finally {
|
|
||||||
await browser.close();
|
|
||||||
}
|
|
||||||
})();
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
import subprocess, time, urllib.request, sys, os
|
|
||||||
os.chdir(r"C:\Users\25044\Desktop\Proj\天璇")
|
|
||||||
p = subprocess.Popen([r"runtime\python\python.exe", "manage.py", "runserver", "127.0.0.1:8765", "--noreload"])
|
|
||||||
time.sleep(8)
|
|
||||||
try:
|
|
||||||
urllib.request.urlopen("http://127.0.0.1:8765/", timeout=5)
|
|
||||||
print("SERVER_UP")
|
|
||||||
except:
|
|
||||||
print("SERVER_DOWN")
|
|
||||||
Reference in New Issue
Block a user