feat: v2.0.1beta - new preprocessing pipeline, 3D UMAP visualization, startup optimization, docs rewrite
This commit is contained in:
PM-pinou
@@ -1,4 +1,4 @@
|
||||
# 天璇 (TianXuan) -Agent Knowledge Base
|
||||
# 天璇 (TianXuan) — Agent Knowledge Base
|
||||
|
||||
## Project Identity
|
||||
|
||||
@@ -6,120 +6,196 @@
|
||||
|-------|-------|
|
||||
| 项目名称 | 天璇 (TianXuan) |
|
||||
| 原始名称 | tls-analyzer |
|
||||
| 核心功能 | TLS 流数据分析、实体画像、聚类3D地球可视化、PCA散点图|
|
||||
| 核心功能 | TLS 流数据分析、实体画像、聚类 3D 地球可视化、UMAP/SVD 降维散点图 |
|
||||
| Python 版本 | 3.12 (embedded portable at `runtime/python/python.exe`) |
|
||||
| Polars 版本 | 1.42.1 (精确锁定) |
|
||||
| 数据框架 | Polars (LazyFrame + streaming) |
|
||||
| 机器学习 | scikit-learn (HDBSCAN, KMeans, UMAP, TruncatedSVD, StandardScaler) |
|
||||
| Web框架 | Django 4.2 + SQLite WAL |
|
||||
| 前端3D | Three.js (603KB, 离线, 地球贴图1.4MB) |
|
||||
| 前端图表 | 纯Canvas散点图(无Chart.js) |
|
||||
| LLM协议 | MCP (stdio transport) |
|
||||
| 启动方式 | `run.bat`（双击即用）/ `runtime\python\python.exe manage.py runserver`（开发） |
|
||||
| 目标设备 | Windows 10 1902, 4GB RAM, 无独立显卡（纯核显） |
|
||||
| 机器学习 | scikit-learn (HDBSCAN, KMeans, MiniBatchKMeans, IsolationForest), UMAP, TruncatedSVD, StandardScaler |
|
||||
| Web 框架 | Django 4.2 + SQLite WAL |
|
||||
| 前端 3D | Three.js (603KB, 离线, 地球贴图 1.4MB) |
|
||||
| 前端图表 | 纯 Canvas 2D 散点图 / Leaflet 地图 / Canvas 地理分布 |
|
||||
| 前端地图 | Leaflet (简易分析模块) |
|
||||
| LLM 协议 | MCP (stdio transport) + OpenAI 兼容 API |
|
||||
| 启动方式 | `run.bat` (双击即用) / `runtime\python\python.exe manage.py runserver` (开发) |
|
||||
| 目标设备 | Windows 10 1902+, 4GB RAM, 无独立显卡 (核显) |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
tianxuan/
|
||||
├── tianxuan/ # Django 项目配置
|
||||
│ ├── settings.py # ALLOWED_HOSTS 自动检测 LOGGING 文件+stderr
|
||||
│ ├── wsgi.py # SQLite 启动自修复
|
||||
│ ├── urls.py
|
||||
│ └── llm_orchestrator.py # LLM 编排(32B/284B 双兼容
|
||||
│ ├── analysis/ # 核心分析模块 (Django app)
|
||||
│ ├── data_loader.py # CSV 加载, BOM检测 schema_strict, 递归glob, 中文路径
|
||||
│ ├── data_profiler.py # 列统计+ 相关性矩阵(无pandas回退:numpy)
|
||||
│ ├── entity_detector.py # 实体列自动检测(tuple关键词+ unique_ratio + null惩罚)
|
||||
│ ├── entity_aggregator.py # 实体聚合 (tuple关键词 lat/lon检测 多列group_by, IP子网)
|
||||
│ ├── session_store.py # 线程安全单例内存存储
|
||||
│ ├── tool_registry.py # 12个MCP工具 + DB持久化(sync_to_async)
|
||||
│ ├── mcp_server.py # MCP stdio server
|
||||
│ ├── views.py # 所有Django视图 (上传/手动/LLM/配置/LLM测试/日志/地球)
|
||||
│ ├── urls.py # 16条路由 ├── models.py # ORM: AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
||||
│ ├── data_validator.py # 列校验(缺失异常值IP有效性
|
||||
│ ├── type_classifier.py # 值优先类型检测(MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON)
|
||||
│ ├── geoip.py # GeoIP 经纬度查询(data/geoip_data.txt)
|
||||
│ ├── admin.py
|
||||
│ └── management/commands/
|
||||
│ ├── start_mcp.py # MCP服务器启动命令
|
||||
│ └── run_pipeline.py # 一键CLI管道命令
|
||||
│ ├── config/
|
||||
│ ├── config.yaml # 自动生成默认配置 (含entity.subnet_masks, columns覆盖)
|
||||
│ ├── loader.py # Pydantic 配置加载 + 文件修改检测
|
||||
│ └── __init__.py
|
||||
│ ├── templates/
|
||||
│ ├── base.html # 导航: 首页/上传/手动/LLM/记录/地球/配置
|
||||
│ └── analysis/
|
||||
│ tianxuan/ # 全部11个页面模板
|
||||
│ ├── dashboard.html # 首页 - 最近运行
|
||||
│ ├── upload.html # CSV上传 - 拖拽+多文件删除+进度
|
||||
│ ├── manual.html # 3步手动向- 选数据→设参数→运行
|
||||
│ ├── auto.html # LLM自动分析 - 实时日志取消
|
||||
│ ├── config.html # 配置编辑 - LLM连通性测试
|
||||
│ ├── run_list.html # 运行记录列表
|
||||
│ ├── run_detail.html # 运行详情摘要
|
||||
│ ├── cluster_overview.html # 聚类概览 - 纯Canvas散点PCA+地理
|
||||
│ ├── cluster_detail.html # 簇详情- 特征实体列表
|
||||
│ ├── entity_profile.html # 实体画像 - 特征偏离+Canvas单点
|
||||
│ ├── globe.html # 3D地球可视- Three.js 流量弧线+经纬国境
|
||||
│ └── log_viewer.html # 日志查看
|
||||
│ ├── static/tianxuan/ # 离线静态资源
|
||||
│ ├── three.min.js # Three.js 3D引擎 (603KB)
|
||||
│ ├── earth_atmos_2048.jpg # 地球贴图 (1.4MB)
|
||||
│ └── world_borders.js # 10国精简国境线轮廓 ├── runtime/python/ # 便携Python 3.12运行时(~593MB)
|
||||
│ ├── scripts/
|
||||
│ ├── gen_test_data.py # 简单测试数据生成(--globe模式使用真实GeoIP范围)
|
||||
│ ├── gen_complex_test.py # 复杂数据生成 (5000行4列含55%缺失)
|
||||
│ ├── column_survey.py # CSV列结构调查(统计各CSV的列类型/分布)
|
||||
│ ├── diagnose_compare.py # 跨机日志对比诊断
|
||||
│ └── debug_db.py # DB持久化调试 ├── tests/ # 92个测试(unit) / 121个测试(all)
|
||||
│ ├── test_data_loader.py # 26: BOM/schema/中文路径/递归glob
|
||||
│ ├── test_entity.py # 17: 关键词检测聚合
|
||||
│ ├── test_e2e.py # 6: 全端到端
|
||||
│ └── test_clustering_edge.py # HDBSCAN零样本降级 ├── docs/
|
||||
│ ├── 工作流.md # 完整分析流水线文档
|
||||
│ └── 故障诊断手册.md # 跨机问题诊断指南
|
||||
│ ├── run.bat # 用户启动 (含PYTHONUTF8=1)
|
||||
├── shell.bat # Django shell
|
||||
├── TlsDB.csv # 参考表头名清单（非实际数据├── README.md
|
||||
天璇/
|
||||
├── tianxuan/ Django 项目配置
|
||||
│ ├── settings.py ALLOWED_HOSTS 自动检测, LOGGING 文件+stderr
|
||||
│ ├── wsgi.py SQLite 启动自修复
|
||||
│ ├── urls.py 路由汇总 (含 simple_analysis 路由)
|
||||
│ └── llm_orchestrator.py LLM 编排 (32B/284B 双兼容, 策略式提示词)
|
||||
├── analysis/ 核心分析模块 (Django app)
|
||||
│ ├── data_loader.py CSV 加载, BOM/编码检测, schema 容错, 递归 glob, ZIP 解压
|
||||
│ ├── data_profiler.py 列统计 + 相关性矩阵 (numpy 无 pandas 回退)
|
||||
│ ├── entity_detector.py 实体列自动检测 (tuple 关键词 + unique_ratio + null 惩罚)
|
||||
│ ├── entity_aggregator.py 实体聚合 (tuple 关键词, lat/lon 检测, 多列 group_by, IP 子网)
|
||||
│ ├── data_validator.py 列校验 (缺失/异常值/IP 有效性)
|
||||
│ ├── type_classifier.py 值优先类型检测 (MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON/BOOL_ENUM)
|
||||
│ ├── geoip.py GeoIP 经纬度查询 (data/geoip_data.txt)
|
||||
│ ├── ip_clustering.py IP 子网聚类与转换
|
||||
│ ├── session_store.py 线程安全单例内存存储 + 特征结果
|
||||
│ ├── tool_registry.py 30 个 MCP 工具注册 + 处理函数 (sync_to_async DB)
|
||||
│ ├── mcp_server.py MCP stdio server
|
||||
│ ├── models.py ORM: AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
||||
│ ├── views.py 所有 Django 视图 (20+ 路由, 上传/手动/LLM/配置/地球/简易)
|
||||
│ ├── urls.py 路由注册
|
||||
│ ├── admin.py Django admin
|
||||
│ └── management/commands/
|
||||
│ ├── start_mcp.py MCP 服务器启动命令
|
||||
│ └── run_pipeline.py 一键 CLI 管道命令
|
||||
├── simple_analysis/ 简易分析模块 (独立 Django app)
|
||||
│ ├── views.py 上传 CSV → 按目标筛选 → 地理/IP 子网聚类
|
||||
│ ├── urls.py 路由: /simple/
|
||||
│ ├── templates/simple_analysis/ Leaflet 地图展示页面
|
||||
│ └── static/simple_analysis/ 模块静态资源
|
||||
├── config/ 配置管理
|
||||
│ ├── config.yaml 自动生成默认配置 (entity, server, data, clustering, llm)
|
||||
│ ├── loader.py Pydantic 配置加载 + 文件修改检测 + 缓存
|
||||
│ └── __init__.py
|
||||
├── templates/ 页面模板
|
||||
│ ├── base.html 导航: 首页/上传/手动/LLM/记录/地球/配置
|
||||
│ └── analysis/ 全部 11 个页面模板
|
||||
│ ├── dashboard.html 首页 - 最近运行
|
||||
│ ├── upload.html CSV 上传 - 拖拽+多文件删除+进度
|
||||
│ ├── manual.html 手动分析 - 工作流构建器 (添加/排序/保存/执行)
|
||||
│ ├── auto.html LLM 自动分析 - 实时日志, thinking panel, 工具调用折叠
|
||||
│ ├── config.html 配置编辑 - LLM 连通性测试
|
||||
│ ├── run_list.html 运行记录列表 (含 retry 按钮)
|
||||
│ ├── run_detail.html 运行详情摘要
|
||||
│ ├── cluster_overview.html 聚类概览 - Canvas 散点图 (PC1/PC2) + 地理分布 + Silhouette
|
||||
│ ├── cluster_detail.html 簇详情 - 特征实体列表
|
||||
│ ├── entity_profile.html 实体画像 - 特征偏离 + Canvas 单点
|
||||
│ ├── globe.html 3D 地球 - Three.js 流量弧线 + 经纬线 + 国境线
|
||||
│ └── log_viewer.html 日志查看
|
||||
├── static/tianxuan/ 离线静态资源
|
||||
│ ├── three.min.js Three.js 3D 引擎 (603KB)
|
||||
│ ├── earth_atmos_2048.jpg 地球贴图 (1.4MB)
|
||||
│ └── world_borders.js 177 国完整国境线 (269KB)
|
||||
├── runtime/python/ 便携 Python 3.12 运行时 (~593MB)
|
||||
├── scripts/ 工具脚本
|
||||
│ ├── gen_test_data.py 简单测试数据生成 (--globe 模式使用真实 GeoIP 范围)
|
||||
│ ├── gen_complex_test.py 复杂数据生成 (5000 行 4 列含 55% 缺失)
|
||||
│ ├── column_survey.py CSV 列结构调查
|
||||
│ ├── diagnose_compare.py 跨机日志对比诊断
|
||||
│ └── debug_db.py DB 持久化调试
|
||||
├── tests/ 测试
|
||||
│ ├── test_e2e_full.py 全端到端测试
|
||||
│ ├── test_e2e_batch.py 批量端到端测试
|
||||
│ ├── start_srv.py 测试服务器启动辅助
|
||||
│ └── _test_hdbscan.py HDBSCAN 零样本降级测试
|
||||
├── docs/
|
||||
│ ├── 工作流.md 完整分析流水线文档
|
||||
│ └── 故障诊断手册.md 跨机问题诊断指南
|
||||
├── run.bat 用户启动入口 (含 PYTHONUTF8=1)
|
||||
├── shell.bat Django shell
|
||||
├── update.bat 增量更新
|
||||
├── rollback.bat 更新回滚
|
||||
├── build.bat 构建打包
|
||||
├── TlsDB.csv 参考表头名清单
|
||||
├── README.md
|
||||
├── AGENTS.md
|
||||
└── manage.py
|
||||
```
|
||||
|
||||
## 12 MCP Tools
|
||||
## 30 MCP Tools
|
||||
|
||||
| # | 工具| 功能 | 必需参数 |
|
||||
|---|--------|------|----------|
|
||||
| 1 | `load_data` | 加载 CSV 文件（glob / 递归 / schema容错| `csv_glob` |
|
||||
| 2 | `profile_data` | 数据集概要统+ 相关性矩阵| `dataset_id` |
|
||||
| 3 | `filter_data` | 按条件过滤（12操作+ AND/OR| `dataset_id`, `filters` |
|
||||
| 4 | `preprocess_data` | 预处理（标准化编码/填充)| `dataset_id`, `columns` |
|
||||
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans + 质量评估| `dataset_id`, `cluster_columns` |
|
||||
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH| `cluster_result_id`, `dataset_id` |
|
||||
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA| `dataset_id`, `cluster_result_id` |
|
||||
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON| `result_id`, `output_path` |
|
||||
| 9 | `list_datasets` | 列出所有活跃数据集 | |
|
||||
| 10 | `drop_dataset` | 删除数据集释放内存| `dataset_id` |
|
||||
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据| `dataset_id` |
|
||||
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
|
||||
系统现有 **30 个 MCP 工具**，分为四层：
|
||||
|
||||
所有工具通过 `analysis/tool_registry.py` 注册，`analysis/mcp_server.py` 暴露标准 MCP stdio 接口
|
||||
### ⚡ 核心工具 (13 个) — 执行分析，可修改数据
|
||||
|
||||
| # | 工具 | 功能 | 必需参数 |
|
||||
|---|------|------|----------|
|
||||
| 1 | `load_data` | 加载 CSV 文件 (glob/递归/ZIP 解压/schema 容错) | `csv_glob` |
|
||||
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
|
||||
| 3 | `filter_data` | 按条件过滤 (12 操作符 + AND/OR 逻辑) | `dataset_id`, `filters` |
|
||||
| 4 | `preprocess_data` | 预处理 (StandardScaler/OneHot/填充/列删除) | `dataset_id`, `columns` |
|
||||
| 5 | `run_clustering` | 执行聚类 (HDBSCAN/KMeans + 自动特征过滤 + 质量评估) | `dataset_id`, `cluster_columns` |
|
||||
| 6 | `evaluate_clustering` | 评估聚类质量 (Silhouette/DB/CH/噪声比/簇大小) | `cluster_result_id`, `dataset_id` |
|
||||
| 7 | `extract_features` | 提取各聚类区分特征 (Z-Score/ANOVA) + UMAP-2D 嵌入 | `dataset_id`, `cluster_result_id` |
|
||||
| 8 | `filter_and_cluster` | 一步完成: 过滤 + 自动选数值列 + 聚类 | `dataset_id`, `filters` |
|
||||
| 9 | `build_entity_profiles` | 自动检测实体列 + 聚合特征 (多列复合键支持) | `dataset_id` |
|
||||
| 10 | `compute_scores` | 自适应代理/异常/威胁评分 (SVD 权重+ IQR 阈值) | `dataset_id` |
|
||||
| 11 | `detect_anomalies` | Isolation Forest 异常检测 (分块处理大数据集) | `dataset_id` |
|
||||
| 12 | `visualize_anomalies` | UMAP-2D 嵌入 + 聚类可视化 (按风险等级/聚类着色) | `dataset_id` |
|
||||
| 13 | `export_results` | 导出结果到磁盘 (CSV/Parquet/JSON) | `result_id`, `output_path` |
|
||||
|
||||
### 🔍 分析工具 (6 个) — 只读深潜，安全随时调用
|
||||
|
||||
| # | 工具 | 功能 |
|
||||
|---|------|------|
|
||||
| 14 | `analyze_patterns` | 流量模式: top 源/目标 IP、端口/TLS 版本/协议分布 |
|
||||
| 15 | `analyze_temporal` | 时间维度流量分析: 小时/日流量计数、繁忙/空闲时段 |
|
||||
| 16 | `analyze_fft` | FFT 频谱分析: 周期模式、频率特征、周期性评分 |
|
||||
| 17 | `analyze_tls_health` | TLS 安全态势: 旧版比例、弱加密、证书问题、SNI 异常 |
|
||||
| 18 | `analyze_geo_distribution` | 地理分布: top 来源国家、不可能旅行、ISP 多样性 |
|
||||
| 19 | `analyze_entity_detail` | 单实体深度调查: 流/版本/目标/时间模式/异常评分 |
|
||||
|
||||
### 🩺 诊断工具 (7 个) — 排查问题
|
||||
|
||||
| # | 工具 | 功能 |
|
||||
|---|------|------|
|
||||
| 20 | `validate_data` | 数据质量: schema 一致性、缺失率、列类型冲突 |
|
||||
| 21 | `explore_distributions` | 列分布统计: 唯一值、空值率、min/max/mean/std、直方图 |
|
||||
| 22 | `find_outliers` | IQR 方法检测数值列统计异常值 |
|
||||
| 23 | `diagnose_clustering` | 聚类效果差时诊断: 方差/相关分析 + 参数建议 |
|
||||
| 24 | `compare_datasets` | 对比两数据集: schema 差异、行数变化、值分布偏移 |
|
||||
| 25 | `export_debug_sample` | 导出原始数据样本 JSON 供外部调试 |
|
||||
| 26 | `repair_schema` | 修复 schema 不匹配: 列名对齐、大小写合并、填充缺失 |
|
||||
|
||||
### 工具 (4 个) — 数据管理
|
||||
|
||||
| # | 工具 | 功能 |
|
||||
|---|------|------|
|
||||
| 27 | `list_datasets` | 列出所有活跃数据集和结果 |
|
||||
| 28 | `drop_dataset` | 删除数据集释放内存 (支持 dataset/cluster/feature 三种类型) |
|
||||
| 29 | `clone_dataset` | 浅拷贝数据集 (LazyFrame 查询计划, 不复制数据) |
|
||||
| 30 | `compute_distance_matrix` | LLM 驱动: 用户编写 Python 距离函数, 逐行执行并返回评分摘要 |
|
||||
|
||||
所有工具通过 `analysis/tool_registry.py` 注册，`analysis/mcp_server.py` 暴露标准 MCP stdio 接口。
|
||||
|
||||
### 手动分析工作流
|
||||
|
||||
手动分析页面 (manual.html) 为**工作流构建器**:
|
||||
- 添加/移除/重排步骤
|
||||
- 每步选择工具 + 填写参数
|
||||
- **保存/加载/删除**方案 (存储于 `.omo/plans/<name>.json`)
|
||||
- 单步执行 / **全部执行** (自动串行)
|
||||
- 执行结果显示在每一步下面
|
||||
|
||||
### 标准分析流程
|
||||
|
||||
```
|
||||
load_data → profile_data → build_entity_profiles → compute_scores → run_clustering → extract_features
|
||||
```
|
||||
|
||||
LLM 自动模式由 `tianxuan/llm_orchestrator.py` 驱动，策略式提示词: 先 profile 了解数据 → 根据数据决策 → 失败自动诊断 → 重复调用检测 + 自动错误恢复。
|
||||
|
||||
## Config.Entity
|
||||
|
||||
`config/loader.py` 中的 Pydantic 模型
|
||||
`config/loader.py` 中的 Pydantic 模型:
|
||||
|
||||
```python
|
||||
class Entity(BaseModel):
|
||||
subnet_masks: list[int] = [] # 子网掩码列表，如 [24, 28]
|
||||
ip_columns: list[str] = ['src_ip', 'dst_ip'] # 需聚合的IP列```
|
||||
subnet_masks: list[int] = [] # 子网掩码列表，如 [24, 28]
|
||||
ip_columns: list[str] = ['src_ip', 'dst_ip'] # 需聚合的 IP 列
|
||||
```
|
||||
|
||||
config.yaml 默认配置:
|
||||
```yaml
|
||||
entity:
|
||||
subnet_masks: [24, 28]
|
||||
ip_columns: ["src_ip", "dst_ip"]
|
||||
subnet_masks: [24, 28]
|
||||
ip_columns: ["src_ip", "dst_ip"]
|
||||
```
|
||||
|
||||
`entity_aggregator.ip_to_subnet()` 负责 IP 到 CIDR 子网前缀的转换，
|
||||
`apply_subnet_aggregation()` 在 group_by 前对 IP 列应用子网掩码:
|
||||
```python
|
||||
ip_to_subnet("192.168.1.42", 24) # → "192.168.1.0/24"
|
||||
ip_to_subnet("192.168.1.42", 28) # → "192.168.1.32/28"
|
||||
```
|
||||
|
||||
---
|
||||
@@ -127,51 +203,54 @@ entity:
|
||||
## Architecture
|
||||
|
||||
- **tuple 替代 frozenset**: 避免跨机 PYTHONHASHSEED 差异
|
||||
- **PYTHONUTF8=1**: 跨机编码一致性- **类型安全聚合**: 聚合前检查dtype，非数值列cast
|
||||
- **sync_to_async ORM**: 异步上下文中Django ORM
|
||||
- **纯Canvas 图表**: 零外部JS 依赖
|
||||
- **上传目录隔离**: %APPDATA%/TianXuan/data/uploads/，与项目路径无关
|
||||
- **通用数值清理_coerce_to_float**: 处理 + / - /空白/N/A 等伪值- **值优先类型检测*: config→值→名称→STRING
|
||||
- **地球深度测试**: depthTest: true, depthWrite: false，r=5.5
|
||||
- **完整国境线*: Natural Earth 110m, 177国 286多边形 269KB
|
||||
- **PYTHONUTF8=1**: 跨机编码一致性 (run.bat 内置)
|
||||
- **类型安全聚合**: 聚合前检查 dtype, 非数值列 cast
|
||||
- **sync_to_async ORM**: 异步上下文中 Django ORM 安全调用
|
||||
- **纯 Canvas 图表**: 零外部 JS 依赖 (散点图/地理分布/Silhouette 柱状图)
|
||||
- **上传目录隔离**: `%APPDATA%/TianXuan/data/uploads/`, 与项目路径无关
|
||||
- **通用数值清理 _coerce_to_float**: 处理 `+` / `-` / 空白 / N/A 等伪值
|
||||
- **值优先类型检测**: config → 值 → 名称 → STRING (type_classifier.py)
|
||||
- **地球深度测试**: depthTest: true, depthWrite: false, r=5.5
|
||||
- **完整国境线**: Natural Earth 110m, 177 国 286 多边形 269KB
|
||||
- **UMAP 降维**: extract_features 中为实体数据计算 UMAP-2D 嵌入, 存储到 EntityProfile 表 (embedding_x/embedding_y), 训练 10K 样本 + 批量变换 1K 批次
|
||||
- **TruncatedSVD**: 聚类时特征 > 50 维自动降维; compute_scores 中学习自适应评分权重
|
||||
- **LLM 驱动距离计算**: compute_distance_matrix 工具允许 LLM 编写 Python 函数在受限命名空间 (math, numpy) 中逐行执行
|
||||
- **简单分析模块**: `simple_analysis/` 独立 Django app, 提供简化的上传→筛选→Leaflet 地图工作流, 零依赖外部地图服务 (OSM 瓦片离线缓存)
|
||||
|
||||
---
|
||||
|
||||
## Testing Protocol
|
||||
|
||||
### 运行全部测试
|
||||
### 运行测试
|
||||
|
||||
```bash
|
||||
# 单元测试（不需要Django)runtime\python\python.exe -m pytest tests/test_data_loader.py tests/test_type_classifier.py -q
|
||||
# 端到端测试 (自动启动后端、模拟前端行为、监控 stderr)
|
||||
runtime\python\python.exe tests\test_e2e_full.py
|
||||
runtime\python\python.exe tests\test_e2e_batch.py
|
||||
|
||||
# 集成测试（自动启动后端、模拟前端行为、监控stderr报错)runtime\python\python.exe tests/test_integration.py
|
||||
# HDBSCAN 零样本降级测试
|
||||
runtime\python\python.exe tests\_test_hdbscan.py
|
||||
```
|
||||
|
||||
```
|
||||
92 passed (unit tests); ALL INTEGRATION TESTS PASSED
|
||||
```
|
||||
|
||||
> ⚠️ **重要**: `tests/test_integration.py` 模拟前端HTML行为（上传CSV 轮询状态访问页面 删除）> **当前端HTML/API发生变化时，必须同步更新此脚*，否则集成测试将不再反映真实用户流程
|
||||
>
|
||||
### 跨机一致性测```bash
|
||||
### 跨机一致性测试
|
||||
|
||||
```bash
|
||||
# 两台机器各自执行
|
||||
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
||||
|
||||
# 对比日志
|
||||
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
|
||||
|
||||
```
|
||||
|
||||
### DB持久化测```bash
|
||||
### DB 持久化测试
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\debug_db.py
|
||||
# 期望: n_features=30 db_saved=True
|
||||
# 验证: run_analysis ClusterFeature 表有数据
|
||||
# 验证: AnalysisRun, ClusterFeature 表有数据
|
||||
```
|
||||
|
||||
### LLM编排测试
|
||||
### LLM 编排测试
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\test_llm_full.py
|
||||
@@ -181,274 +260,43 @@ runtime\python\python.exe scripts\test_llm_full.py
|
||||
### 生成测试数据
|
||||
|
||||
```bash
|
||||
# 简单数据runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
# 简单数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# Globe模式 (IP在真实GeoIP范围)
|
||||
# Globe 模式 (IP 在真实 GeoIP 范围)
|
||||
runtime\python\python.exe scripts\gen_test_data.py --globe
|
||||
|
||||
# 复杂数据 (5000行4列含55%缺失)
|
||||
# 复杂数据 (5000 行 4 列含 55% 缺失)
|
||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
|
||||
```
|
||||
|
||||
### 列结构调查```bash
|
||||
### 列结构调查
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Polars Version Compatibility
|
||||
|
||||
| API | 旧版(<1.0) | 新版(.0) |
|
||||
| API | 旧版 (<1.0) | 新版 (>=1.0) |
|
||||
|-----|-----------|-----------|
|
||||
| mode | `pl.mode()` | `pl.col(col).mode()` |
|
||||
| encoding | `encoding='utf-8'` | `encoding='utf8'` |
|
||||
| value_counts col | `'index'` | 原列名|
|
||||
| value_counts col | `'index'` | 原列名 |
|
||||
| streaming | `collect(streaming=True)` | `collect(engine='streaming')` |
|
||||
| truth value | `if series:` | `if series.item():` |
|
||||
|
||||
## 32B LLM Optimization
|
||||
|
||||
- Step guidance: 每步告诉模型下一步做什么- Truncation: 工具结果截断1200 chars
|
||||
- Timeout: 90s (32B 推理更慢)
|
||||
- Max steps: 15
|
||||
- Tool description: 一行简洁描述
|
||||
---
|
||||
|
||||
## v1.1.3 修复 (2026-07-21)
|
||||
|
||||
13 issues fixed across build system, frontend, pipeline, and data handling.
|
||||
|
||||
| # | 问题 | 状态| 详情 |
|
||||
|---|------|------|------|
|
||||
| 1 | 增量包构建修复| | update_info.json含SHA256、文件在files/子目录下、测试数据排除、__delete__支持 |
|
||||
| 2 | 构建顺序+三段版本号| | 先commit后bump再tag、bump_version支持vX.Y.Z、update.bat加PYTHONUTF8 |
|
||||
| 3 | 前端导航跳转 | | 上传后→手动分析页、LLM完成后→聚类概览|
|
||||
| 4 | 聚类前手动筛选| | manual.html下filter_data工具UI |
|
||||
| 5 | 地球→态势+拖拽修复 | | 导航改名、拖拽方向修正、惯性仅松开时施加|
|
||||
| 6 | DB模型同步 | | 所有模型与SQLite表完全匹配|
|
||||
| 7 | LLM后强制聚类| | LLM完成后自动运行clustering+PCA pipeline |
|
||||
| 8 | 列名不一致处理| | 保留所有列(union)、不丢弃额外列、gen_test_data支持--missing-cols |
|
||||
| 9 | FFT频谱分析 | | 新MCP工具analyze_fft(numpy.fft)、周期性特征提取|
|
||||
| 10 | 异常处理修复 | | 24处裸except/except:pass全部修复为带日志处理 |
|
||||
| 11 | LLM上下文增加| | 发送列类型+中文含义+空值率+众数给LLM |
|
||||
| 12 | 工作流自动保存| | 最0条自动方案保存到手动分析页、支持固定防淘汰 |
|
||||
| 13 | 列类型遵循TlsDB | | tls_ref_data作为权威类型源、未知列严格回退 |
|
||||
|
||||
## 修复计划 (2026-07-16)
|
||||
|
||||
| # | 问题 | 状态| 详情 |
|
||||
|---|------|------|------|
|
||||
| 1 | 重启后旧数据可分析| | SessionStore 元数据持久化为JSON 文件(`%APPDATA%/TianXuan/.session_store.json`)，wsgi.py 启动时自动恢复|
|
||||
| 2 | Web图标所有页| | `<link rel="icon">` 在 base.html 中，所有页面继承|
|
||||
| 3 | 上传按钮在顶部| | `position: sticky` |
|
||||
| 4 | Traceback完整 | | 移除 views.py 4处 + run_pipeline.py 1处`[:200]` 截断 |
|
||||
| 5 | 大数据不卡顿 | | 降采样`head=` 参数支持，手动POST 可传 head，LLM 自动决定，clamp 500|
|
||||
| 6 | 进度条| | progress_pct → 10/30/50/70/90/100 各阶段更新|
|
||||
| 7 | TLS 0ver hex识别 | | TLS_HEX_MAP 移至 type_classifier.py 值优先检测链ver列自动ENUM("TLSv1.2") |
|
||||
| 8 | Globe响应式缩放| | Canvas resize 事件绑定，vh 单位动态计算|
|
||||
| 9 | Globe ?data= | | 异常安全：try/except 包裹，空数据仍渲染页面|
|
||||
| 10 | 文件上传无限| | DATA_UPLOAD_MAX_NUMBER_FIELDS=1000000, FILE_UPLOAD_MAX_MEMORY_SIZE=100MB |
|
||||
| 11 | 上传页删除功能| | removeFile() + 按钮每文件|
|
||||
| 12 | LLM日志 | | progress_callback 写入 run_run_log |
|
||||
| 13 | 聚类图表 | | cluster_overview.html 含散点图/柱状态Silhouette |特征纯Canvas) |
|
||||
| 14 | 低内存不崩溃 | | MiniBatchKMeans + PCA降维 + 降采样head 参数 |
|
||||
| 15 | +号字符串识别为数| | type_classifier.py FLOAT检测前排除 `+` 前缀 |
|
||||
| 16 | 空闲高IO | | 日志级别 INFO，无需修改 |
|
||||
| 17 | 聚类效果 | ⚠️ | 预置了特征过滤降维，但具体效果需业务验证 |
|
||||
| 18 | 列名精确匹配 | | `re.match` 精确匹配 + `_find_column` tuple 关键词|
|
||||
|
||||
## v7 Final Verification (2026-07-20)
|
||||
|
||||
All **19 issues** fixed and verified. 92/92 unit tests passing. Full regression run completed.
|
||||
|
||||
| # | 问题 | 状态| 详情 |
|
||||
|---|------|------|------|
|
||||
| 1 | `handle_call` imported in llm_orchestrator.py | | `from analysis.tool_registry import handle_call` at line 7 |
|
||||
| 2 | Retry button + run_type filter | | Retry form in `run_list.html` (line 51), run_type dropdown filter, URL route `retry_run` added |
|
||||
| 3 | auto.html logs show tool+result | | `liveLog` polling every 3s from `/logs/?pos=` endpoint |
|
||||
| 4 | Data saved to SQLite tables | | `sqlite_table` field on `AnalysisRun` model, `drop_sqlite_table` in data_loader |
|
||||
| 5 | Error toast in base.html | | Complete toast system with error/success types, auto-dismiss, slide animations |
|
||||
| 6 | display_id in all URLs | | All 6 URL patterns (`run_detail`, `run_status`, `cluster_overview`, `cluster_detail`, `retry_run`, `delete_upload`) use `display_id` |
|
||||
| 7 | Delete buttons on dashboard+run_list | | Both pages have styled delete buttons with confirmation dialog |
|
||||
| 8 | Progress bar 100% on completion | | `progress_pct` field updated through pipeline stages (10/30/50/70/90/100) |
|
||||
| 9 | LLM thinking panel in auto.html | | `thinkingPanel` div with `llm_thinking` text area and step indicator |
|
||||
| 10 | Tool call accordion in auto.html | | `renderToolCalls()` with toggle-able accordion sections per tool step |
|
||||
| 11 | LazyFrame.sample() fixed | | Uses `lf.sample(n=...)` with explicit row count parameter (not deprecated `.sample()` API) |
|
||||
| 12 | `filter_and_cluster` tool exists | | Registered at line 407 in `tool_registry.py` as tool #13 |
|
||||
| 13 | Tool descriptions in Chinese | | All tool descriptions and parameter docs use Chinese |
|
||||
| 14 | Globe: lines, timestamp, inertia, polar flip | | Flow arcs (Line geometry), timestamp-based dot animation, quaternion momentum decay, camera-relative rotation (no polar flip) |
|
||||
| 15 | Config button in nav | | `{% url 'analysis:config' %}` link 配置 in base.html nav bar |
|
||||
| 16 | config host:port affects broadcast | | `settings.py` reads `_cfg.server.host` - auto-adds to `ALLOWED_HOSTS` |
|
||||
| 17 | SQLite timeout+retry mechanism | | `retry_on_lock` decorator in `db_utils.py`: 3 retries, exponential backoff, logs `[DB_LOCKED]` warning |
|
||||
| 18 | Delete crash fixed (null csv_glob) | | `delete_upload` view checks `if run.csv_glob:` before accessing path (line 366) |
|
||||
| 19 | No file count limit, streaming upload | | `DATA_UPLOAD_MAX_NUMBER_FIELDS=10000000`, `FILE_UPLOAD_MAX_MEMORY_SIZE=100MB` |
|
||||
|
||||
### Remaining issues
|
||||
- `test_e2e.py`, `test_entity.py`, `test_pipeline.py` still reference removed `entity_detector` module - need updating for new clustering pipeline
|
||||
- `test_clustering_edge.py` does not exist on disk (only referenced in AGENTS.md)
|
||||
- `retry_run` URL route was missing from `urls.py` - fixed during final verification
|
||||
|
||||
### Test Results (2026-07-20)
|
||||
| Test suite | Tests | Result |
|
||||
|------------|-------|--------|
|
||||
| `test_data_loader.py` | 28 | All passed |
|
||||
| `test_type_classifier.py` | 64 | All passed |
|
||||
| **Total** | **92** | ** All passed** |
|
||||
|
||||
## v8 Fix (2026-07-20)
|
||||
|
||||
| # | 问题 | 状态|
|
||||
|---|------|------|
|
||||
| 1 | 服务器启动config模块找不到| 修复 |
|
||||
| 2 | 首次启动无数据库| 修复 |
|
||||
| 3 | CSV中+'无法解析为f64 | 修复 |
|
||||
| 4 | 上传分块处理+释放临时文件 | 修复 |
|
||||
| 5 | 集成测试脚本 | 添加 |
|
||||
| 6 | TlsDB.csv文档说明 | 更新 |
|
||||
| 7 | load_from_db类型冲突 | 修复 |
|
||||
| 8 | CSV未合并处理| 修复 |
|
||||
|
||||
### Final Verification (2026-07-20)
|
||||
| Check | Result |
|
||||
|-------|--------|
|
||||
| `manage.py check` (system) | 0 errors |
|
||||
| Unit tests (data_loader + type_classifier) | 92 passed |
|
||||
| Integration test (full upload→load→analyze) | ALL INTEGRATION TESTS PASSED |
|
||||
| Git commit `v8` | `555ec29` |
|
||||
|
||||
## MCP 工具系统 (2026-07-19)
|
||||
|
||||
系统现有 **27 |MCP 工具**，分为三层：
|
||||
|
||||
### 工具架构
|
||||
|
||||
| | 数量 | 标签 | 用|
|
||||
|---|------|------|------|
|
||||
| Core | 7 | | 执行分析，可修改数据 |
|
||||
| Analysis | 5 | 🔍 | 只读深潜，安全随时调用|
|
||||
| Diagnostic | 7 | 🩺 | 排查问题 |
|
||||
| 其他 | 8 | | 数据加载/过滤/导出|
|
||||
|
||||
### 手动分析工作流
|
||||
手动分析页面已改为 **工作流构建器**:
|
||||
- 添加/移除/重排步骤
|
||||
- 每步选择工具 + 填写参数
|
||||
- **保存/加载/删除**方案（存储`.omo/plans/<name>.json`)- 单步执行 |**▶▶ 全部执行**（自动串行）
|
||||
- 执行结果显示在每一步下面
|
||||
### LLM 自动编排
|
||||
|
||||
LLM orchestrator 重写，支持：
|
||||
- 策略式系统提示词（先 profile 根据数据决策 失败自动诊断- 重复调用检测+ 自动错误恢复
|
||||
- 27 个工具全部可用- 中文总结分析结果
|
||||
|
||||
### 全链路测试
|
||||
| 测试 | 用例 | 结果 |
|
||||
|------|------|------|
|
||||
| 单元测试 | 120 | |
|
||||
| 管道测试 | 8 (3 datasets × 2 algos + edge) | |
|
||||
| CLI pipeline | e2e 5 stages | |
|
||||
| MCP 工具集成 | 10 (profile/patterns/validate/tls/geo/build/scores/detect/detail/visualize) | |
|
||||
|
||||
## v3 修复 (2026-07-17)
|
||||
|
||||
| # | 问题 | 状态| 详情 |
|
||||
|---|------|------|------|
|
||||
| 1 | 经纬度列名匹配`:ips.latd`) | | views.py直接字符串匹配+ entity_aggregator.py _LAT_KEYWORDS增加.latd后缀 |
|
||||
| 2 | cnrs/isrs布尔类型 | | entity_aggregator.py聚合前将"+"→True/空白→False |
|
||||
| 3 | 聚类图表（legend/柱状态负值） | | 纯Canvas：legend用HTML div+overflow、柱状图动态带宽、负值标签重新定义|
|
||||
| 4 | 地球缩放改变数据流大小| | Three.js arc sphere尺寸与camera距离反比，zoom时视觉大小不变|
|
||||
| 5 | 恶意/代理流量分析 | | entity_aggregator.py规则引擎：non_std_port_rate + modern_tls_rate + sni_missing_ratio + multi_country + proxy_score |
|
||||
| 6 | EntityProfile批量IO | | views.py + tool_registry.py 改为 bulk_create(ignore_conflicts=True, batch_size=1000) |
|
||||
| 7 | 重试分析 | 🗑| 已移除(v7 新架构不再需要entity_column 概念) |
|
||||
| 8 | D5降采样导致特征提取行数不匹配 | | _handle_run_clustering存储_ds_idx抽样索引，_handle_extract_features按索引过滤数据|
|
||||
|
||||
## v4 修复 (2026-07-17)
|
||||
|
||||
| # | 问题 | 状态| 详情 |
|
||||
|---|------|------|------|
|
||||
| 1 | 进度条位置| | DOM重排: submitBar → progressArea → fileList progressArea → fileList |
|
||||
| 2 | PCA sklearn RuntimeWarning | | `warnings.filterwarnings('ignore', category=RuntimeWarning, module='sklearn')` |
|
||||
| 3 | cluster_overview N+1查询 | | `prefetch_related('features')` + `.only()`字段限制 |
|
||||
| 4 | SQLite PRAGMA未优| | WAL + synchronous=NORMAL + cache_size=20MB + temp_store=MEMORY + mmap_size=3GB |
|
||||
| 5 | 零方差过滤崩| | 全零方差时回退`np.arange(data.shape[1])`保留原始特征 |
|
||||
|
||||
### 大规模测试结(2026-07-17)
|
||||
|
||||
| 测试 | 规模 | 结果 |
|
||||
|------|------|------|
|
||||
| 单元测试 | 120 tests | 全部通过 |
|
||||
| 管道测试 | 3 datasets × 2 algorithms | 8 tests OK |
|
||||
| 中等规模 | **100文件 × 10000行= 1M行 825MB** | 全部完成: 加载→实体检测→聚合→聚2285个实体→特征提取|
|
||||
|
||||
## Cleanup (2026-07-16)
|
||||
|
||||
对 v6 迭代引入的垃圾文件和代码破坏进行全面清理
|
||||
| 操作 | 文件 |
|
||||
|------|------|
|
||||
| 🗑删除 | 15个垃圾脚(wnl_to_user_csv, eval_clustering, debug_db2/3, debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_loop/format, test_e2e_full, test_32b, test_chinese_path, verify_runtime) |
|
||||
| 🗑删除 | data/wnl_converted.csv, data/globe_test.csv |
|
||||
| 🗑删除 | analysis/value_normalizer.py + 关联测试 + data_loader/entity_aggregator中的_norm引用 |
|
||||
| 🗑删除 | .omo/plans/ 旧计划文(tianxiu-v4/v5/v6, globe-v6) |
|
||||
| 🔧 重构 | views.py: 统一 _run_analysis_worker → _run_llm_analysis → _run_pipeline_worker |
|
||||
| 🔧 修复 | settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB→100MB |
|
||||
| 🔧 修复 | upload.html: 添加每文件删除按钮 + removeFile() |
|
||||
| 🔧 修复 | tool_registry.py: async函数中的同步ORM调用加sync_to_async包装 |
|
||||
| 🔧 修复 | run_pipeline.py: 改为调用 _run_pipeline_worker |
|
||||
| 🔧 替换 | gen_test_data.py: 3画像版→简单随机生成版 |
|
||||
| 验证 | 120/120 测试通过, 4次端到端管道测试全部成功 |
|
||||
|
||||
### v6 (2026-07-16) 已清理（不再维护
|
||||
> ⚠️ v6 引入value_normalizer 模块已被删除，_norm 列全部移除> entity_aggregator 回退到使用原始列名（0ver/0cph/cipher-suite等）
|
||||
| 模块 | 变动 | 当前状态|
|
||||
|------|------|---------|
|
||||
| value_normalizer.py | ~~新增: hex→enum归一化~~ | 🗑已删|
|
||||
| data_loader.py | ~~集成normalize_lf~~ | 🔧 恢复原始 |
|
||||
| entity_aggregator.py | ~~_norm列依赖~~ | 🔧 回退原始列名 |
|
||||
| gen_test_data.py | ~~3类流量画像~~ | 🔧 简单随机生|
|
||||
| 特征维度 | 20→3| 特征维度保留3维），去掉_norm列后的纯原始列特征|
|
||||
|
||||
## Completed (2026-07-16)
|
||||
|
||||
18个问题全部修复，3次端到端管道测试全部通过（简单CSV、用户自定义列名CSV、大规模CSV）
|
||||
| 工作流| 问题 | 状态|
|
||||
|--------|------|------|
|
||||
| A1 | 重启后旧数据可分析| 目录不存在时优雅降级，标记为failed |
|
||||
| A2 | Web图标所有页面显示| 创建favicon.svg + base.html添加link |
|
||||
| A3 | 上传按钮在顶部| 移至文件列表上方，sticky定位 |
|
||||
| A10 | 文件上传限制 | 去除了Django字段限制，增大至256MB/50GB |
|
||||
| A11 | 上传页删除功能| 所有状态显示删除按钮，处理中弹出确认|
|
||||
| B7 | TLS 0ver hex格式 | 添加TLS_HEX_MAP，识别3 03/03 04 |
|
||||
| B15 | "+"字符串识别为数字 | FLOAT检测添加/前缀检查|
|
||||
| B18 | 列名映射 | 使用精确^exact_name$匹配，无模糊猜测 |
|
||||
| C4 | Traceback截断 | tool_registry.py移除[:200] |
|
||||
| D5 | 大数据量卡顿 | 添加head参数10K降采样、MAX_ROWS=500 |
|
||||
| D6 | 进度条| 添加progress_pct/progress_msg字段 |
|
||||
| D14 | 低内存崩溃| MiniBatchKMeans、PCA降维、内存检测|
|
||||
| D16 | 空闲高IO | 日志级别已为INFO，无需修改 |
|
||||
| E8 | Globe响应式缩放| 动态H计算、vh单位、resize处理 |
|
||||
| E9 | Globe ?data=参数 | 支持从SessionStore直接加载数据|
|
||||
| F12 | LLM日志 | 添加callback机制、run_log字段 |
|
||||
| G13 | 聚类图表 | 添加簇大小柱状图、Silhouette对比|
|
||||
| H17 | 聚类质量 | 方差过滤、相关过滤、HDBSCAN自动调参 |
|
||||
|
||||
## v1.1.6 (2026-07-22)
|
||||
|
||||
Cleanup + minor fixes release.
|
||||
|
||||
| # | 问题 | 状态 | 描述 |
|
||||
|---|------|------|------|
|
||||
| 1 | schema sort fix | 修复 | 排序逻辑修改，确保最后一行被包含 |
|
||||
| 2 | batch upload | 修复 | 批量上传流程加固 |
|
||||
| 3 | temp dir to D: | 修复 | 临时目录移到 D: 盘避免 C: 盘空间占用 |
|
||||
| 4 | update.bat wildcard fix | 修复 | 通配符匹配优化使更新更可靠 |
|
||||
| 5 | cleanup test artifacts | 修复 | 删除 test CSV data, temp scripts, gen_fast.py |
|
||||
| 6 | VERSION sync | 修复 | VERSION 文件已更新为 v1.1.6 |
|
||||
|
||||
### Pending
|
||||
- tests/ 需要更新 entity_detector 相关
|
||||
- 大文件上传存在内存泄漏需要进一步排查
|
||||
- **Step guidance**: 每步告诉模型下一步做什么, 减少 32B 模型推理发散
|
||||
- **Truncation**: 工具结果截断 1200 chars (避免上下文窗口溢出)
|
||||
- **Timeout**: 90s (32B 推理更慢)
|
||||
- **Max steps**: 15
|
||||
- **Tool description**: 一行简洁描述 (中文)
|
||||
- **策略式提示词**: 先 profile → 根据数据决策 → 失败自动诊断 → 重复调用检测 + 自动错误恢复
|
||||
- **工具结果自动截断**: `_truncate_response()` 确保单次返回不超过 64 KB, 长列表/矩阵自动压缩
|
||||
|
||||
---
|
||||
|
||||
@@ -458,7 +306,7 @@ Cleanup + minor fixes release.
|
||||
|
||||
2. **全量测试** — After all changes, run full E2E test: generate 2000 CSVs, upload through frontend, verify complete pipeline. This validates the entire system end-to-end before considering work complete.
|
||||
|
||||
3. **更新文档和Git** — After tests pass: update AGENTS.md, commit all changes, push, clean up temp files and running processes. Do not leave dirty state (orphan processes, temp files, uncommitted changes).
|
||||
3. **更新文档和 Git** — After tests pass: update AGENTS.md, commit all changes, push, clean up temp files and running processes. Do not leave dirty state (orphan processes, temp files, uncommitted changes).
|
||||
|
||||
4. **同步依赖** — Use `git pull` to sync latest code, then `runtime\python\python.exe -m pip install -r requirements.txt` to install any new packages. Do this before starting any new work session.
|
||||
|
||||
|
||||
@@ -1,541 +1,95 @@
|
||||
# 天璇 (TianXuan) — TLS 流数据分析与实体画像系统
|
||||
# 天璇 (TianXuan) — TLS流数据分析平台
|
||||
|
||||
基于 Polars + scikit-learn + Three.js 的 TLS 流数据分析工具链。支持 CSV 批量导入、自动实体检测、多维聚合、聚类分析、特征提取、**3D 地球流量可视化**。提供 Django Web 操作界面和 MCP 协议接口，支持 32B 大语言模型编排分析流程。**全离线运行**，所有 JS 库/地图贴图/GeoIP 数据库均已内嵌。
|
||||
基于 Polars + scikit-learn + Three.js 的 TLS 流量分析工具。CSV 上传后支持手动分步分析（工作流构建器）、LLM 全自动分析（MCP 工具链）、聚类 3D 地球可视化，以及简易即用的一键上传→筛选→聚类→地图模块。全离线运行，内嵌 Python 3.12 运行时。
|
||||
|
||||
## TlsDB.csv — 参考表头名清单
|
||||
## 功能概述
|
||||
|
||||
`TlsDB.csv` 是 TLS 流量数据的**参考表头名清单**，包含常见的列名、含义和数据类型对照表。
|
||||
**此文件为参考对照表，非实际数据文件。**
|
||||
实际 CSV 文件的列名可能与表头名不同，分析工具会自动匹配。
|
||||
|
||||
---
|
||||
|
||||
## 目录
|
||||
|
||||
1. [系统概述](#1-系统概述)
|
||||
2. [快速开始](#2-快速开始)
|
||||
3. [用户指南](#3-用户指南)
|
||||
4. [页面功能说明](#4-页面功能说明)
|
||||
5. [CLI 管道](#5-cli-管道)
|
||||
6. [MCP 工具文档](#6-mcp-工具文档)
|
||||
7. [测试](#7-测试)
|
||||
8. [项目结构](#8-项目结构)
|
||||
9. [常见问题](#9-常见问题)
|
||||
|
||||
---
|
||||
|
||||
## 1. 系统概述
|
||||
|
||||
### 1.1 解决的问题
|
||||
|
||||
企业网络中 TLS 流量通常以 CSV 形式导出（每行一条流记录），数据量大、列名不统一、缺乏标签。本系统提供一条从原始 CSV 到实体画像的自动化分析流水线：
|
||||
|
||||
```
|
||||
CSV 文件 → 上传/加载 → 自动检测实体列 → 按实体聚合特征
|
||||
→ 实体级聚类 → 提取每簇区分特征 → Web 可视化 / MCP 接口
|
||||
```
|
||||
|
||||
### 1.2 核心能力
|
||||
|
||||
| 模块 | 功能说明 |
|
||||
|------|---------|
|
||||
| **数据加载** | 多文件合并、BOM 自动检测、跨文件 schema 容错、递归 `**/*.csv`、中文路径支持 |
|
||||
| **实体检测** | 基于列名关键词 + 唯一值比率 + 数据类型的三维打分，自动推荐实体列（如 `src_ip`、`sni`、`user`） |
|
||||
| **实体聚合** | 按实体分组计算流统计、目标多样性、TLS 特征、协议特征、时间模式、地理位置（经纬度） |
|
||||
| **IP 子网聚合** | 支持 /24 和 /28 子网掩码，同一子网下所有 IP 合并为一个实体节点 |
|
||||
| **多列复合键** | 支持 `(src_ip, dst_ip)` 等多列组合作为实体标识 |
|
||||
| **聚类分析** | HDBSCAN（自动确定簇数）/ KMeans，零样本自动降级，Silhouette / Davies-Bouldin 质量评估 |
|
||||
| **特征提取** | Z-Score / ANOVA 方法计算每个聚类的区分性特征 |
|
||||
| **3D 地球** | 基于 Three.js 的 TLS 流量弧线可视化，多数据源叠加，经纬线网格，国境线轮廓 |
|
||||
| **地理可视化** | PCA 散点图 + Canvas 原生渲染（无外部 JS 依赖） |
|
||||
| **LLM 集成** | OpenAI 兼容 API、LLM 自动编排分析流程（profile → 实体 → 聚类 → 特征） |
|
||||
| **Web 界面** | CSV 上传拖拽、手动分步分析向导、LLM 自动分析、配置管理、运行记录查看 |
|
||||
| **MCP 协议** | 12 个 JSON-RPC 工具，供外部 LLM 编排调用 |
|
||||
| **便携运行时** | 项目内置 Python 3.12 运行时（593MB），零系统依赖，`run.bat` 双击即用 |
|
||||
|
||||
### 1.3 技术栈
|
||||
|
||||
| 组件 | 技术选型 |
|
||||
|------|---------|
|
||||
| **后端框架** | Django 4.2 (SQLite WAL) |
|
||||
| **数据处理** | Polars 1.42.1 (LazyFrame 延迟计算 + streaming) |
|
||||
| **机器学习** | scikit-learn (StandardScaler, HDBSCAN, KMeans, PCA) |
|
||||
| **MCP 协议** | Python MCP SDK (stdio 传输) |
|
||||
| **前端 3D** | Three.js (603KB, 离线) |
|
||||
| **前端图表** | 纯 Canvas 2D API 原生绘制 |
|
||||
| **配置** | Pydantic + YAML |
|
||||
| **LLM 接口** | OpenAI chat/completions 兼容 API |
|
||||
| **运行时** | 嵌入式 Python 3.12 (Win7 兼容版 adang1345) |
|
||||
|
||||
---
|
||||
|
||||
## 2. 快速开始
|
||||
|
||||
### 2.1 用户模式（无需预装 Python）
|
||||
|
||||
```bat
|
||||
1. 解压项目文件夹到任意目录
|
||||
2. 双击 run.bat
|
||||
3. 等待命令行显示 "Starting TianXuan..."
|
||||
4. 浏览器自动打开 http://127.0.0.1:8000/
|
||||
5. 点击「上传数据」→ 选择 CSV 文件 → 上传 → 等待预处理完成
|
||||
6. 进入「手动分析」→ 选择数据集 → 配置参数 → 运行分析
|
||||
7. 查看结果：聚类概览、簇详情、实体画像、地球分布图
|
||||
```
|
||||
|
||||
### 2.2 开发者模式
|
||||
|
||||
```bash
|
||||
# 初始化数据库
|
||||
runtime\python\python.exe manage.py migrate
|
||||
|
||||
# 生成测试数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# 启动开发服务器
|
||||
runtime\python\python.exe manage.py runserver
|
||||
|
||||
# 浏览器打开 http://127.0.0.1:8000/
|
||||
```
|
||||
|
||||
### 2.3 脚本一览
|
||||
|
||||
| 脚本 | 功能 |
|
||||
| 模块 | 说明 |
|
||||
|------|------|
|
||||
| `run.bat` | 启动 Django Web 界面（:8000），自动打开浏览器 |
|
||||
| `shell.bat` | 打开 Django Shell（Python 交互式环境） |
|
||||
|
||||
### 2.4 一键分析管道
|
||||
|
||||
```bash
|
||||
# 加载 CSV → 自动检测实体 → 聚合 → 聚类 → 特征提取 → 写入数据库
|
||||
runtime\python\python.exe manage.py run_pipeline "data/input.csv"
|
||||
|
||||
# 指定聚类算法
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --algo kmeans
|
||||
|
||||
# 手动指定实体列（跳过自动检测）
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --entity-col src_ip
|
||||
|
||||
# 导出结果
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --output ./results/
|
||||
```
|
||||
|
||||
输出示例：
|
||||
```
|
||||
[1/5] 加载 CSV: data/test_flows.csv → 500 rows, 1 files, 0.2 MB
|
||||
[2/5] 检测实体列 → 自动检测: src_ip (共 6 候选)
|
||||
[3/5] 实体聚合 → 50 个实体, 18 维特征
|
||||
[4/5] 聚类 (hdbscan) → 2 个簇, 噪声比 0.24, Silhouette: 0.2557
|
||||
[5/5] 特征提取 → 30 个特征已保存到数据库
|
||||
*** 分析完成! Run ID: #1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 用户指南
|
||||
|
||||
### 3.1 上传数据
|
||||
|
||||
1. 点击导航栏「上传数据」
|
||||
2. 拖拽 CSV 文件到虚线区域，或点击选择文件
|
||||
3. 支持多文件上传，自动拼接相同 schema 的文件
|
||||
4. 支持 `schema_strict=false` 模式（默认）：不同列名的文件会自动合并，缺失列填 null
|
||||
5. 支持 `.zip` 压缩包自动解压
|
||||
6. 上传后自动在后台进行预处理（加载 → 检测实体列 → 聚合），完成后状态变为 `ready`
|
||||
7. 进度条实时显示处理状态
|
||||
|
||||
**数据格式要求**：
|
||||
- 每行一条 TLS 流记录
|
||||
- 建议包含列：`src_ip`、`dst_ip`、`proto`、`bytes_sent`、`bytes_rev`、`duration`、`packets` 等
|
||||
- 可选地理列：`latitude` / `lat` / `y` 和 `longitude` / `lon` / `lng` / `x`（自动识别）
|
||||
- 可选时间列：`timestamp` / `ts` / `time`（自动识别）
|
||||
|
||||
### 3.2 手动分析
|
||||
|
||||
3 步向导完成分析：
|
||||
|
||||
**Step 1 - 选择数据集**：从已预处理完成的数据集中选择一个（显示行数、已检测的实体列）
|
||||
|
||||
**Step 2 - 配置参数**：
|
||||
- 聚类算法：HDBSCAN（自动确定簇数，推荐）/ KMeans（需预设 k 值）
|
||||
- 最小簇大小（min_cluster_size）：默认 5，数据不足时自动降级
|
||||
|
||||
**Step 3 - 确认运行**：回顾配置，点击「开始分析」。后台自动完成聚类 + 特征提取，完成后跳转到结果页。
|
||||
|
||||
### 3.3 LLM 自动分析
|
||||
|
||||
需要先在「配置」页面设置 LLM 的 base_url 和 API Key：
|
||||
|
||||
1. 进入「LLM 分析」页面
|
||||
2. 选择一个已预处理的数据集
|
||||
3. 点击「开始自动分析」
|
||||
4. 后端 LLM 编排器会自动调用 profile → build_entity_profiles → run_clustering → extract_features
|
||||
5. 进度条实时显示当前步骤
|
||||
|
||||
### 3.4 查看结果
|
||||
|
||||
#### 运行详情页
|
||||
- 总流数、实体数、簇数 摘要卡片
|
||||
- 簇列表：每个簇的大小、占比、Silhouette 分数
|
||||
|
||||
#### 聚类概览页
|
||||
- **PCA 散点图**：实体在二维主成分空间中的分布，颜色区分聚类
|
||||
- **地理分布图**：实体在地理坐标上的分布（需 CSV 包含经纬度列），支持缺失值
|
||||
- 每个簇的详情卡片：Top 5 区分特征
|
||||
|
||||
#### 簇详情页
|
||||
- 特征表：每列的均值、标准差、中位数、区分度分数（Top 50）
|
||||
- 实体列表：该簇包含的实体（Top 50），可点击查看详情
|
||||
|
||||
#### 实体画像页
|
||||
- 实体标识、所属簇
|
||||
- 聚合特征值和相对簇均值的偏离（Z-score）
|
||||
- 偏差显著的特征以颜色高亮（|Z| > 2）
|
||||
|
||||
#### 3D 地球页
|
||||
- Three.js 3D 地球，TLS 流量弧线连接源目 IP
|
||||
- 弧线颜色标识 TLS 版本（TLSv1.3 蓝 / v1.2 粉 / 其他青）
|
||||
- 鼠标拖拽旋转、滚轮缩放
|
||||
- 经纬线网格 + 国境线轮廓
|
||||
- 多数据源叠加复选框
|
||||
- 脉冲光点动画（背面剔除）
|
||||
|
||||
---
|
||||
|
||||
## 4. 页面功能说明
|
||||
|
||||
| 页面 | 路由 | 功能 |
|
||||
|------|------|------|
|
||||
| **首页** | `/` | 最近运行记录概览 |
|
||||
| **上传数据** | `/upload/` | 拖拽上传 CSV，自动预处理 |
|
||||
| **手动分析** | `/analyze/manual/` | 3 步向导：选数据→设参数→运行 |
|
||||
| **LLM 分析** | `/analyze/auto/` | LLM 自动编排分析流程 |
|
||||
| **运行记录** | `/runs/` | 所有分析运行列表 |
|
||||
| **运行详情** | `/runs/<id>/` | 单次运行摘要 + 簇列表 |
|
||||
| **聚类概览** | `/clusters/<run_id>/` | PCA 散点图 + 每簇区分特征 |
|
||||
| **簇详情** | `/clusters/<run_id>/<label>/` | 特征表 + 实体列表 |
|
||||
| **实体画像** | `/entities/<id>/` | 单实体特征 + 簇偏离 |
|
||||
| **3D 地球** | `/globe/` | Three.js 流量弧线可视化 |
|
||||
| **配置** | `/config/` | 系统配置编辑 + LLM 测试 |
|
||||
| **日志** | `/logs/` | 运行日志查看 |
|
||||
|
||||
---
|
||||
|
||||
## 5. CLI 管道
|
||||
|
||||
```bash
|
||||
# 完整管道（HDBSCAN 默认）
|
||||
runtime\python\python.exe manage.py run_pipeline data/complex_test.csv
|
||||
|
||||
# 指定 KMeans
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --algo kmeans
|
||||
|
||||
# 手动指定实体列
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --entity-col src_ip
|
||||
|
||||
# 设置子网聚合掩码
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --subnet-masks 24 28
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. MCP 工具文档
|
||||
|
||||
### 6.1 工具总览
|
||||
|
||||
系统通过 MCP (Model Context Protocol) 暴露 12 个工具，支持 LLM 以 JSON-RPC 方式编排调用。
|
||||
|
||||
| # | 工具名 | 功能 | 必需参数 |
|
||||
|---|--------|------|----------|
|
||||
| 1 | `load_data` | 加载 CSV 文件（支持 glob、递归、schema 容错） | `csv_glob` |
|
||||
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
|
||||
| 3 | `filter_data` | 按条件过滤（12 操作符 + AND/OR） | `dataset_id`, `filters` |
|
||||
| 4 | `preprocess_data` | 预处理（标准化/编码/填充） | `dataset_id`, `columns` |
|
||||
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans，带质量评估） | `dataset_id`, `cluster_columns` |
|
||||
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH） | `cluster_result_id`, `dataset_id` |
|
||||
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA） | `dataset_id`, `cluster_result_id` |
|
||||
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON） | `result_id`, `output_path` |
|
||||
| 9 | `list_datasets` | 列出所有活跃数据集 | 无 |
|
||||
| 10 | `drop_dataset` | 删除数据集释放内存 | `dataset_id` |
|
||||
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据） | `dataset_id` |
|
||||
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
|
||||
|
||||
### 6.2 响应约定
|
||||
|
||||
- 成功时返回 JSON 对象，含 `truncated: false`
|
||||
- 超过阈值时自动截断，设置 `truncated: true` + `omitted_columns`
|
||||
- 错误时返回 `{ "error": "描述", "truncated": false }`
|
||||
- 所有响应通过 `TextContent` 包装返回
|
||||
|
||||
### 6.3 JSON-RPC 调用示例
|
||||
|
||||
```
|
||||
→ {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"load_data","arguments":{"csv_glob":"data/*.csv"}}}
|
||||
← {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"{\"dataset_id\":\"ds_...\",\"row_count\":500}"}]}}
|
||||
|
||||
→ {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"build_entity_profiles","arguments":{"dataset_id":"ds_...","auto_detect":true}}}
|
||||
← {"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"{\"entity_dataset_id\":\"entity_...\",\"entity_count\":50,\"entity_column_used\":\"src_ip\"}"}]}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 测试
|
||||
|
||||
```bash
|
||||
# 运行全部测试
|
||||
runtime\python\python.exe -m pytest tests -q
|
||||
|
||||
# 运行特定测试文件
|
||||
runtime\python\python.exe -m pytest tests/test_entity.py -v
|
||||
|
||||
# 运行端到端测试
|
||||
runtime\python\python.exe -m pytest tests/test_e2e.py -v
|
||||
```
|
||||
|
||||
### 生成测试数据
|
||||
```bash
|
||||
# 简单数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# Globe 模式（IP 在真实 GeoIP 范围内，弧线可见）
|
||||
runtime\python\python.exe scripts\gen_test_data.py --globe
|
||||
|
||||
# 复杂数据（5000 行 24 列含 55% 缺失）
|
||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
|
||||
```
|
||||
|
||||
### 列结构调查
|
||||
```bash
|
||||
# 统计各 CSV 文件的列名、类型、分布
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 项目结构
|
||||
|
||||
```
|
||||
tianxuan/
|
||||
├── tianxuan/ # Django 项目配置
|
||||
│ ├── settings.py # Django 配置（数据库、中间件）
|
||||
│ ├── urls.py # 根路由
|
||||
│ └── llm_orchestrator.py # LLM 编排后端（tool_calls 循环）
|
||||
│
|
||||
├── analysis/ # 核心分析模块（Django app）
|
||||
│ ├── data_loader.py # CSV 加载、BOM 检测、schema 容错、递归 glob
|
||||
│ ├── data_profiler.py # 数据集概要统计与相关性矩阵
|
||||
│ ├── entity_detector.py # 实体列自动检测（关键词 + unique_ratio 打分）
|
||||
│ ├── entity_aggregator.py # 实体聚合（流→实体画像），含 IP 子网聚合
|
||||
│ ├── type_classifier.py # 值优先类型检测（MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON）
|
||||
│ ├── data_validator.py # 列校验（缺失率/异常值/IP有效性）
|
||||
│ ├── geoip.py # GeoIP 经纬度查询
|
||||
│ ├── session_store.py # 线程安全内存会话存储（Singleton + RLock）
|
||||
│ ├── tool_registry.py # 12 个 MCP 工具定义 + 异步处理函数
|
||||
│ ├── mcp_server.py # MCP stdio 服务器
|
||||
│ ├── views.py # 所有 Django 视图
|
||||
│ ├── urls.py # 16 条 URL 路由
|
||||
│ ├── models.py # ORM 模型
|
||||
│ └── management/commands/
|
||||
│ ├── start_mcp.py # manage.py start_mcp
|
||||
│ └── run_pipeline.py # manage.py run_pipeline（一键 CLI 管道）
|
||||
│
|
||||
├── config/
|
||||
│ ├── config.yaml # 配置文件（自动生成，带注释）
|
||||
│ └── loader.py # Pydantic 配置加载器
|
||||
│
|
||||
├── templates/tianxuan/ # 11 个页面模板
|
||||
├── static/tianxuan/ # Three.js + 地球贴图 + 国境线
|
||||
├── runtime/python/ # 便携 Python 3.12 运行时（593MB）
|
||||
├── tests/ # 53 个测试 (pytest)
|
||||
├── scripts/ # 工具脚本
|
||||
├── docs/ # 文档
|
||||
├── run.bat # 用户启动脚本（双击即用）
|
||||
└── manage.py # Django 管理入口
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 常见问题
|
||||
|
||||
### Q1: 双击 run.bat 后浏览器没打开？
|
||||
A: 手动打开浏览器访问 `http://127.0.0.1:8000/`。如果也无法访问，检查命令行是否有报错。
|
||||
|
||||
### Q2: 压缩时提示 `db.sqlite3-wal 被占用`？
|
||||
A: 有残留的 Python 进程持有 SQLite 连接。运行以下命令后重试：
|
||||
```powershell
|
||||
Get-Process python -ErrorAction SilentlyContinue | Stop-Process -Force
|
||||
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
### Q3: 地图不显示？
|
||||
A: CSV 需包含 lat/lon 列（列名含 `lat`/`latitude`/`lon`/`longitude`/`lng` 自动识别），缺失值不显示。页面会提示跳过的数量。
|
||||
|
||||
### Q4: 上传 CSV 后显示 "处理失败"？
|
||||
A: 在运行详情页查看错误消息。常见原因：
|
||||
- CSV 编码不是 UTF-8
|
||||
- 所有列都是非数值类型（无法聚类，返回 0 簇）
|
||||
- 数据量太小（< 3 条实体记录，HDBSCAN 自动跳过）
|
||||
|
||||
### Q5: 如何配置 LLM？
|
||||
A: 导航到「配置」页面，填写 base_url 和 API Key，点击「测试连通性」验证。支持 OpenAI 兼容的任何 API。
|
||||
|
||||
### Q6: 支持哪些 CSV 编码？
|
||||
A: 自动检测 UTF-8 BOM、UTF-16LE BOM、UTF-16BE BOM。无 BOM 时默认 UTF-8。中文路径完全支持。
|
||||
|
||||
### Q7: 不同 CSV 文件的列名不一致怎么办？
|
||||
A: 默认 `schema_strict=false`，系统自动取所有文件的列名并集，缺失列填 null。如需严格模式，在配置中将 `schema_strict` 改为 `true`。
|
||||
|
||||
### Q8: 项目太大，如何缩小？
|
||||
```powershell
|
||||
Remove-Item -Recurse *.pyc, __pycache__ -Force
|
||||
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. 增量更新（部署到离线机）
|
||||
|
||||
天璇支持**增量更新**，只需将变更代码打包成小 zip 包，复制到离线机双击 `update.bat` 即可完成更新，无需每次传输整个项目（1.5GB）。
|
||||
|
||||
### 10.1 首次部署
|
||||
|
||||
在离线机上第一次部署时，需要完成数据库迁移：
|
||||
| **CSV 上传** | 多文件拖拽、BOM 自动检测、schema 容错、ZIP 自动解压 |
|
||||
| **手动分析** | 工作流构建器：添加/排序/保存工具链，串行执行 |
|
||||
| **LLM 自动分析** | OpenAI 兼容 API，自动编排 profile→entity→cluster→feature |
|
||||
| **聚类可视化** | HDBSCAN/KMeans，Silhouette/DB 评估，Canvas 散点图 + 地理分布 |
|
||||
| **3D 地球** | Three.js 流量弧线，TLS 版本着色，经纬线+国境线 |
|
||||
| **降维** | UMAP + TruncatedSVD 用于高维数据降维与可视化 |
|
||||
| **实体画像** | 聚合特征 + Z-Score 簇偏离量，自定义实体列/IP子网掩码 |
|
||||
| **简易分析** | 独立模块: 上传 CSV → 按目标筛选 → 地理/IP子网聚类 → Leaflet 地图 |
|
||||
| **MCP 协议** | 27 个 JSON-RPC 工具供外部 LLM 编排调用 |
|
||||
| **增量更新** | update.bat + rollback.bat，用户数据/代码分离 |
|
||||
|
||||
## 技术栈
|
||||
|
||||
| 组件 | 选型 |
|
||||
|------|------|
|
||||
| 后端 | Django 4.2 + SQLite WAL |
|
||||
| 数据处理 | Polars 1.42 (LazyFrame + streaming) |
|
||||
| 机器学习 | scikit-learn (HDBSCAN, KMeans, StandardScaler), UMAP, TruncatedSVD |
|
||||
| 前端 3D | Three.js (603KB, 离线, 地球贴图 1.4MB) |
|
||||
| 前端图表 | 纯 Canvas 2D / Leaflet |
|
||||
| 配置管理 | Pydantic + YAML |
|
||||
| LLM 协议 | MCP SDK (stdio) + OpenAI 兼容 API |
|
||||
| 运行时 | 内嵌 Python 3.12 (593MB, Win7 兼容版) |
|
||||
|
||||
## 安装
|
||||
|
||||
项目自包含，无需预装 Python。解压后即可使用。
|
||||
|
||||
```bat
|
||||
:: 确保数据库在 %APPDATA%/TianXuan/ 下
|
||||
runtime\python\python.exe scripts\migrate_db_to_appdata.py
|
||||
|
||||
:: 初始化数据库表
|
||||
:: 首次部署 — 初始化数据库
|
||||
runtime\python\python.exe manage.py migrate
|
||||
|
||||
:: 导入 TLS 引用数据
|
||||
runtime\python\python.exe manage.py import_tlsdb
|
||||
```
|
||||
|
||||
> 如果之前已在使用天璇，`migrate_db_to_appdata.py` 会自动将已有 `db.sqlite3` 复制到新位置。
|
||||
## 使用方法
|
||||
|
||||
### 10.2 开发机：构建更新包
|
||||
```bat
|
||||
:: 双击启动 Web 界面 (http://127.0.0.1:8000/)
|
||||
run.bat
|
||||
|
||||
在**开发机**上完成代码修改并提交后：
|
||||
:: 命令行一键管道
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv"
|
||||
|
||||
```bash
|
||||
# 确保工作树干净（无未提交变更）
|
||||
git status
|
||||
|
||||
# 构建增量更新包
|
||||
python scripts/build_update.py
|
||||
:: 简易分析模块入口: http://127.0.0.1:8000/simple/
|
||||
```
|
||||
|
||||
脚本会自动：
|
||||
1. 检测当前 git tag 与最近一次 tag 之间的文件变更
|
||||
2. 计算每个文件的 SHA256 校验和
|
||||
3. 打包为 `tianxuan_update_<旧版本>-<新版本>.zip`
|
||||
4. 更新 `VERSION` 文件
|
||||
5. 如果检测到依赖变更，会输出提示信息
|
||||
**Web 工作流:** 上传 CSV → 手动分析（工作流构建器）或 LLM 自动分析 → 检视聚类概览/簇详情/实体画像/3D 地球
|
||||
|
||||
**输出示例**：
|
||||
```
|
||||
==================================================
|
||||
天璇 增量更新包构建工具
|
||||
==================================================
|
||||
**CLI 管道:** 加载 CSV → 实体检测 → 聚合 → 聚类 → 特征提取 → 写入数据库
|
||||
|
||||
当前 VERSION: v1.0.0
|
||||
最新 git tag: v1.0
|
||||
|
||||
变更文件: 5 个修改/新增, 1 个删除
|
||||
|
||||
==================================================
|
||||
✅ 增量更新包构建完成！
|
||||
包: tianxuan_update_v1.0-v1.1.zip
|
||||
大小: 0.05 MB
|
||||
从: v1.0
|
||||
到: v1.1
|
||||
文件数: 5
|
||||
删除: 1
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 10.3 离线机：应用更新
|
||||
|
||||
在**离线机**上：
|
||||
|
||||
1. 将 `tianxuan_update_*.zip` 复制到天璇项目目录（与 `run.bat` 同目录）
|
||||
2. **双击 `update.bat`**（或拖拽 zip 到 `update.bat` 上）
|
||||
3. 等待更新完成
|
||||
4. 关闭窗口，重新双击 `run.bat` 启动新版本
|
||||
|
||||
更新脚本会自动：
|
||||
1. ✅ 验证更新包完整性（SHA256 校验）
|
||||
2. ✅ 备份当前配置（`config/config.yaml`）
|
||||
3. ✅ 删除废弃文件
|
||||
4. ✅ 覆盖更新文件
|
||||
5. ✅ 智能合并配置（保留你的设置，仅增加新配置项）
|
||||
6. ✅ 清理 Python 缓存
|
||||
7. ✅ 运行数据库迁移
|
||||
8. ✅ 更新版本号
|
||||
|
||||
### 10.4 回滚
|
||||
|
||||
如果更新后出现问题：
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
双击 rollback.bat
|
||||
→ 输入 y 确认回滚
|
||||
→ 等待恢复完成
|
||||
→ 重新双击 run.bat 启动旧版本
|
||||
天璇/
|
||||
├── analysis/ 核心分析模块 (Django app)
|
||||
│ ├── data_loader.py CSV 加载、BOM/编码检测、schema 容错
|
||||
│ ├── entity_detector.py 实体列自动检测 (关键词+唯一值比率打分)
|
||||
│ ├── entity_aggregator.py 按实体聚合特征 (含 IP 子网)
|
||||
│ ├── tool_registry.py 27 个 MCP 工具注册 + 处理函数
|
||||
│ ├── mcp_server.py MCP stdio 服务器
|
||||
│ ├── type_classifier.py 值优先类型检测 (MAC/端口/IPv4/URL/HEX/经纬度)
|
||||
│ ├── ip_clustering.py IP 子网聚类与转换工具
|
||||
│ ├── data_profiler.py 数据集概要 + 相关性矩阵
|
||||
│ ├── geoip.py GeoIP 经纬度查询
|
||||
│ ├── session_store.py 线程安全内存会话存储
|
||||
│ └── views.py Django 视图 (20+ 路由)
|
||||
├── simple_analysis/ 简易分析模块 (独立上传→筛选→聚类→地图)
|
||||
├── tianxuan/ Django 项目配置 + LLM 编排器
|
||||
├── config/ 配置 (config.yaml + Pydantic 加载器)
|
||||
├── templates/ 页面模板 (analysis/ tianxuan/ base.html)
|
||||
├── static/ 离线静态资源 (Three.js 地球贴图 国境线)
|
||||
├── runtime/python/ 内嵌 Python 3.12 运行时
|
||||
├── scripts/ 工具脚本 (测试数据生成/跨机诊断/增量更新)
|
||||
├── tests/ 单元 + 集成测试
|
||||
├── docs/ 工作流文档
|
||||
├── run.bat 用户启动入口
|
||||
├── update.bat 增量更新
|
||||
└── rollback.bat 更新回滚
|
||||
```
|
||||
|
||||
回滚会从 `%APPDATA%/TianXuan/backups/` 恢复最近一次备份。
|
||||
## 目标设备
|
||||
|
||||
### 10.5 依赖变更时的特殊处理
|
||||
|
||||
当 `pyproject.toml` 或 `requirements.txt` 变更时（即添加/更新了 Python 依赖），构建脚本会输出警告：
|
||||
|
||||
```
|
||||
⚠️ 依赖已变更！请手动将 runtime/ 目录同步到离线机
|
||||
```
|
||||
|
||||
此时需要额外步骤：
|
||||
|
||||
1. 在开发机上将 `runtime/` 目录**完整压缩**（约 645MB）
|
||||
2. 复制到离线机，覆盖 `runtime/` 目录
|
||||
3. 或者使用差异更新：仅覆盖变更的包目录（`runtime/python/Lib/site-packages/`）
|
||||
|
||||
> 依赖变更是低频操作（通常只在添加新功能时需要），所以这不会影响日常的代码增量更新。
|
||||
|
||||
### 10.6 文件说明
|
||||
|
||||
| 文件 | 用途 | 需要 git 跟踪？ |
|
||||
|------|------|----------------|
|
||||
| `VERSION` | 记录当前版本号 | ✅ 是 |
|
||||
| `scripts/build_update.py` | 开发机：构建增量包 | ✅ 是 |
|
||||
| `scripts/apply_update.py` | 离线机：应用更新 | ✅ 是 |
|
||||
| `scripts/merge_config.py` | 合并用户配置与源码配置 | ✅ 是 |
|
||||
| `scripts/migrate_db_to_appdata.py` | 首次迁移数据库到 APPDATA | ✅ 是 |
|
||||
| `scripts/version_utils.py` | 版本工具函数 | ✅ 是 |
|
||||
| `update.bat` | 离线机双击入口 | ✅ 是 |
|
||||
| `rollback.bat` | 回滚入口 | ✅ 是 |
|
||||
| `analysis/appdata.py` | APPDATA 路径共享模块 | ✅ 是 |
|
||||
| `.update_cache/runtime_snapshot.json` | 运行时依赖快照（自动生成）| ❌ 否 |
|
||||
| `RUNTIME_CHANGED.txt` | 依赖变更提示（自动生成）| ❌ 否 |
|
||||
|
||||
### 10.7 目录结构变化
|
||||
|
||||
更新系统将用户数据与项目代码分离：
|
||||
|
||||
```
|
||||
项目目录（可安全覆盖） 用户数据目录（永不覆盖）
|
||||
天璇/ %APPDATA%/TianXuan/
|
||||
├── analysis/ ← 代码 ├── db.sqlite3 ← 数据库
|
||||
├── config/config.yaml ← 配置 ├── data/uploads/ ← 上传的 CSV
|
||||
├── runtime/ ← 运行时 ├── .session_store.json
|
||||
├── templates/ ← 模板 └── backups/ ← 更新备份
|
||||
├── static/ ← 静态资源
|
||||
├── scripts/ ← 工具脚本
|
||||
├── update.bat ← 更新入口
|
||||
└── VERSION ← 版本号
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
| 条件 | 规格 |
|
||||
|------|------|
|
||||
| 操作系统 | Windows 10 1902+ |
|
||||
| 内存 | 4 GB |
|
||||
| GPU | 无独立显卡（核显可运行 Three.js） |
|
||||
| 磁盘 | SSD 推荐（SQLite WAL 优化） |
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
import csv, os
|
||||
os.chdir(r'E:\hjq\天璇')
|
||||
with open('TlsDB.csv', 'r', encoding='utf-8') as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
dtype = row.get('data_type', '').strip().upper()
|
||||
code = row.get('field_code', '').strip().lower()
|
||||
if 'BOOL' in dtype or code in ('cnrs', 'isrs'):
|
||||
print(f"{row['field_code']:20s} {dtype:15s} {row.get('meaning', '')[:50]}")
|
||||
Binary file not shown.
@@ -1,73 +0,0 @@
|
||||
import urllib.request, json, time, sys
|
||||
|
||||
BASE = 'http://127.0.0.1:18766'
|
||||
RUN_ID = 2 # the upload we just did
|
||||
|
||||
# Step 1: Trigger manual analysis (full pipeline: clustering + UMAP + SVD)
|
||||
print('[1] Starting manual analysis pipeline...')
|
||||
body = json.dumps({
|
||||
'run_id': int(RUN_ID),
|
||||
'algorithm': 'hdbscan',
|
||||
'min_cluster_size': 5,
|
||||
'cluster_mode': 'raw',
|
||||
}).encode('utf-8')
|
||||
|
||||
req = urllib.request.Request(f'{BASE}/analyze/run/', data=body,
|
||||
headers={'Content-Type': 'application/json'})
|
||||
try:
|
||||
r = urllib.request.urlopen(req, timeout=10)
|
||||
resp = json.loads(r.read())
|
||||
r.close()
|
||||
print(f'[1] Started: {resp}')
|
||||
except Exception as e:
|
||||
print(f'[1] FAILED: {e}')
|
||||
sys.exit(1)
|
||||
|
||||
# Step 2: Poll until completed or failed
|
||||
print('[2] Polling analysis progress...')
|
||||
for i in range(240):
|
||||
try:
|
||||
r = urllib.request.urlopen(f'{BASE}/runs/{RUN_ID}/status/', timeout=5)
|
||||
s = json.loads(r.read())
|
||||
r.close()
|
||||
except:
|
||||
time.sleep(2)
|
||||
continue
|
||||
|
||||
status = s['status']
|
||||
pct = s.get('progress_pct', 0)
|
||||
msg = s.get('progress_msg', '')
|
||||
if i % 10 == 0:
|
||||
clusters = s.get('cluster_count') or '?'
|
||||
entities = s.get('entity_count') or '?'
|
||||
print(f' [{i}] {status} ({pct}%) clusters={clusters} entities={entities} {msg[:60]}')
|
||||
|
||||
if status == 'completed':
|
||||
print(f'\n[2] DONE!')
|
||||
print(f' status: {status}')
|
||||
print(f' cluster_count: {s.get("cluster_count")}')
|
||||
print(f' entity_count: {s.get("entity_count")}')
|
||||
print(f' progress: {pct}%')
|
||||
break
|
||||
if status == 'failed':
|
||||
err = s.get('error_message', '')[:800]
|
||||
print(f'\n[2] FAILED: {err}')
|
||||
break
|
||||
time.sleep(3)
|
||||
else:
|
||||
print('[2] TIMEOUT after 12 minutes')
|
||||
|
||||
# Step 3: Verify cluster overview page
|
||||
print('[3] Checking cluster overview...')
|
||||
try:
|
||||
r = urllib.request.urlopen(f'{BASE}/clusters/{RUN_ID}/', timeout=10)
|
||||
body = r.read().decode('utf-8', errors='replace')
|
||||
r.close()
|
||||
has_scatter = 'scatter_data' in body
|
||||
has_geo = 'geo_data' in body
|
||||
has_umap = 'UMAP' in body
|
||||
print(f' scatter_data: {has_scatter}')
|
||||
print(f' geo_data: {has_geo}')
|
||||
print(f' UMAP text: {has_umap}')
|
||||
except Exception as e:
|
||||
print(f' FAILED: {e}')
|
||||
@@ -383,6 +383,79 @@ def load_from_db(
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_from_db_lazy(
|
||||
table_name: str,
|
||||
schema_overrides: dict[str, pl.DataType | str] | None = None,
|
||||
) -> Optional[pl.LazyFrame]:
|
||||
"""Load a SQLite table into a LazyFrame using Polars native DB reader.
|
||||
|
||||
Uses ``pl.read_database`` for efficient bulk reading — an order of
|
||||
magnitude faster than the manual batch-processing path in
|
||||
:func:`load_from_db`. Returns a :class:`pl.LazyFrame` so downstream
|
||||
operations remain lazy (query-plan, not materialised).
|
||||
|
||||
Args:
|
||||
table_name: SQLite table name.
|
||||
schema_overrides: Optional ``{column_name: dtype}`` dict to cast
|
||||
columns after loading. Values can be ``pl.DataType`` instances
|
||||
(e.g. ``pl.Float64``) or dtype strings (e.g. ``'Float64'``).
|
||||
|
||||
Returns:
|
||||
``None`` if the table does not exist or is empty.
|
||||
"""
|
||||
from django.conf import settings
|
||||
|
||||
db_path = settings.DATABASES['default']['NAME']
|
||||
conn = sqlite3.connect(db_path)
|
||||
|
||||
try:
|
||||
# Check table existence (fast metadata query)
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table_name,),
|
||||
)
|
||||
if cursor.fetchone() is None:
|
||||
return None
|
||||
|
||||
# Bulk-read with Polars native engine — leverages the DBAPI2
|
||||
# connection directly for efficient C-level data transfer.
|
||||
query = f'SELECT * FROM "{table_name}"'
|
||||
df = pl.read_database(connection=conn, query=query)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
if df.height == 0:
|
||||
return None
|
||||
|
||||
# Apply type overrides to restore original column types
|
||||
# (SQLite stores everything as TEXT/INTEGER/REAL/BLOB;
|
||||
# we need to cast back to Float64 / Int64 / etc.)
|
||||
if schema_overrides:
|
||||
casts = {}
|
||||
for col, dtype in schema_overrides.items():
|
||||
if col not in df.columns:
|
||||
continue
|
||||
dtype_obj = (
|
||||
dtype if isinstance(dtype, pl.DataType)
|
||||
else _str_to_polars_dtype(dtype)
|
||||
)
|
||||
if df[col].dtype != dtype_obj:
|
||||
try:
|
||||
casts[col] = pl.col(col).cast(dtype_obj)
|
||||
except Exception:
|
||||
pass # uncastable column — keep as-is
|
||||
if casts:
|
||||
df = df.with_columns(list(casts.values()))
|
||||
|
||||
rows = df.height
|
||||
cols = df.width
|
||||
logger.info(
|
||||
'[LOAD_FROM_DB_LAZY] table=%s rows=%d cols=%d',
|
||||
table_name, rows, cols,
|
||||
)
|
||||
return df.lazy()
|
||||
|
||||
|
||||
def drop_sqlite_table(table_name: str) -> None:
|
||||
"""Drop a SQLite data table silently if it exists."""
|
||||
from django.conf import settings
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,28 @@
|
||||
# Generated by Django 4.2.30 on 2026-07-23 04:21
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('tianxuan_analysis', '0007_populate_display_id'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='entityprofile',
|
||||
name='embedding_z',
|
||||
field=models.FloatField(blank=True, help_text='UMAP-3D Z coordinate', null=True),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name='entityprofile',
|
||||
name='embedding_x',
|
||||
field=models.FloatField(blank=True, help_text='UMAP-2D X coordinate', null=True),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name='entityprofile',
|
||||
name='embedding_y',
|
||||
field=models.FloatField(blank=True, help_text='UMAP-2D Y coordinate', null=True),
|
||||
),
|
||||
]
|
||||
@@ -96,6 +96,7 @@ class EntityProfile(models.Model):
|
||||
help_text="Aggregated entity features as dict")
|
||||
embedding_x = models.FloatField(null=True, blank=True, help_text="UMAP-2D X coordinate")
|
||||
embedding_y = models.FloatField(null=True, blank=True, help_text="UMAP-2D Y coordinate")
|
||||
embedding_z = models.FloatField(null=True, blank=True, help_text="UMAP-3D Z coordinate")
|
||||
|
||||
class Meta:
|
||||
unique_together = ['run', 'entity_value']
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
"""
|
||||
Template-based natural language feature descriptions.
|
||||
|
||||
Uses value-range → text mappings to generate Chinese natural language
|
||||
descriptions of cluster features. Consumed by cluster_overview.html and
|
||||
entity_profile.html for human-readable cluster summaries.
|
||||
|
||||
Data sources
|
||||
------------
|
||||
- ``analysis.tls_ref.get_tls_ref_dict()`` : ``field_code → {meaning, data_type}``
|
||||
- ``ClusterFeature`` model rows : feature_name, mean, std, distinguishing_score
|
||||
|
||||
Usage::
|
||||
|
||||
from analysis.nl_describe import describe_feature, describe_cluster
|
||||
|
||||
text = describe_feature("8ack", 720.5, std_dev=2.1)
|
||||
# "中等数据包 (720 bytes)，比全局均值高2.1个标准差"
|
||||
|
||||
summary = describe_cluster(features, top_n=5)
|
||||
# "此簇特征: 低延迟(23ms), 大数据包(1520 bytes), ..."
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional, Sequence
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Category mapping: field_code → semantic category
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
CATEGORY_EXACT: dict[str, str] = {
|
||||
# ── latency ──
|
||||
"4dur": "latency",
|
||||
"2tmo": "latency",
|
||||
"8dbd": "latency",
|
||||
# ── payload bytes ──
|
||||
"8ack": "packet_bytes",
|
||||
"8byt": "packet_bytes",
|
||||
# ── packet count ──
|
||||
"8pak": "packet_count",
|
||||
"8ppk": "packet_count",
|
||||
# ── port ──
|
||||
":prs": "port",
|
||||
":prd": "port",
|
||||
# ── TLS version ──
|
||||
"0ver": "tls_version",
|
||||
# ── key size ──
|
||||
"4ksz": "key_size",
|
||||
# ── cipher code ──
|
||||
"0cph": "cipher_code",
|
||||
# ── curve code ──
|
||||
"0crv": "curve_code",
|
||||
# ── session / offset ──
|
||||
"8seq": "session_offset",
|
||||
"8ses": "session_offset",
|
||||
"8did": "session_offset",
|
||||
# ── IP protocol ──
|
||||
"1ipp": "ip_protocol",
|
||||
# ── TLS random ──
|
||||
"0rnt": "tls_random_time",
|
||||
"0rnd": "tls_random",
|
||||
# ── database refs ──
|
||||
"4dbn": "database_ref",
|
||||
"4srs": "device_ref",
|
||||
# ── recovery flags ──
|
||||
"cnrs": "recovery",
|
||||
"isrs": "recovery",
|
||||
}
|
||||
|
||||
# Substring-based matching (checked after exact match fails)
|
||||
CATEGORY_SUBSTR: list[tuple[str, str]] = [
|
||||
("latd", "latitude"),
|
||||
("lond", "longitude"),
|
||||
("orgn", "organization"),
|
||||
("ispn", "isp"),
|
||||
("anon", "anonymity"),
|
||||
("doma", "domain"),
|
||||
("city", "city"),
|
||||
("_lat", "latitude"),
|
||||
("_lon", "longitude"),
|
||||
("latitude", "latitude"),
|
||||
("longitude", "longitude"),
|
||||
("duration", "latency"),
|
||||
("delay", "latency"),
|
||||
("bytes", "packet_bytes"),
|
||||
("packets", "packet_count"),
|
||||
("count", "count"),
|
||||
("ratio", "ratio"),
|
||||
("tls_ver", "tls_version"),
|
||||
("version", "tls_version"),
|
||||
]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TLS version code → human description
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TLS_VERSION_MAP: dict[int, str] = {
|
||||
0x0000: "SSL 2.0",
|
||||
0x0300: "SSL 3.0",
|
||||
0x0301: "TLS 1.0",
|
||||
0x0302: "TLS 1.1",
|
||||
0x0303: "TLS 1.2",
|
||||
0x0304: "TLS 1.3",
|
||||
0x7F00: "TLS 1.3 (draft)",
|
||||
0x7F12: "TLS 1.3 (draft 18)",
|
||||
0xFEFF: "DTLS 1.0",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Templates: category → [(predicate, template_string), ...]
|
||||
#
|
||||
# Template variables available:
|
||||
# {v} — value (formatted to 1 decimal unless overridden)
|
||||
# {v:.0f} — value with custom format
|
||||
# {sv} — signed value (e.g. "+2.1" for positive zscore)
|
||||
# {av} — absolute value (e.g. "2.1")
|
||||
# {z:.1f} — z-score signed (preserved in deviation suffix)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TEMPLATES: dict[str, list[tuple]] = {
|
||||
"latency": [
|
||||
(lambda v: v > 1000, "极高延迟 ({v:.0f}ms)"),
|
||||
(lambda v: v > 500, "高延迟 ({v:.0f}ms)"),
|
||||
(lambda v: v > 200, "中等延迟 ({v:.0f}ms)"),
|
||||
(lambda v: v > 50, "较低延迟 ({v:.0f}ms)"),
|
||||
(lambda v: True, "低延迟 ({v:.0f}ms)"),
|
||||
],
|
||||
"packet_bytes": [
|
||||
(lambda v: v > 1500, "大数据包 ({v:.0f} bytes, 超过MTU)"),
|
||||
(lambda v: v > 1000, "较大数据包 ({v:.0f} bytes)"),
|
||||
(lambda v: v > 400, "中等数据包 ({v:.0f} bytes)"),
|
||||
(lambda v: True, "小数据包 ({v:.0f} bytes)"),
|
||||
],
|
||||
"packet_count": [
|
||||
(lambda v: v > 200, "大量数据包 ({v:.0f} 个)"),
|
||||
(lambda v: v > 50, "中等数据包数 ({v:.0f} 个)"),
|
||||
(lambda v: True, "少量数据包 ({v:.0f} 个)"),
|
||||
],
|
||||
"port": [
|
||||
(lambda v: v <= 1024, "系统端口 ({v:.0f})"),
|
||||
(lambda v: v <= 49151, "注册端口 ({v:.0f})"),
|
||||
(lambda v: True, "动态/私有端口 ({v:.0f})"),
|
||||
],
|
||||
"tls_version": [
|
||||
(lambda v: 0x0304 <= v <= 0x0304, "TLS 1.3 (最新)"),
|
||||
(lambda v: 0x0303 <= v < 0x0304, "TLS 1.2 (主流)"),
|
||||
(lambda v: 0x0302 <= v < 0x0303, "TLS 1.1 (已弃用)"),
|
||||
(lambda v: 0x0301 <= v < 0x0302, "TLS 1.0 (过时)"),
|
||||
(lambda v: 0x0300 <= v < 0x0301, "SSL 3.0 (不安全)"),
|
||||
(lambda v: v < 0x0300, "SSL 2.0 (极不安全)"),
|
||||
(lambda v: v > 0x0304, "DTLS/实验性TLS (0x{v:.0X})"),
|
||||
(lambda v: True, "未知TLS版本 (0x{v:.0X})"),
|
||||
],
|
||||
"key_size": [
|
||||
(lambda v: v >= 384, "高强度密钥 ({v:.0f} bits)"),
|
||||
(lambda v: v >= 256, "强密钥 ({v:.0f} bits)"),
|
||||
(lambda v: v >= 128, "标准密钥 ({v:.0f} bits)"),
|
||||
(lambda v: v >= 64, "弱密钥 ({v:.0f} bits)"),
|
||||
(lambda v: True, "极弱密钥 ({v:.0f} bits)"),
|
||||
],
|
||||
"ip_protocol": [
|
||||
(lambda v: abs(v - 6) < 0.5, "TCP 协议 ({v:.0f})"),
|
||||
(lambda v: abs(v - 17) < 0.5, "UDP 协议 ({v:.0f})"),
|
||||
(lambda v: abs(v - 1) < 0.5, "ICMP 协议 ({v:.0f})"),
|
||||
(lambda v: abs(v - 2) < 0.5, "IGMP 协议 ({v:.0f})"),
|
||||
(lambda v: True, "IP 协议 #{v:.0f}"),
|
||||
],
|
||||
"session_offset": [
|
||||
(lambda v: v > 10000, "大型会话 (偏移 {v:.0f})"),
|
||||
(lambda v: v > 1000, "中型会话 (偏移 {v:.0f})"),
|
||||
(lambda v: True, "小型会话 (偏移 {v:.0f})"),
|
||||
],
|
||||
"latitude": [
|
||||
(lambda v: v > 60, "高纬度地区 ({v:.1f}°)"),
|
||||
(lambda v: v > 23.5, "中纬度地区 — 北温带 ({v:.1f}°)"),
|
||||
(lambda v: v >= -23.5, "低纬度地区 — 热带 ({v:.1f}°)"),
|
||||
(lambda v: v >= -60, "中纬度地区 — 南温带 ({v:.1f}°)"),
|
||||
(lambda v: True, "高纬度地区 — 南极 ({v:.1f}°)"),
|
||||
],
|
||||
"longitude": [
|
||||
(lambda v: v > 120 or v < -120, "远东地区 ({v:.1f}°)"),
|
||||
(lambda v: v > 60 or v < -60, "中亚/东欧地区 ({v:.1f}°)"),
|
||||
(lambda v: v > 0 or v < 0, "西欧/非洲地区 ({v:.1f}°)"),
|
||||
(lambda v: True, "本初子午线附近 ({v:.1f}°)"),
|
||||
],
|
||||
"cipher_code": [
|
||||
(lambda v: True, "加密套件代码 0x{v:.0X}"),
|
||||
],
|
||||
"curve_code": [
|
||||
(lambda v: True, "椭圆曲线代码 0x{v:.0X}"),
|
||||
],
|
||||
"tls_random_time": [
|
||||
(lambda v: True, "TLS 随机时间戳 ({v:.0f})"),
|
||||
],
|
||||
"tls_random": [
|
||||
(lambda v: True, "TLS 随机值"),
|
||||
],
|
||||
"database_ref": [
|
||||
(lambda v: True, "数据库引用 #{v:.0f}"),
|
||||
],
|
||||
"device_ref": [
|
||||
(lambda v: True, "设备标识 #{v:.0f}"),
|
||||
],
|
||||
"recovery": [
|
||||
(lambda v: v >= 0.5, "可恢复流"),
|
||||
(lambda v: True, "不可恢复流"),
|
||||
],
|
||||
"organization": [
|
||||
(lambda v: True, "组织指标 ({v:.1f})"),
|
||||
],
|
||||
"isp": [
|
||||
(lambda v: True, "ISP 指标 ({v:.1f})"),
|
||||
],
|
||||
"anonymity": [
|
||||
(lambda v: True, "匿名状态指标 ({v:.1f})"),
|
||||
],
|
||||
"domain": [
|
||||
(lambda v: True, "域名指标 ({v:.1f})"),
|
||||
],
|
||||
"city": [
|
||||
(lambda v: True, "城市指标 ({v:.1f})"),
|
||||
],
|
||||
"count": [
|
||||
(lambda v: v > 100, "计数很高 ({v:.0f})"),
|
||||
(lambda v: v > 50, "计数较高 ({v:.0f})"),
|
||||
(lambda v: True, "计数正常 ({v:.0f})"),
|
||||
],
|
||||
"ratio": [
|
||||
(lambda v: v > 0.8, "占比极高 ({v:.1%})"),
|
||||
(lambda v: v > 0.5, "占比偏高 ({v:.1%})"),
|
||||
(lambda v: v >= 0.0, "占比较低 ({v:.1%})"),
|
||||
(lambda v: True, "占比 ({v:.1%})"),
|
||||
],
|
||||
}
|
||||
|
||||
# Default template for unrecognised categories / fallback
|
||||
DEFAULT_TEMPLATES: list[tuple] = [
|
||||
(lambda v: True, "{meaning} = {v:.2f}"),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _resolve_category(feature_name: str) -> str | None:
|
||||
"""Map a ``feature_name`` (column name) to a template category string.
|
||||
|
||||
Tries exact match against ``CATEGORY_EXACT`` first, then substring
|
||||
match against ``CATEGORY_SUBSTR``. Returns ``None`` if no match.
|
||||
"""
|
||||
# Exact match
|
||||
cat = CATEGORY_EXACT.get(feature_name)
|
||||
if cat is not None:
|
||||
return cat
|
||||
|
||||
# Substring match
|
||||
lowered = feature_name.lower()
|
||||
for sub, cat_label in CATEGORY_SUBSTR:
|
||||
if sub in lowered:
|
||||
return cat_label
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _lookup_meaning(feature_name: str) -> str:
|
||||
"""Return the Chinese meaning of *feature_name* from the TLS reference DB.
|
||||
|
||||
Falls back to *feature_name* itself when no entry is found.
|
||||
"""
|
||||
try:
|
||||
from analysis.tls_ref import get_tls_ref_dict
|
||||
|
||||
ref = get_tls_ref_dict()
|
||||
entry = ref.get(feature_name)
|
||||
if entry is not None:
|
||||
return entry.get("meaning", feature_name)
|
||||
except Exception:
|
||||
pass
|
||||
return feature_name
|
||||
|
||||
|
||||
def describe_feature(
|
||||
feature_name: str,
|
||||
value: float,
|
||||
std_dev: float | None = None,
|
||||
) -> str:
|
||||
"""Generate a Chinese natural-language description for a single feature.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
feature_name : str
|
||||
Column name (e.g. ``"8ack"``, ``":ips.latd"``).
|
||||
value : float
|
||||
The feature's mean value in the current context (cluster mean).
|
||||
std_dev : float | None
|
||||
Optional Z-score magnitude (``abs(distinguishing_score)``). When
|
||||
provided and >= 1.0, appends a deviation suffix like
|
||||
"比全局均值高2.1个标准差".
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
Natural-language description string, e.g.
|
||||
``"中等数据包 (720 bytes)，比全局均值高2.1个标准差"``.
|
||||
"""
|
||||
meaning = _lookup_meaning(feature_name)
|
||||
category = _resolve_category(feature_name)
|
||||
|
||||
# Select templates
|
||||
tmpl_list = TEMPLATES.get(category) if category else None
|
||||
if tmpl_list is None:
|
||||
tmpl_list = DEFAULT_TEMPLATES
|
||||
|
||||
# Evaluate predicates
|
||||
text = ""
|
||||
for predicate, template in tmpl_list:
|
||||
try:
|
||||
if predicate(value):
|
||||
text = template.format(v=value, sv=_signed(value), av=abs(value),
|
||||
meaning=meaning)
|
||||
break
|
||||
except Exception:
|
||||
# Fall through to next template on format errors
|
||||
continue
|
||||
|
||||
if not text:
|
||||
text = f"{meaning} = {value:.2f}"
|
||||
|
||||
# Append deviation context
|
||||
if std_dev is not None and std_dev >= 1.0:
|
||||
text += f"，比全局均值高{std_dev:.1f}个标准差"
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def describe_cluster(
|
||||
features: Sequence[dict],
|
||||
top_n: int = 5,
|
||||
) -> str:
|
||||
"""Generate a 2-3 sentence natural-language summary for a cluster.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
features : list[dict]
|
||||
Each dict must include:
|
||||
|
||||
- ``feature_name`` : str
|
||||
- ``mean`` : float (cluster mean)
|
||||
- ``distinguishing_score`` : float (Z-score; may be signed)
|
||||
|
||||
Optional keys used when present:
|
||||
|
||||
- ``std`` : float (cluster std dev)
|
||||
|
||||
top_n : int
|
||||
Number of most-distinguishing features to include (default 5).
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
A short paragraph describing the cluster's distinguishing traits.
|
||||
|
||||
Example:
|
||||
``"低延迟 (23ms)，比全局均值高3.1个标准差; 大数据包 (1520 bytes)，比全局均值高2.3个标准差; 系统端口 (443)"``
|
||||
"""
|
||||
if not features:
|
||||
return "无显著区分特征"
|
||||
|
||||
# Sort by |distinguishing_score| descending
|
||||
ranked = sorted(
|
||||
features,
|
||||
key=lambda f: abs(f.get("distinguishing_score", 0.0)),
|
||||
reverse=True,
|
||||
)[:top_n]
|
||||
|
||||
sentences: list[str] = []
|
||||
for f in ranked:
|
||||
name = f.get("feature_name", "")
|
||||
mean = f.get("mean")
|
||||
score = abs(f.get("distinguishing_score", 0.0))
|
||||
|
||||
if mean is None or name == "":
|
||||
continue
|
||||
|
||||
desc = describe_feature(name, float(mean), std_dev=score if score >= 1.0 else None)
|
||||
sentences.append(desc)
|
||||
|
||||
if not sentences:
|
||||
return "无显著区分特征"
|
||||
|
||||
return "；".join(sentences) + "。"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _signed(value: float) -> str:
|
||||
"""Return a signed string representation, e.g. ``"+2.1"``."""
|
||||
return f"{value:+.1f}"
|
||||
+166
-51
@@ -46,6 +46,18 @@ def _generate_id(prefix: str = 'ds') -> str:
|
||||
return f"{prefix}_{uuid.uuid4().hex[:12]}"
|
||||
|
||||
|
||||
def _estimate_row_count(lf: pl.LazyFrame) -> int:
|
||||
"""Estimate row count from a LazyFrame (fast, streaming).
|
||||
|
||||
Uses ``select(pl.len()).collect(streaming=True)`` which scans the
|
||||
underlying source efficiently without loading all columns.
|
||||
"""
|
||||
try:
|
||||
return lf.select(pl.len()).collect(streaming=True).item()
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def _get_store_path() -> Path:
|
||||
"""Return the persistent JSON file path for dataset metadata.
|
||||
|
||||
@@ -78,6 +90,7 @@ class SessionStore:
|
||||
cls._instance = super().__new__(cls)
|
||||
cls._instance._stores: dict[str, dict[str, Any]] = {}
|
||||
cls._instance._datalock = threading.RLock()
|
||||
cls._instance._restore_done = threading.Event()
|
||||
return cls._instance
|
||||
|
||||
# ── persistence ────────────────────────────────────────────────────
|
||||
@@ -85,8 +98,9 @@ class SessionStore:
|
||||
def _save_to_disk(self) -> None:
|
||||
"""Persist current dataset metadata to JSON file.
|
||||
|
||||
Only metadata (schema, csv_glob, row_count, …) is saved — never the
|
||||
LazyFrame itself, which is an in-memory query plan.
|
||||
Only metadata (schema, row_count, file_count, sqlite_table, …) is
|
||||
saved — never the LazyFrame itself. The resulting file is typically
|
||||
under 10 KB even with many datasets.
|
||||
"""
|
||||
with self._datalock:
|
||||
serializable = {}
|
||||
@@ -94,9 +108,16 @@ class SessionStore:
|
||||
if entry.get('type') != 'dataset':
|
||||
continue
|
||||
meta = entry.get('metadata', {})
|
||||
# Store only restore-critical fields; omit runtime noise
|
||||
slim_meta = {
|
||||
k: v for k, v in meta.items()
|
||||
if k in ('row_count', 'file_count', 'sqlite_table',
|
||||
'csv_glob', 'columns', 'memory_mb',
|
||||
'schema_overrides', 'svd_components')
|
||||
}
|
||||
serializable[ds_id] = {
|
||||
'schema': entry.get('schema'),
|
||||
'metadata': meta,
|
||||
'metadata': slim_meta,
|
||||
'csv_glob': meta.get('csv_glob'),
|
||||
}
|
||||
path = _get_store_path()
|
||||
@@ -105,17 +126,15 @@ class SessionStore:
|
||||
encoding='utf-8',
|
||||
)
|
||||
|
||||
def restore_from_disk(self) -> int:
|
||||
"""Re-register datasets from persisted metadata.
|
||||
def _restore_metadata(self) -> int:
|
||||
"""Read metadata from disk and store as lightweight *stubs*.
|
||||
|
||||
Attempts two strategies in order:
|
||||
1. **SQLite table** — when ``metadata.sqlite_table`` exists (fast,
|
||||
always available, no CSV dependency).
|
||||
2. **CSV glob** — falls back to ``load_csv_directory`` for datasets
|
||||
that were stored before the SQLite migration or whose SQLite table
|
||||
was dropped.
|
||||
A stub contains everything needed to reconstruct the LazyFrame on
|
||||
demand but performs **no I/O** beyond reading the JSON file. Real
|
||||
data loading is deferred until the first :meth:`get_dataset` call
|
||||
(or the background thread).
|
||||
|
||||
Returns the number of successfully restored datasets.
|
||||
Returns the number of stubs created.
|
||||
"""
|
||||
path = _get_store_path()
|
||||
if not path.exists():
|
||||
@@ -125,47 +144,131 @@ class SessionStore:
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return 0
|
||||
|
||||
# Lazy imports to avoid circular dependency at module level
|
||||
from analysis.data_loader import load_csv_directory, load_from_db # fmt: skip
|
||||
count = 0
|
||||
with self._datalock:
|
||||
for ds_id, info in data.items():
|
||||
# Skip if already loaded (idempotent on re-calls)
|
||||
if ds_id in self._stores:
|
||||
continue
|
||||
meta = dict(info.get('metadata', {}))
|
||||
schema = dict(info.get('schema', {}))
|
||||
csv_glob = info.get('csv_glob')
|
||||
self._stores[ds_id] = {
|
||||
'type': 'dataset',
|
||||
'_stub': {
|
||||
'sqlite_table': meta.get('sqlite_table'),
|
||||
'csv_glob': csv_glob,
|
||||
'schema_overrides': meta.get('schema_overrides'),
|
||||
},
|
||||
'schema': schema,
|
||||
'metadata': meta,
|
||||
'parent_id': None,
|
||||
}
|
||||
count += 1
|
||||
self._restore_done.set()
|
||||
return count
|
||||
|
||||
restored = 0
|
||||
for ds_id, info in data.items():
|
||||
meta = dict(info.get('metadata', {}))
|
||||
schema = dict(info.get('schema', {}))
|
||||
restored_ok = False
|
||||
def _load_pending_dataset(self, ds_id: str) -> bool:
|
||||
"""Resolve a stub entry into a real LazyFrame.
|
||||
|
||||
# Strategy 1: SQLite table (fast path)
|
||||
sqlite_table = meta.get('sqlite_table')
|
||||
if sqlite_table:
|
||||
try:
|
||||
lf = load_from_db(sqlite_table)
|
||||
if lf is not None:
|
||||
self.store_dataset(ds_id, lf, schema=schema, metadata=meta)
|
||||
restored += 1
|
||||
restored_ok = True
|
||||
except Exception:
|
||||
pass # fall through to CSV restore
|
||||
Called on the first :meth:`get_dataset` for a dataset that was
|
||||
restored via :meth:`_restore_metadata`.
|
||||
|
||||
if restored_ok:
|
||||
continue
|
||||
Attempts two strategies:
|
||||
1. **SQLite table** (fast — uses :func:`load_from_db_lazy`).
|
||||
2. **CSV glob** (legacy fallback — scans the filesystem).
|
||||
|
||||
# Strategy 2: CSV glob (legacy / fallback)
|
||||
csv_glob = info.get('csv_glob')
|
||||
if not csv_glob:
|
||||
continue
|
||||
Returns ``True`` if the dataset was successfully loaded.
|
||||
"""
|
||||
from analysis.data_loader import load_csv_directory, load_from_db_lazy # fmt: skip
|
||||
|
||||
entry = self._stores.get(ds_id)
|
||||
if not entry or '_stub' not in entry:
|
||||
return True # nothing to do
|
||||
|
||||
stub = entry.pop('_stub')
|
||||
meta = entry.get('metadata', {})
|
||||
schema = entry.get('schema', {})
|
||||
|
||||
# Strategy 1: SQLite table (fast, native Polars reader)
|
||||
sqlite_table = stub.get('sqlite_table')
|
||||
if sqlite_table:
|
||||
try:
|
||||
lf, schema_csv, row_count, file_count, memory_mb = load_csv_directory(csv_glob)
|
||||
# Preserve original metadata counts from disk — CSV re-scan may
|
||||
# detect different file/row counts if files changed on disk.
|
||||
if 'file_count' not in meta and file_count is not None:
|
||||
meta['file_count'] = file_count
|
||||
if 'row_count' not in meta and row_count is not None:
|
||||
meta['row_count'] = row_count
|
||||
self.store_dataset(ds_id, lf, schema=schema_csv, metadata=meta)
|
||||
restored += 1
|
||||
lf = load_from_db_lazy(sqlite_table, schema_overrides=stub.get('schema_overrides'))
|
||||
if lf is not None:
|
||||
entry['lazyframe'] = lf
|
||||
if 'row_count' not in meta:
|
||||
meta['row_count'] = _estimate_row_count(lf)
|
||||
return True
|
||||
except Exception:
|
||||
# File(s) no longer exist or schema changed — skip gracefully
|
||||
continue
|
||||
pass # fall through to CSV restore
|
||||
|
||||
# Strategy 2: CSV glob (legacy fallback)
|
||||
csv_glob = stub.get('csv_glob')
|
||||
if not csv_glob:
|
||||
# No recoverable data source — entry remains without LazyFrame
|
||||
return False
|
||||
try:
|
||||
lf, schema_csv, row_count, file_count, memory_mb = load_csv_directory(csv_glob)
|
||||
if 'file_count' not in meta and file_count is not None:
|
||||
meta['file_count'] = file_count
|
||||
if 'row_count' not in meta and row_count is not None:
|
||||
meta['row_count'] = row_count
|
||||
# Merge CSV-scanned schema (may have more/different columns)
|
||||
merged_schema = {**schema, **schema_csv}
|
||||
entry['lazyframe'] = lf
|
||||
entry['schema'] = merged_schema
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def start_async_restore(self) -> int:
|
||||
"""Restore dataset metadata immediately; load data in background.
|
||||
|
||||
At startup this reads the small metadata JSON and stores *stubs*
|
||||
(metadata only, no data I/O). A daemon thread then resolves each
|
||||
stub into a real LazyFrame without blocking the main thread.
|
||||
|
||||
Returns the number of datasets queued for restoration.
|
||||
"""
|
||||
count = self._restore_metadata()
|
||||
if count > 0:
|
||||
t = threading.Thread(
|
||||
target=self._background_restore,
|
||||
daemon=True,
|
||||
name='session-restore',
|
||||
)
|
||||
t.start()
|
||||
return count
|
||||
|
||||
def _background_restore(self) -> None:
|
||||
"""Pre-load all stubs in a daemon thread (best-effort)."""
|
||||
with self._datalock:
|
||||
ds_ids = [
|
||||
ds_id for ds_id, entry in self._stores.items()
|
||||
if entry.get('type') == 'dataset' and '_stub' in entry
|
||||
]
|
||||
for ds_id in ds_ids:
|
||||
try:
|
||||
self._load_pending_dataset(ds_id)
|
||||
except Exception:
|
||||
pass # best-effort; stub remains for on-demand fallback
|
||||
|
||||
def restore_from_disk(self) -> int:
|
||||
"""Legacy synchronous restore (kept for backward compatibility).
|
||||
|
||||
Prefer :meth:`start_async_restore` for non-blocking startup.
|
||||
"""
|
||||
self._restore_metadata()
|
||||
with self._datalock:
|
||||
ds_ids = [
|
||||
ds_id for ds_id, entry in self._stores.items()
|
||||
if entry.get('type') == 'dataset' and '_stub' in entry
|
||||
]
|
||||
restored = 0
|
||||
for ds_id in ds_ids:
|
||||
if self._load_pending_dataset(ds_id):
|
||||
restored += 1
|
||||
return restored
|
||||
|
||||
# ── dataset operations ──────────────────────────────────────────────
|
||||
@@ -199,12 +302,23 @@ class SessionStore:
|
||||
self._save_to_disk()
|
||||
|
||||
def get_dataset(self, dataset_id: str) -> Optional[dict]:
|
||||
"""Retrieve a stored dataset entry (or *None*)."""
|
||||
"""Retrieve a stored dataset entry (or *None*).
|
||||
|
||||
If the entry is a *stub* (restored from metadata but not yet
|
||||
materialised), this method triggers on-demand loading of the
|
||||
LazyFrame before returning.
|
||||
"""
|
||||
with self._datalock:
|
||||
entry = self._stores.get(dataset_id)
|
||||
if entry and entry.get('type') == 'dataset':
|
||||
return entry
|
||||
return None
|
||||
if not entry or entry.get('type') != 'dataset':
|
||||
return None
|
||||
needs_load = '_stub' in entry
|
||||
|
||||
if needs_load:
|
||||
self._load_pending_dataset(dataset_id)
|
||||
|
||||
with self._datalock:
|
||||
return self._stores.get(dataset_id)
|
||||
|
||||
def drop_dataset(self, dataset_id: str) -> bool:
|
||||
"""Remove a dataset from the store and force garbage collection.
|
||||
@@ -255,6 +369,7 @@ class SessionStore:
|
||||
'svd_components': meta.get('svd_components', 0),
|
||||
'columns': list(entry.get('schema', {}).keys()),
|
||||
'column_count': len(entry.get('schema', {})),
|
||||
'is_stub': '_stub' in entry,
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
@@ -25,6 +25,12 @@ from .data_loader import load_csv_directory
|
||||
from .data_profiler import profile_dataset as _profile_dataset
|
||||
from .session_store import SessionStore, _generate_id
|
||||
|
||||
# Conditional line_profiler support (no-op when not installed)
|
||||
try:
|
||||
profile # type: ignore[name-defined]
|
||||
except NameError:
|
||||
profile = lambda f: f
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Tool metadata
|
||||
@@ -992,7 +998,7 @@ async def _handle_filter_data(
|
||||
|
||||
# Estimate row count (fast streaming count)
|
||||
try:
|
||||
row_count = filtered_lf.select(pl.lit(1).count()).collect(streaming=True).item()
|
||||
row_count = filtered_lf.select(pl.len()).collect(streaming=True).item()
|
||||
except Exception:
|
||||
row_count = None
|
||||
|
||||
@@ -1680,6 +1686,7 @@ async def _handle_extract_features(
|
||||
}
|
||||
|
||||
|
||||
@profile
|
||||
def _save_features_to_db(
|
||||
cluster_result_id: str,
|
||||
features: list[dict],
|
||||
@@ -2053,6 +2060,7 @@ _ANOMALY_FEATURE_COLS = [
|
||||
]
|
||||
|
||||
|
||||
@profile
|
||||
def _add_adaptive_scores(lf: pl.LazyFrame, feature_names: list[str]) -> pl.LazyFrame:
|
||||
"""Add anomaly scores using data-driven weights (SVD + IQR + percentile).
|
||||
|
||||
@@ -2207,7 +2215,7 @@ async def _handle_filter_and_cluster(
|
||||
|
||||
# Estimate row count (fast streaming count)
|
||||
try:
|
||||
row_count = filtered_lf.select(pl.lit(1).count()).collect(streaming=True).item()
|
||||
row_count = filtered_lf.select(pl.len()).collect(streaming=True).item()
|
||||
except Exception:
|
||||
row_count = None
|
||||
|
||||
|
||||
+180
-43
@@ -14,6 +14,13 @@ from django.views.decorators.csrf import csrf_exempt
|
||||
|
||||
from config import get_config, save_config, Config
|
||||
from .models import AnalysisRun, ClusterResult, EntityProfile
|
||||
from . import nl_describe
|
||||
|
||||
# Conditional line_profiler support (no-op when not installed)
|
||||
try:
|
||||
profile # type: ignore[name-defined]
|
||||
except NameError:
|
||||
profile = lambda f: f
|
||||
|
||||
|
||||
def dashboard(request):
|
||||
@@ -80,13 +87,28 @@ def cluster_overview(request, display_id):
|
||||
clusters = run.clusters.exclude(cluster_label=-1).order_by('-size').prefetch_related('features')
|
||||
noise = run.clusters.filter(cluster_label=-1).first()
|
||||
|
||||
# Build UMAP scatter data for Canvas
|
||||
entities = run.entities.exclude(embedding_x=None).exclude(embedding_y=None).only('embedding_x', 'embedding_y', 'cluster_label', 'entity_value')
|
||||
# Build UMAP scatter data for Canvas / Three.js 3D
|
||||
entities = run.entities.exclude(embedding_x=None).exclude(embedding_y=None).only(
|
||||
'embedding_x', 'embedding_y', 'embedding_z', 'cluster_label', 'entity_value'
|
||||
)
|
||||
scatter_data = [
|
||||
{'x': e.embedding_x, 'y': e.embedding_y, 'label': e.cluster_label, 'entity': e.entity_value}
|
||||
{'x': e.embedding_x, 'y': e.embedding_y, 'z': e.embedding_z,
|
||||
'label': e.cluster_label, 'entity': e.entity_value}
|
||||
for e in entities
|
||||
]
|
||||
|
||||
# Noise cluster features for enhanced noise card
|
||||
noise_features = []
|
||||
noise_summary = ''
|
||||
if noise:
|
||||
noise_features_qs = noise.features.all().order_by('-distinguishing_score')[:10]
|
||||
noise_features = [
|
||||
{'feature_name': f.feature_name, 'score': f.distinguishing_score, 'mean': f.mean}
|
||||
for f in noise_features_qs
|
||||
]
|
||||
if noise_features:
|
||||
noise_summary = nl_describe.describe_cluster(noise_features)
|
||||
|
||||
# Build geo scatter data from feature_json (lat/lon)
|
||||
geo_data = []
|
||||
geo_skipped = 0
|
||||
@@ -110,7 +132,7 @@ def cluster_overview(request, display_id):
|
||||
cluster_sizes = [{'label': c.cluster_label, 'size': c.size, 'silhouette': c.silhouette_score}
|
||||
for c in clusters]
|
||||
|
||||
# Top features per cluster from ClusterFeature
|
||||
# Top features per cluster from ClusterFeature + NL summaries
|
||||
top_features = {}
|
||||
for c in clusters:
|
||||
features_qs = c.features.all().order_by('-distinguishing_score')[:10]
|
||||
@@ -118,6 +140,9 @@ def cluster_overview(request, display_id):
|
||||
{'feature_name': f.feature_name, 'score': f.distinguishing_score, 'mean': f.mean}
|
||||
for f in features_qs
|
||||
]
|
||||
# Generate NL summary — attach to cluster object for template access
|
||||
feats = top_features[str(c.cluster_label)]
|
||||
c.nl_summary = nl_describe.describe_cluster(feats) if feats else ''
|
||||
|
||||
return render(request, 'analysis/cluster_overview.html', {
|
||||
'run': run,
|
||||
@@ -129,6 +154,8 @@ def cluster_overview(request, display_id):
|
||||
'geo_count': len(geo_data),
|
||||
'cluster_sizes_json': json.dumps(cluster_sizes),
|
||||
'top_features_json': json.dumps(top_features),
|
||||
'noise_features': noise_features,
|
||||
'noise_summary': noise_summary,
|
||||
})
|
||||
|
||||
|
||||
@@ -423,6 +450,7 @@ import traceback, logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@profile
|
||||
def _background_process(run_id, upload_dir):
|
||||
"""Background: stream-load CSVs → save to SQLite → ready for analysis."""
|
||||
import django
|
||||
@@ -796,6 +824,7 @@ def _run_pipeline_worker(run_id, analysis_fn, **ctx):
|
||||
|
||||
# ── Shared clustering pipeline ─────────────────────────────────────────
|
||||
|
||||
@profile
|
||||
def _run_clustering_pipeline(run, store, entity_ds_id, feature_columns, algorithm,
|
||||
min_cluster_size, run_umap=True, head=None):
|
||||
"""Unified clustering pipeline: clustering → feature extraction → UMAP → ORM save.
|
||||
@@ -917,7 +946,46 @@ def _run_clustering_pipeline(run, store, entity_ds_id, feature_columns, algorith
|
||||
if 'error' in feat_result:
|
||||
raise Exception(feat_result['error'])
|
||||
|
||||
# ── Step 3: UMAP-2D embedding (sampled training for large data) ─────────
|
||||
# ── Step 2b: Cluster SVD feature extraction ──────────────────────
|
||||
run.progress_msg = '正在进行聚类SVD特征提取...'
|
||||
run.save(update_fields=['progress_msg'])
|
||||
try:
|
||||
from analysis.distance import cluster_svd_extract
|
||||
lf = entry['lazyframe']
|
||||
cluster_entry = store.get_cluster_result(cluster_id)
|
||||
labels_list = cluster_entry.get('labels', []) if cluster_entry else []
|
||||
|
||||
if labels_list and feature_cols:
|
||||
svd_results = cluster_svd_extract(lf, labels_list, feature_cols)
|
||||
|
||||
# Persist SVD features to ClusterFeature (method='cluster_svd')
|
||||
from analysis.models import ClusterFeature as CF
|
||||
from analysis.models import ClusterResult as CR
|
||||
for label, svd_info in svd_results.items():
|
||||
cr = CR.objects.filter(run=run, cluster_label=label).first()
|
||||
if cr is None:
|
||||
continue
|
||||
# Delete old zscore entries for this cluster, replace with SVD
|
||||
CF.objects.filter(cluster=cr, distinguishing_method='zscore').delete()
|
||||
for feat_name in svd_info.get('features', [])[:10]:
|
||||
idx = svd_info['features'].index(feat_name) if feat_name in svd_info['features'] else -1
|
||||
strength = (svd_info.get('feature_strength', [])[idx]
|
||||
if idx >= 0 and idx < len(svd_info.get('feature_strength', []))
|
||||
else None)
|
||||
CF.objects.get_or_create(
|
||||
cluster=cr,
|
||||
feature_name=feat_name,
|
||||
defaults={
|
||||
'mean': None, 'std': None, 'median': None,
|
||||
'p25': None, 'p75': None, 'missing_rate': None,
|
||||
'distinguishing_score': float(strength) if strength is not None else None,
|
||||
'distinguishing_method': 'cluster_svd',
|
||||
},
|
||||
)
|
||||
except Exception as svd_err:
|
||||
logger.warning(f'Cluster SVD extraction skipped: {svd_err}')
|
||||
|
||||
# ── Step 3: UMAP-3D embedding (2D fallback) ───────────────────────
|
||||
if run_umap:
|
||||
try:
|
||||
lf = entry['lazyframe']
|
||||
@@ -947,34 +1015,53 @@ def _run_clustering_pipeline(run, store, entity_ds_id, feature_columns, algorith
|
||||
else:
|
||||
df_umap = lf.collect(streaming=True)
|
||||
|
||||
if len(df_umap) > 2:
|
||||
if n_total > MAX_UMAP_TRAIN:
|
||||
# Train UMAP on 10K sample, batch-transform full dataset
|
||||
mat_sample = df_sample.select(num_cols).to_numpy()
|
||||
scaler = StandardScaler().fit(mat_sample)
|
||||
mat_sample_scaled = np.nan_to_num(scaler.transform(mat_sample), nan=0.0)
|
||||
reducer = umap.UMAP(n_components=2, random_state=42,
|
||||
n_neighbors=15, min_dist=0.1,
|
||||
metric='euclidean')
|
||||
reducer.fit(mat_sample_scaled)
|
||||
# Batch transform in 1K-row batches
|
||||
coords_list = []
|
||||
for start in range(0, len(df_umap), BATCH_SIZE):
|
||||
end = min(start + BATCH_SIZE, len(df_umap))
|
||||
mat_batch = scaler.transform(
|
||||
df_umap[start:end].select(num_cols).to_numpy())
|
||||
mat_batch = np.nan_to_num(mat_batch, nan=0.0)
|
||||
coords_list.append(reducer.transform(mat_batch))
|
||||
coords = np.vstack(coords_list)
|
||||
else:
|
||||
mat = np.nan_to_num(StandardScaler().fit_transform(
|
||||
df_umap.select(num_cols).to_numpy()), nan=0.0)
|
||||
reducer = umap.UMAP(n_components=2, random_state=42,
|
||||
n_neighbors=15, min_dist=0.1,
|
||||
metric='euclidean')
|
||||
coords = reducer.fit_transform(mat)
|
||||
umap_components = 3 # try 3D first
|
||||
coords = None
|
||||
reducer = None
|
||||
|
||||
non_num = [c for c in df_umap.columns if c not in num_cols and not c.startswith('_')]
|
||||
if len(df_umap) > 2:
|
||||
for attempt_n in (3, 2):
|
||||
try:
|
||||
if n_total > MAX_UMAP_TRAIN:
|
||||
# Train UMAP on 10K sample, batch-transform full dataset
|
||||
mat_sample = df_sample.select(num_cols).to_numpy()
|
||||
scaler = StandardScaler().fit(mat_sample)
|
||||
mat_sample_scaled = np.nan_to_num(
|
||||
scaler.transform(mat_sample), nan=0.0)
|
||||
reducer = umap.UMAP(n_components=attempt_n,
|
||||
random_state=42,
|
||||
n_neighbors=15, min_dist=0.1,
|
||||
metric='euclidean')
|
||||
reducer.fit(mat_sample_scaled)
|
||||
# Batch transform in 1K-row batches
|
||||
coords_list = []
|
||||
for start in range(0, len(df_umap), BATCH_SIZE):
|
||||
end = min(start + BATCH_SIZE, len(df_umap))
|
||||
mat_batch = scaler.transform(
|
||||
df_umap[start:end].select(num_cols).to_numpy())
|
||||
mat_batch = np.nan_to_num(mat_batch, nan=0.0)
|
||||
coords_list.append(reducer.transform(mat_batch))
|
||||
coords = np.vstack(coords_list)
|
||||
else:
|
||||
mat = np.nan_to_num(StandardScaler().fit_transform(
|
||||
df_umap.select(num_cols).to_numpy()), nan=0.0)
|
||||
reducer = umap.UMAP(n_components=attempt_n,
|
||||
random_state=42,
|
||||
n_neighbors=15, min_dist=0.1,
|
||||
metric='euclidean')
|
||||
coords = reducer.fit_transform(mat)
|
||||
umap_components = attempt_n
|
||||
break
|
||||
except Exception as e_umap:
|
||||
if attempt_n == 2:
|
||||
raise
|
||||
logger.warning(f'UMAP 3D failed ({e_umap}), falling back to 2D')
|
||||
|
||||
if coords is None:
|
||||
raise Exception('UMAP produced no output')
|
||||
|
||||
non_num = [c for c in df_umap.columns
|
||||
if c not in num_cols and not c.startswith('_')]
|
||||
ent_col = non_num[0] if non_num else df_umap.columns[0]
|
||||
|
||||
cluster_entry = store.get_cluster_result(cluster_id)
|
||||
@@ -991,19 +1078,23 @@ def _run_clustering_pipeline(run, store, entity_ds_id, feature_columns, algorith
|
||||
lbl = labels_list[i] if i < len(labels_list) else -1
|
||||
cache_key = f'{run.id}_{lbl}'
|
||||
if cache_key not in cr_cache:
|
||||
cr_cache[cache_key] = CR.objects.filter(run=run, cluster_label=lbl).first()
|
||||
cr_cache[cache_key] = CR.objects.filter(
|
||||
run=run, cluster_label=lbl).first()
|
||||
has_z = umap_components >= 3 and coords.shape[1] >= 3
|
||||
profiles.append(EP(
|
||||
entity_value=ev, run=run, cluster_label=lbl,
|
||||
cluster=cr_cache[cache_key],
|
||||
embedding_x=float(coords[i, 0]) if i < len(coords) else None,
|
||||
embedding_y=float(coords[i, 1]) if i < len(coords) else None,
|
||||
embedding_z=float(coords[i, 2]) if has_z
|
||||
and i < len(coords) else None,
|
||||
))
|
||||
if profiles:
|
||||
EP.objects.bulk_create(profiles, ignore_conflicts=True, batch_size=1000)
|
||||
run.entity_count = len(profiles)
|
||||
run.save(update_fields=['entity_count'])
|
||||
except Exception as umap_err:
|
||||
logger.warning(f'UMAP-2D embedding skipped: {umap_err}')
|
||||
logger.warning(f'UMAP embedding skipped: {umap_err}')
|
||||
|
||||
# Fallback: if entity_count not set (UMAP skipped), derive from cluster sizes
|
||||
if run.entity_count is None:
|
||||
@@ -1392,17 +1483,43 @@ def run_llm_analysis_view(request):
|
||||
|
||||
def auto_page(request):
|
||||
"""LLM-driven auto analysis page."""
|
||||
from analysis.session_store import SessionStore
|
||||
import json as _json
|
||||
|
||||
cfg = get_config()
|
||||
selected_run = None
|
||||
run_id = request.GET.get('run_id')
|
||||
if run_id:
|
||||
try:
|
||||
selected_run = AnalysisRun.objects.get(display_id=int(run_id))
|
||||
except (ValueError, AnalysisRun.DoesNotExist):
|
||||
pass
|
||||
|
||||
# Get upload runs for dataset selector (with status badges)
|
||||
runs = AnalysisRun.objects.filter(run_type='upload').order_by('-created_at')[:50]
|
||||
|
||||
# Build datasets_with_status for filter tabs
|
||||
from analysis.models import AnalysisRun as _AR
|
||||
upload_runs = _AR.objects.filter(run_type='upload').order_by('-created_at')[:50]
|
||||
datasets_with_status = []
|
||||
for r in upload_runs:
|
||||
s = r.status
|
||||
if s in ('completed',):
|
||||
label = '已分析'
|
||||
elif s in ('failed',):
|
||||
label = '错误'
|
||||
elif s in ('pending', 'loading', 'profiling', 'aggregating', 'clustering', 'extracting'):
|
||||
label = '分析中'
|
||||
else:
|
||||
label = '待分析'
|
||||
datasets_with_status.append({
|
||||
'dataset_id': r.display_id,
|
||||
'display_id': r.display_id,
|
||||
'run_status': s,
|
||||
'status_label': label,
|
||||
'row_count': r.total_flows,
|
||||
'file_count': getattr(r, 'file_count', None),
|
||||
'csv_glob': r.csv_glob or '',
|
||||
'sqlite_table': r.sqlite_table or '',
|
||||
})
|
||||
|
||||
return render(request, 'tianxuan/auto.html', {
|
||||
'llm_configured': bool(cfg.llm.base_url and cfg.llm.api_key),
|
||||
'selected_run': selected_run,
|
||||
'runs': runs,
|
||||
'datasets_with_status_json': _json.dumps(datasets_with_status, ensure_ascii=False),
|
||||
})
|
||||
|
||||
|
||||
@@ -1729,7 +1846,27 @@ def tool_lab_run(request):
|
||||
if not tool_name:
|
||||
return JsonResponse({'error': 'Missing tool name'}, status=400)
|
||||
|
||||
from analysis.tool_registry import handle_call
|
||||
# ── Backend validation: check required params + dataset existence ──
|
||||
from analysis.tool_registry import handle_call, get_tools_meta
|
||||
from analysis.session_store import SessionStore
|
||||
tools_meta = get_tools_meta()
|
||||
tool_schema = None
|
||||
for t in tools_meta:
|
||||
if t.name == tool_name:
|
||||
tool_schema = t.inputSchema
|
||||
break
|
||||
if tool_schema:
|
||||
required_fields = tool_schema.get('required', [])
|
||||
for field in required_fields:
|
||||
val = params.get(field)
|
||||
if val is None or (isinstance(val, str) and val.strip() == ''):
|
||||
return JsonResponse({'error': f'Missing required parameter: {field}'}, status=400)
|
||||
# If field is a dataset_id, verify it exists in the store
|
||||
if 'dataset' in field.lower():
|
||||
store = SessionStore()
|
||||
if not store.get_dataset(str(val)):
|
||||
return JsonResponse({'error': f'Dataset not found: {val}'}, status=400)
|
||||
|
||||
import asyncio
|
||||
result = asyncio.run(handle_call(tool_name, params))
|
||||
return JsonResponse({'status': 'ok', 'result': result})
|
||||
|
||||
+222
-687
File diff suppressed because it is too large
Load Diff
+480
@@ -0,0 +1,480 @@
|
||||
# 天璇 (TianXuan) 故障诊断手册
|
||||
|
||||
> 常见问题排查指南。使用 `scripts/diagnose_compare.py` 进行跨机日志对比诊断。
|
||||
|
||||
---
|
||||
|
||||
## 目录
|
||||
|
||||
1. [启动问题](#1-启动问题)
|
||||
2. [数据加载问题](#2-数据加载问题)
|
||||
3. [聚类问题](#3-聚类问题)
|
||||
4. [UMAP 问题](#4-umap-问题)
|
||||
5. [数据库问题](#5-数据库问题)
|
||||
6. [LLM 集成问题](#6-llm-集成问题)
|
||||
7. [前端问题](#7-前端问题)
|
||||
8. [跨机一致性问题](#8-跨机一致性问题)
|
||||
9. [性能问题](#9-性能问题)
|
||||
10. [诊断工具使用](#10-诊断工具使用)
|
||||
|
||||
---
|
||||
|
||||
## 1. 启动问题
|
||||
|
||||
### 1.1 `config` 模块找不到
|
||||
|
||||
**症状**: `ModuleNotFoundError: No module named 'config'`
|
||||
|
||||
**原因**: Python 路径未包含项目根目录。
|
||||
|
||||
**解决**:
|
||||
```bat
|
||||
:: 在项目根目录下运行
|
||||
set PYTHONUTF8=1
|
||||
chcp 65001
|
||||
start /B runtime\python\python.exe manage.py runserver
|
||||
```
|
||||
|
||||
### 1.2 首次启动无数据库
|
||||
|
||||
**症状**: `OperationalError: no such table`
|
||||
|
||||
**解决**:
|
||||
```bat
|
||||
runtime\python\python.exe manage.py migrate
|
||||
```
|
||||
|
||||
`wsgi.py` 已包含 SQLite 启动自修复逻辑，自动创建缺失的数据库文件。
|
||||
|
||||
### 1.3 端口被占用
|
||||
|
||||
**症状**: `Error: [WinError 10048] 通常每个套接字地址只允许使用一次`
|
||||
|
||||
**解决**:
|
||||
```bat
|
||||
:: 查占用进程
|
||||
netstat -ano | findstr :8000
|
||||
:: 改端口 (config.yaml)
|
||||
server:
|
||||
port: 8080
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 数据加载问题
|
||||
|
||||
### 2.1 Schema 不匹配 (多文件列名不一致)
|
||||
|
||||
**症状**: `ValueError: schema mismatch` 或列名不一致时报错
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
```
|
||||
|
||||
**解决**:
|
||||
- 方案 A: 设置 `schema_strict=False` (默认), 自动并集合并, 缺失列填 null
|
||||
- 方案 B: 调用 `repair_schema` MCP 工具 — 自动对齐列名 (大小写合并、下划线/连字符规范化)
|
||||
- 方案 C: 在 `config.yaml` 中通过 `columns` 映射手动统一列名
|
||||
|
||||
### 2.2 CSV 编码问题 (乱码)
|
||||
|
||||
**症状**: 中文列名或数据出现乱码、`UnicodeDecodeError`
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
:: 检查文件编码
|
||||
runtime\python\python.exe -c "
|
||||
with open('data/your.csv', 'rb') as f:
|
||||
print(f.read(10).hex())
|
||||
"
|
||||
```
|
||||
|
||||
**解决**:
|
||||
- BOM 标记: 系统自动检测 BOM → UTF-8/UTF-16LE/UTF-16BE
|
||||
- 无 BOM: 默认 UTF-8, 可手动指定 `encoding="latin-1"` 或 `encoding="gbk"`
|
||||
- 所有命令加 `chcp 65001` + `set PYTHONUTF8=1`
|
||||
|
||||
### 2.3 `+` 号字符串无法解析为数字
|
||||
|
||||
**症状**: `"+"` 在数值列中导致 `cannot cast to Float64`
|
||||
|
||||
**解决**: 已在 `_coerce_to_float()` 中修复 — `"+"` / `"-"` / 空白 / `"N/A"` 等伪值在聚合前自动转为 null。日志标记 `[CLEAN] numeric coercion applied`。
|
||||
|
||||
### 2.4 列类型检测错误 (IP 列被识别为 String)
|
||||
|
||||
**症状**: 某列应为 IPv4 但被检测为 STRING
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
:: 在 MCP 中调用
|
||||
validate_data(dataset_id="ds_...")
|
||||
```
|
||||
|
||||
**解决**:
|
||||
- 在 `config.yaml` 中手动指定: `columns: { src_ip: ipv4, dst_ip: ipv4 }`
|
||||
- 配置优先级: config → 值优先检测 → 名称推断 → STRING 回退
|
||||
|
||||
---
|
||||
|
||||
## 3. 聚类问题
|
||||
|
||||
### 3.1 聚类返回 0 个簇或全噪声
|
||||
|
||||
**症状**: `n_clusters=0` 或所有标签为 -1 (HDBSCAN 噪声)
|
||||
|
||||
**诊断**:
|
||||
```python
|
||||
diagnose_clustering(dataset_id="...", cluster_result_id="...")
|
||||
```
|
||||
输出包含: 特征方差分析、相关矩阵、稀疏性检查、参数建议。
|
||||
|
||||
**常见原因与解决**:
|
||||
|
||||
| 原因 | 诊断信号 | 解决 |
|
||||
|------|---------|------|
|
||||
| 样本太少 | n < 3 | 检查数据加载是否成功; 增大数据量 |
|
||||
| min_cluster_size 太大 | 日志中 min_cluster_size > n/2 | 系统已自动调整为 n//20, 如仍失败手动降低 |
|
||||
| 全零方差特征 | 方差 < 1e-10 | 系统已自动过滤零方差列; 检查特征选择是否合理 |
|
||||
| 特征高度相关 | Pearson > 0.95 | 系统已自动去相关; 仍有问题则减少特征数 |
|
||||
| 数据全是 NaN | 日志 all-NaN columns dropped | 检查数据源; 增加样本量 |
|
||||
|
||||
### 3.2 "mean on str" 错误
|
||||
|
||||
**症状**: `TypeError: can't convert type 'str' to numerator/denominator`
|
||||
|
||||
**原因**: 跨机 dtype 推断不一致 (一台推断为 Float, 另一台为 String)
|
||||
|
||||
**解决**: 已在 `build_aggregation_params()` 中修复 — 聚合前检查 dtype, 非数值列自动 cast。`_coerce_to_float()` 兜底。
|
||||
|
||||
### 3.3 KMeans 簇数与预期不符
|
||||
|
||||
**症状**: KMeans 产生的簇数与 `n_clusters` 参数不一致
|
||||
|
||||
**原因**: MiniBatchKMeans 在大数据集上可能产生少于预期的簇
|
||||
|
||||
**解决**:
|
||||
```python
|
||||
# 尝试 HDBSCAN (自动确定 k)
|
||||
run_clustering(dataset_id="...", cluster_columns=[...], algorithm="hdbscan")
|
||||
# 或增大 n_clusters
|
||||
run_clustering(..., algorithm="kmeans", params={"n_clusters": 5})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. UMAP 问题
|
||||
|
||||
### 4.1 UMAP 嵌入失败 (`[UMAP_ERR]`)
|
||||
|
||||
**症状**: 日志中出现 `[UMAP_ERR]` 但流程继续 (UMAP 计算失败不影响核心分析)
|
||||
|
||||
**常见原因**:
|
||||
- 数据量过大 (>100K): 系统已实现训练 10K + 批量变换 1K 批次
|
||||
- 全 NaN 列: UMAP 前已有 StandardScaler + `np.nan_to_num`
|
||||
- 内存不足: UMAP 需要额外内存
|
||||
|
||||
**解决**:
|
||||
```bash
|
||||
:: 检查 stderr
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv 2> umap_err.log
|
||||
```
|
||||
|
||||
### 4.2 UMAP 坐标全为 0
|
||||
|
||||
**症状**: EntityProfile 的 embedding_x/y 全为 0.0
|
||||
|
||||
**原因**: UMAP 计算过程中异常被 `try/except` 捕获, 静默回退
|
||||
|
||||
**解决**:
|
||||
- 检查 stderr 是否有 `[UMAP_ERR]` 日志
|
||||
- 确认 `umap-learn` 包已安装: `runtime\python\python.exe -c "import umap; print(umap.__version__)"`
|
||||
- 减少数据量重试
|
||||
|
||||
### 4.3 TruncatedSVD 降维导致聚类结果变化
|
||||
|
||||
**症状**: 特征 >50 维时聚类结果与预期不同
|
||||
|
||||
**原因**: SVD 降维会损失信息 (保留前 50 个主成分)
|
||||
|
||||
**解决**:
|
||||
- 手动选择更少的基础特征 (<50 个), 避免触发 SVD 降维
|
||||
- 通过 `cluster_columns` 参数精确控制输入特征列
|
||||
|
||||
---
|
||||
|
||||
## 5. 数据库问题
|
||||
|
||||
### 5.1 SQLite 锁 (`[DB_LOCKED]`)
|
||||
|
||||
**症状**: 日志中出现 `[DB_LOCKED]` 警告
|
||||
|
||||
**说明**: 系统已内置 `retry_on_lock` 装饰器 (3 次重试, 指数退避)。SQLite WAL 模式下读写并发性良好。
|
||||
|
||||
**解决**:
|
||||
- 减少并发写入 (多个 Django 进程同时写)
|
||||
- 增大超时: `sqlite3_db.config(busy_timeout=5000)`
|
||||
- 检查是否有未关闭的长时间事务
|
||||
|
||||
### 5.2 ClusterFeature 表无数据 (`db_saved=False`)
|
||||
|
||||
**症状**: `extract_features` 返回 `db_saved=False`
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
runtime\python\python.exe scripts\debug_db.py
|
||||
```
|
||||
|
||||
**常见原因**:
|
||||
- Django ORM sync_to_async 包装缺失 (已在 tool_registry.py 中修复)
|
||||
- AnalysisRun 记录未找到 (csv_glob 匹配失败)
|
||||
- 权限问题 (SQLite 文件只读)
|
||||
|
||||
### 5.3 EntityProfile 表无 embedding 坐标
|
||||
|
||||
**症状**: EntityProfile 记录存在但 embedding_x/y 为 null
|
||||
|
||||
**原因**: UMAP 计算失败 (见 4.1)
|
||||
|
||||
---
|
||||
|
||||
## 6. LLM 集成问题
|
||||
|
||||
### 6.1 400 错误
|
||||
|
||||
**症状**: LLM 返回 HTTP 400
|
||||
|
||||
**原因**: API 地址错误 / Key 无效 / 模型名错误
|
||||
|
||||
**诊断**: 配置页 → 填写 LLM 配置 → 点击「测试连通性」
|
||||
|
||||
**解决**:
|
||||
1. 确认 `base_url` 格式: `https://api.openai.com/v1` (注意末尾的 /v1)
|
||||
2. 确认 `api_key` 无多余空格
|
||||
3. 确认 `model` 名称与 API 支持的一致
|
||||
|
||||
### 6.2 超时
|
||||
|
||||
**症状**: 日志中 `latency_ms` 很高, LLM 步骤无响应
|
||||
|
||||
**解决**:
|
||||
- 增大 `max_tokens` (config.yaml)
|
||||
- 检查网络延迟
|
||||
- 32B 模型推理慢: 系统已设置 90s 超时, 如仍不足可增加
|
||||
|
||||
### 6.3 工具调用失败
|
||||
|
||||
**症状**: `[TOOL]` 日志后出现错误
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
:: 查看完整 LLM 交互日志
|
||||
type logs\tianxuan.log | findstr "[TOOL] [LLM]"
|
||||
```
|
||||
|
||||
**解决**:
|
||||
- 检查工具返回的错误信息
|
||||
- LLM 可能生成了格式错误的工具调用 JSON → 重试
|
||||
- 重复调用检测: orchestrator 会自动跳过重复的工具调用
|
||||
|
||||
### 6.4 分析结果为空
|
||||
|
||||
**症状**: LLM 完成但无聚类结果
|
||||
|
||||
**常见原因**:
|
||||
- 实体检测失败 → 手动指定 `entity_column`
|
||||
- 数据量太小 → 增大样本
|
||||
- 实体列检测到非 IP 列 → 通过 `entity_columns` 强制指定
|
||||
|
||||
---
|
||||
|
||||
## 7. 前端问题
|
||||
|
||||
### 7.1 Canvas 散点图不显示
|
||||
|
||||
**症状**: 聚类概览页空白
|
||||
|
||||
**诊断**:
|
||||
```bash
|
||||
:: 检查浏览器控制台 (F12)
|
||||
:: 常见错误: "Cannot read properties of undefined"
|
||||
```
|
||||
|
||||
**解决**:
|
||||
- 确认聚类已完成 (检查 AnalysisRun.status == 'completed')
|
||||
- 确认有 UMAP/PCA 坐标数据 (EntityProfile.embedding_x 非 null)
|
||||
- 清除浏览器缓存 (Ctrl+F5)
|
||||
|
||||
### 7.2 3D 地球空白
|
||||
|
||||
**症状**: `/globe/` 页面渲染为空白
|
||||
|
||||
**原因**:
|
||||
- Three.js 未加载 (检查 Network tab, 603KB 的 three.min.js)
|
||||
- 地球贴图未加载 (1.4MB earth_atmos_2048.jpg)
|
||||
- 数据源中没有经纬度信息
|
||||
|
||||
**解决**:
|
||||
- 确认 `static/tianxuan/three.min.js` 和 `earth_atmos_2048.jpg` 存在
|
||||
- 数据中优先使用 `src_latitude/src_longitude` 列, 无则 GeoIP 回退
|
||||
- 确保 GeoIP 数据文件 `data/geoip_data.txt` 存在
|
||||
|
||||
### 7.3 Leaflet 地图不加载 (简易分析模块)
|
||||
|
||||
**症状**: `/simple/` 页面地图灰色
|
||||
|
||||
**原因**: OSM 瓦片服务器不可达 (离线环境)
|
||||
|
||||
**解决**: 简易分析模块的 Leaflet 使用 OSM 瓦片, 离线时需预先缓存或切换到本地瓦片源。
|
||||
|
||||
---
|
||||
|
||||
## 8. 跨机一致性问题
|
||||
|
||||
### 8.1 核心诊断流程
|
||||
|
||||
```bat
|
||||
:: 两台机器各自运行管道, 保存日志
|
||||
del logs\tianxuan.log
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
||||
copy logs\tianxuan.log log_%COMPUTERNAME%.txt
|
||||
|
||||
:: 对比
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_MACHINE_A.txt log_MACHINE_B.txt
|
||||
```
|
||||
|
||||
首个 `[DIFF]` 行就是问题根源。
|
||||
|
||||
### 8.2 常见跨机差异
|
||||
|
||||
| 差异类型 | 日志标记 | 根因 | 检查 |
|
||||
|---------|---------|------|------|
|
||||
| 列类型不同 | `[LOAD_SCHEMA]` | Polars 版本不一致 或 PYTHONUTF8 未生效 | `polars.__version__`, `sys.flags.utf8_mode` |
|
||||
| 关键词匹配不同 | `[FIND_COL]` | frozenset 哈希随机性 | 已通过 tuple 替代 frozenset 修复 |
|
||||
| 聚合结果不同 | `[CLUSTER_INPUT]` | random_state 不一致 或 StandardScaler 差异 | 确认 random_state=42, sklearn 版本一致 |
|
||||
| 编码问题 | `UnicodeDecodeError` | 系统编码不一致 | `sys.getdefaultencoding()`, `sys.getfilesystemencoding()` |
|
||||
| dtype 推断不同 | `Utf8 vs Float64` | Polars 推断漂移 | 已通过类型安全聚合 + `_coerce_to_float` 防护 |
|
||||
|
||||
### 8.3 运行时自检脚本
|
||||
|
||||
```bash
|
||||
:: 在两台机器上分别运行, 对比输出
|
||||
runtime\python\python.exe -c "
|
||||
import sys, polars, numpy, sklearn
|
||||
print(f'Python: {sys.version}')
|
||||
print(f'Polars: {polars.__version__}')
|
||||
print(f'NumPy: {numpy.__version__}')
|
||||
print(f'sklearn: {sklearn.__version__}')
|
||||
print(f'defaultencoding: {sys.getdefaultencoding()}')
|
||||
print(f'fsencoding: {sys.getfilesystemencoding()}')
|
||||
print(f'utf8_mode: {sys.flags.utf8_mode}')
|
||||
print(f'hashseed: {sys.flags.hash_randomization}')
|
||||
"
|
||||
```
|
||||
|
||||
### 8.4 已知已修复的跨机问题
|
||||
|
||||
| 问题 | 修复方案 | 代码位置 |
|
||||
|------|---------|---------|
|
||||
| "mean on str" | frozenset → tuple | entity_aggregator.py |
|
||||
| 编码漂移 | PYTHONUTF8=1 | run.bat |
|
||||
| DB 特征不保存 | sync_to_async 包装 | tool_registry.py |
|
||||
| Chart.js 缺失 | 纯 Canvas 2D | 全部图表 |
|
||||
| 聚类结果随机 | random_state=42 | run_clustering/run_pipeline |
|
||||
|
||||
---
|
||||
|
||||
## 9. 性能问题
|
||||
|
||||
### 9.1 大数据集 OOM
|
||||
|
||||
**症状**: 内存不足导致进程崩溃
|
||||
|
||||
**系统已实现的防护**:
|
||||
- 可用内存 < 2GB → 自动降采样 50K (psutil 检测)
|
||||
- 样本 > 100K → 采样 100K
|
||||
- 样本 > 50K → 采样 50K
|
||||
- MiniBatchKMeans 替代 KMeans (batch_size=1024)
|
||||
- SVD 降维 (>50 特征 → 50)
|
||||
- UMAP 训练 10K + 批量变换 1K 批次
|
||||
- Polars LazyFrame + streaming 惰性执行
|
||||
|
||||
**手动优化**:
|
||||
```python
|
||||
# 在 load_data 或 run_clustering 时降采样
|
||||
load_data(csv_glob="data/*.csv")
|
||||
# → 后续 run_clustering 会自动降采样
|
||||
# 或使用 filter_and_cluster 先过滤再聚类
|
||||
```
|
||||
|
||||
### 9.2 SQLite 写入慢
|
||||
|
||||
**症状**: DB 持久化耗时较长
|
||||
|
||||
**解决**:
|
||||
- WAL 模式已开启 (synchronous=NORMAL)
|
||||
- batch_size=1000 批量写入 (bulk_create)
|
||||
- 增大 cache_size: SQLite PRAGMA cache_size=20000 (20MB)
|
||||
- 临时目录移到 SSD
|
||||
|
||||
### 9.3 UMAP 计算慢
|
||||
|
||||
**症状**: extract_features 步骤耗时过长
|
||||
|
||||
**说明**: UMAP 计算分为训练 (≤10K 样本) + 批量变换 (1K 批次), 已是最优方案。
|
||||
|
||||
**如需加速**: 减小 `MAX_UMAP_TRAIN` (tool_registry.py 中的硬编码常量, 默认 10000)。
|
||||
|
||||
---
|
||||
|
||||
## 10. 诊断工具使用
|
||||
|
||||
### 10.1 MCP 诊断工具链
|
||||
|
||||
```
|
||||
问题 → validate_data (数据质量) → explore_distributions (分布检查) → find_outliers (异常值)
|
||||
→ diagnose_clustering (聚类诊断) → compare_datasets (处理前后对比) → export_debug_sample (导出样本)
|
||||
```
|
||||
|
||||
### 10.2 列结构调查
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
# 输出: 每个 CSV 的列名/类型/空值率/分布
|
||||
```
|
||||
|
||||
### 10.3 DB 持久化调试
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\debug_db.py
|
||||
# 期望: n_features=30 db_saved=True
|
||||
```
|
||||
|
||||
### 10.4 跨机日志对比
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_machine_a.txt log_machine_b.txt
|
||||
# 输出: 首个差异行的内容 (两侧对比)
|
||||
```
|
||||
|
||||
### 10.5 关键日志标记汇总
|
||||
|
||||
| 标记 | 文件 | 含义 |
|
||||
|------|------|------|
|
||||
| `[LOAD]` | data_loader.py | CSV 加载完成 |
|
||||
| `[LOAD_SCHEMA]` | data_loader.py | 每列推断类型 (最重要!) |
|
||||
| `[CLEAN]` | data_loader.py | 清洗操作 |
|
||||
| `[VALIDATE]` | data_validator.py | 数据校验 |
|
||||
| `[FIND_COL]` | entity_aggregator.py | 关键词匹配 |
|
||||
| `[CLUSTER_INPUT]` | tool_registry.py | 聚类输入形状 |
|
||||
| `[CLUSTER_SCALE]` | tool_registry.py | StandardScaler |
|
||||
| `[SCORE]` | tool_registry.py | 自适应评分权重 |
|
||||
| `[UMAP_ERR]` | tool_registry.py | UMAP 异常 |
|
||||
| `[DB_SAVE_ERROR]` | tool_registry.py | DB 持久化失败 |
|
||||
| `[DB_LOCKED]` | db_utils.py | SQLite 锁重试 |
|
||||
| `[LLM]` | llm_orchestrator.py | LLM 调用 |
|
||||
| `[TOOL]` | llm_orchestrator.py | LLM 调用工具 |
|
||||
|
||||
---
|
||||
|
||||
> **相关文档**: `AGENTS.md` (架构/配置/操作流程), `docs/工作流.md` (完整分析流水线)
|
||||
Binary file not shown.
@@ -22,19 +22,29 @@
|
||||
border-radius: 2px; flex-shrink: 0;
|
||||
}
|
||||
</style>
|
||||
<script src="{% static 'tianxuan/three.min.js' %}"></script>
|
||||
<div class="card">
|
||||
<h2>Cluster Overview — Run #{{ run.display_id }}</h2>
|
||||
<p>{{ run.total_flows }} flows, {{ run.cluster_count }} clusters</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>UMAP-2D Embedding</h2>
|
||||
<div class="chart-wrapper">
|
||||
<h2>UMAP Embedding
|
||||
<span id="viewToggle" style="display:none;font-size:0.85rem;margin-left:1rem;">
|
||||
<button id="btn2D" class="btn btn-sm" style="font-weight:bold;padding:2px 10px;">2D</button>
|
||||
<button id="btn3D" class="btn btn-sm btn-outline" style="padding:2px 10px;">3D</button>
|
||||
</span>
|
||||
</h2>
|
||||
<div id="scatter2DContainer" class="chart-wrapper">
|
||||
<div class="chart-container">
|
||||
<canvas id="scatterChart" class="scatter-canvas"></canvas>
|
||||
<div id="scatterLegend" class="canvas-legend"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="scatter3DContainer" style="display:none;width:100%;height:500px;">
|
||||
<div id="scatter3D" style="width:100%;height:500px;"></div>
|
||||
<div id="scatter3DLegend" class="canvas-legend"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% if geo_count > 0 %}
|
||||
@@ -77,6 +87,9 @@
|
||||
<h3>Cluster #{{ c.cluster_label }} <span class="badge badge-info">{{ c.size }} entities</span></h3>
|
||||
<p>Proportion: {{ c.proportion|floatformat:3 }}</p>
|
||||
<p>Silhouette: {{ c.silhouette_score|floatformat:4|default:"-" }}</p>
|
||||
{% if c.nl_summary %}
|
||||
<p style="color:#555;margin:0.5rem 0;font-style:italic;font-size:0.85rem;line-height:1.5;">{{ c.nl_summary }}</p>
|
||||
{% endif %}
|
||||
<h4 style="margin-top:0.75rem;font-size:0.9rem;">Top Distinguishing Features</h4>
|
||||
<table>
|
||||
<thead>
|
||||
@@ -105,8 +118,27 @@
|
||||
|
||||
{% if noise %}
|
||||
<div class="card">
|
||||
<h3>Noise (Cluster -1)</h3>
|
||||
<p>{{ noise.size }} entities classified as noise</p>
|
||||
<h3>Noise (Cluster -1) <span class="badge badge-info">{{ noise.size }} entities</span></h3>
|
||||
{% if noise_summary %}
|
||||
<p style="color:#555;margin:0.5rem 0;font-style:italic;font-size:0.85rem;line-height:1.5;">{{ noise_summary }}</p>
|
||||
{% endif %}
|
||||
{% if noise_features %}
|
||||
<h4 style="margin-top:0.75rem;font-size:0.9rem;">Top Noise Features</h4>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Feature</th><th>Score</th><th>Mean</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for f in noise_features %}
|
||||
<tr>
|
||||
<td>{{ f.feature_name }}</td>
|
||||
<td>{{ f.score|floatformat:3|default:"-" }}</td>
|
||||
<td>{{ f.mean|floatformat:3|default:"-" }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
@@ -115,6 +147,222 @@
|
||||
const scatterData = {{ scatter_data_json|safe }};
|
||||
const colors = ['#4361ee','#f72585','#7209b7','#4cc9f0','#f8961e','#43aa8b','#f94144','#577590','#e9c46a','#9b5de5'];
|
||||
|
||||
// ── 3D Three.js scatter ──
|
||||
let scatter3DRenderer = null;
|
||||
let scatter3DScene = null;
|
||||
let scatter3DCamera = null;
|
||||
let scatter3DPoints = null;
|
||||
let scatter3DInitialized = false;
|
||||
let is3DMode = false;
|
||||
|
||||
function hasWebGL2() {
|
||||
try {
|
||||
var c = document.createElement('canvas');
|
||||
return !!c.getContext('webgl2');
|
||||
} catch(e) { return false; }
|
||||
}
|
||||
|
||||
function build3DLegend() {
|
||||
var legendEl = document.getElementById('scatter3DLegend');
|
||||
if (!legendEl) return;
|
||||
legendEl.innerHTML = '';
|
||||
var uniqueLabels = [];
|
||||
var seen = {};
|
||||
scatterData.forEach(function(d) {
|
||||
if (!seen[d.label]) { seen[d.label] = true; uniqueLabels.push(d.label); }
|
||||
});
|
||||
uniqueLabels.sort(function(a, b) { return a - b; });
|
||||
uniqueLabels.forEach(function(label) {
|
||||
var color = label === -1 ? 'rgba(150,150,150,0.8)' : colors[Math.abs(label) % colors.length];
|
||||
var item = document.createElement('span');
|
||||
item.className = 'legend-item';
|
||||
item.innerHTML = '<span class="legend-swatch" style="background:' + color + '"></span>' + (label === -1 ? 'Noise' : 'Cluster ' + label);
|
||||
legendEl.appendChild(item);
|
||||
});
|
||||
}
|
||||
|
||||
function initScatter3D() {
|
||||
if (scatter3DInitialized) return;
|
||||
var container = document.getElementById('scatter3D');
|
||||
if (!container || scatterData.length === 0) return;
|
||||
var W = container.clientWidth || 600;
|
||||
var H = 500;
|
||||
|
||||
// Normalize coordinates to [-2, 2] cube
|
||||
var xs = scatterData.map(function(d) { return d.x; });
|
||||
var ys = scatterData.map(function(d) { return d.y; });
|
||||
var zs = scatterData.map(function(d) { return d.z || 0; });
|
||||
var xMin = Math.min.apply(null, xs), xMax = Math.max.apply(null, xs);
|
||||
var yMin = Math.min.apply(null, ys), yMax = Math.max.apply(null, ys);
|
||||
var zMin = Math.min.apply(null, zs), zMax = Math.max.apply(null, zs);
|
||||
var cx = (xMin + xMax) / 2, cy = (yMin + yMax) / 2, cz = (zMin + zMax) / 2;
|
||||
var range = Math.max(xMax - xMin, yMax - yMin, zMax - zMin) || 1;
|
||||
|
||||
function norm(v, center, r) { return ((v - center) / r) * 2; }
|
||||
|
||||
// Scene
|
||||
var scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0xf8f9fa);
|
||||
scatter3DScene = scene;
|
||||
|
||||
var camera = new THREE.PerspectiveCamera(55, W / H, 0.1, 50);
|
||||
camera.position.set(3.5, 2.5, 5);
|
||||
camera.lookAt(0, 0, 0);
|
||||
scatter3DCamera = camera;
|
||||
|
||||
var renderer = new THREE.WebGLRenderer({ antialias: true, alpha: false });
|
||||
renderer.setSize(W, H);
|
||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
||||
container.appendChild(renderer.domElement);
|
||||
scatter3DRenderer = renderer;
|
||||
|
||||
// Points geometry
|
||||
var positions = new Float32Array(scatterData.length * 3);
|
||||
var pointColors = new Float32Array(scatterData.length * 3);
|
||||
|
||||
for (var i = 0; i < scatterData.length; i++) {
|
||||
var d = scatterData[i];
|
||||
positions[i * 3] = norm(d.x, cx, range);
|
||||
positions[i * 3 + 1] = norm(d.y, cy, range);
|
||||
positions[i * 3 + 2] = norm((d.z || 0), cz, range);
|
||||
var hex = d.label === -1 ? '#999999' : colors[Math.abs(d.label) % colors.length];
|
||||
var c3 = new THREE.Color(hex);
|
||||
pointColors[i * 3] = c3.r;
|
||||
pointColors[i * 3 + 1] = c3.g;
|
||||
pointColors[i * 3 + 2] = c3.b;
|
||||
}
|
||||
|
||||
var geometry = new THREE.BufferGeometry();
|
||||
geometry.setAttribute('position', new THREE.BufferAttribute(positions, 3));
|
||||
geometry.setAttribute('color', new THREE.BufferAttribute(pointColors, 3));
|
||||
|
||||
// Create circular point sprite
|
||||
var spriteCanvas = document.createElement('canvas');
|
||||
spriteCanvas.width = 32; spriteCanvas.height = 32;
|
||||
var sctx = spriteCanvas.getContext('2d');
|
||||
sctx.beginPath(); sctx.arc(16, 16, 14, 0, Math.PI * 2);
|
||||
sctx.fillStyle = '#ffffff'; sctx.fill();
|
||||
var spriteTexture = new THREE.CanvasTexture(spriteCanvas);
|
||||
|
||||
var material = new THREE.PointsMaterial({
|
||||
size: 0.12,
|
||||
map: spriteTexture,
|
||||
vertexColors: true,
|
||||
sizeAttenuation: true,
|
||||
depthWrite: true,
|
||||
blending: THREE.NormalBlending,
|
||||
transparent: true,
|
||||
opacity: 0.85
|
||||
});
|
||||
|
||||
var points = new THREE.Points(geometry, material);
|
||||
scene.add(points);
|
||||
scatter3DPoints = points;
|
||||
|
||||
// Mouse rotation
|
||||
var isDragging = false, prevMouse = { x: 0, y: 0 };
|
||||
renderer.domElement.addEventListener('mousedown', function(e) {
|
||||
isDragging = true;
|
||||
prevMouse = { x: e.clientX, y: e.clientY };
|
||||
});
|
||||
window.addEventListener('mouseup', function() { isDragging = false; });
|
||||
window.addEventListener('mousemove', function(e) {
|
||||
if (!isDragging || !is3DMode) return;
|
||||
var dx = e.clientX - prevMouse.x;
|
||||
var dy = e.clientY - prevMouse.y;
|
||||
points.rotation.y += dx * 0.005;
|
||||
points.rotation.x += dy * 0.005;
|
||||
points.rotation.x = Math.max(-Math.PI / 2, Math.min(Math.PI / 2, points.rotation.x));
|
||||
prevMouse = { x: e.clientX, y: e.clientY };
|
||||
});
|
||||
|
||||
// Scroll zoom
|
||||
renderer.domElement.addEventListener('wheel', function(e) {
|
||||
if (!is3DMode) return;
|
||||
e.preventDefault();
|
||||
camera.position.z += e.deltaY * 0.008;
|
||||
camera.position.z = Math.max(1.5, Math.min(15, camera.position.z));
|
||||
}, { passive: false });
|
||||
|
||||
// Touch support
|
||||
renderer.domElement.addEventListener('touchstart', function(e) {
|
||||
if (e.touches.length === 1) {
|
||||
isDragging = true;
|
||||
prevMouse = { x: e.touches[0].clientX, y: e.touches[0].clientY };
|
||||
}
|
||||
});
|
||||
window.addEventListener('touchmove', function(e) {
|
||||
if (!isDragging || !is3DMode) return;
|
||||
var dx = e.touches[0].clientX - prevMouse.x;
|
||||
var dy = e.touches[0].clientY - prevMouse.y;
|
||||
points.rotation.y += dx * 0.005;
|
||||
points.rotation.x += dy * 0.005;
|
||||
points.rotation.x = Math.max(-Math.PI / 2, Math.min(Math.PI / 2, points.rotation.x));
|
||||
prevMouse = { x: e.touches[0].clientX, y: e.touches[0].clientY };
|
||||
});
|
||||
window.addEventListener('touchend', function() { isDragging = false; });
|
||||
|
||||
function animate() {
|
||||
if (!is3DMode) { requestAnimationFrame(animate); return; }
|
||||
requestAnimationFrame(animate);
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
animate();
|
||||
scatter3DInitialized = true;
|
||||
}
|
||||
|
||||
// ── 2D / 3D toggle ──
|
||||
(function() {
|
||||
var supports3D = hasWebGL2();
|
||||
if (supports3D) {
|
||||
document.getElementById('viewToggle').style.display = 'inline';
|
||||
}
|
||||
|
||||
var btn2D = document.getElementById('btn2D');
|
||||
var btn3D = document.getElementById('btn3D');
|
||||
var container2D = document.getElementById('scatter2DContainer');
|
||||
var container3D = document.getElementById('scatter3DContainer');
|
||||
|
||||
function show2D() {
|
||||
is3DMode = false;
|
||||
container2D.style.display = '';
|
||||
container3D.style.display = 'none';
|
||||
btn2D.style.fontWeight = 'bold';
|
||||
btn2D.className = 'btn btn-sm';
|
||||
btn3D.style.fontWeight = 'normal';
|
||||
btn3D.className = 'btn btn-sm btn-outline';
|
||||
// Re-draw 2D canvas (may have resized)
|
||||
drawScatter('scatterChart', scatterData, 'UMAP-1', 'UMAP-2');
|
||||
}
|
||||
|
||||
function show3D() {
|
||||
is3DMode = true;
|
||||
container2D.style.display = 'none';
|
||||
container3D.style.display = '';
|
||||
btn3D.style.fontWeight = 'bold';
|
||||
btn3D.className = 'btn btn-sm';
|
||||
btn2D.style.fontWeight = 'normal';
|
||||
btn2D.className = 'btn btn-sm btn-outline';
|
||||
initScatter3D();
|
||||
build3DLegend();
|
||||
}
|
||||
|
||||
btn2D.addEventListener('click', show2D);
|
||||
btn3D.addEventListener('click', show3D);
|
||||
|
||||
// Resize 3D renderer on window resize
|
||||
window.addEventListener('resize', function() {
|
||||
if (!scatter3DRenderer || !is3DMode) return;
|
||||
var container = document.getElementById('scatter3D');
|
||||
if (!container) return;
|
||||
var W = container.clientWidth || 600;
|
||||
var H = 500;
|
||||
scatter3DRenderer.setSize(W, H);
|
||||
scatter3DCamera.aspect = W / H;
|
||||
scatter3DCamera.updateProjectionMatrix();
|
||||
});
|
||||
})();
|
||||
|
||||
function drawScatter(canvasId, data, xLabel, yLabel) {
|
||||
const canvas = document.getElementById(canvasId);
|
||||
if (!canvas || data.length === 0) return;
|
||||
|
||||
+124
-14
@@ -104,6 +104,22 @@
|
||||
.tool-call-body pre code {
|
||||
font-family: 'Cascadia Code', 'Fira Code', 'Consolas', monospace;
|
||||
}
|
||||
|
||||
/* Dataset selector table */
|
||||
.ds-table { width:100%; border-collapse:collapse; font-size:0.85rem; }
|
||||
.ds-table th { background:#f5f5f5; padding:0.4rem 0.5rem; text-align:left; font-weight:600; border-bottom:2px solid #ddd; position:sticky; top:0; }
|
||||
.ds-table td { padding:0.35rem 0.5rem; border-bottom:1px solid #eee; }
|
||||
.ds-table tr { cursor:pointer; transition: background 0.15s; }
|
||||
.ds-table tr:hover { background:#e8f0fe; }
|
||||
.ds-table tr.selected { background:#d4e6ff; font-weight:600; }
|
||||
.status-badge { display:inline-block; padding:0.15rem 0.5rem; border-radius:3px; font-size:0.75rem; font-weight:600; }
|
||||
.status-badge.ready { background:#e8f5e9; color:#2e7d32; }
|
||||
.status-badge.analyzing { background:#fff3e0; color:#e65100; }
|
||||
.status-badge.completed { background:#e3f2fd; color:#1565c0; }
|
||||
.status-badge.failed { background:#ffebee; color:#c62828; }
|
||||
.filter-btn { padding:0.3rem 0.75rem; border:1px solid #ccc; border-radius:4px; background:#fafafa; cursor:pointer; font-size:0.8rem; transition: all 0.15s; }
|
||||
.filter-btn.active { background:#4361ee; color:#fff; border-color:#4361ee; }
|
||||
.filter-btn:hover:not(.active) { background:#e8e8e8; }
|
||||
</style>
|
||||
|
||||
<div class="card">
|
||||
@@ -116,6 +132,22 @@
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<!-- Dataset selector with status badges -->
|
||||
<div class="card" id="datasetSelectorCard">
|
||||
<h3>📁 选择数据集</h3>
|
||||
<div style="display:flex;gap:0.5rem;margin-bottom:0.75rem;" id="statusFilter">
|
||||
<button class="btn filter-btn active" data-filter="all">全部</button>
|
||||
<button class="btn filter-btn" data-filter="ready">待分析</button>
|
||||
<button class="btn filter-btn" data-filter="analyzing">分析中</button>
|
||||
<button class="btn filter-btn" data-filter="completed">已分析</button>
|
||||
<button class="btn filter-btn" data-filter="failed">错误</button>
|
||||
</div>
|
||||
<div id="datasetTable" style="max-height:350px;overflow-y:auto;border:1px solid #e0e0e0;border-radius:4px;">
|
||||
<p style="text-align:center;color:#999;padding:1rem;">加载中...</p>
|
||||
</div>
|
||||
<button class="btn btn-primary" style="margin-top:1rem;" id="startAutoBtn" onclick="startAuto()" {% if not llm_configured %}disabled{% endif %}>开始自动分析</button>
|
||||
</div>
|
||||
|
||||
<div id="autoProgress" style="display:none;margin-top:1rem;">
|
||||
<p>LLM 编排中... <span id="autoStatus"></span></p>
|
||||
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
|
||||
@@ -136,6 +168,79 @@
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const DATASETS_WITH_STATUS = {{ datasets_with_status_json|safe }};
|
||||
let activeFilter = 'all';
|
||||
let selectedRunId = null;
|
||||
|
||||
// ── Dataset table rendering ──
|
||||
function buildDatasetTable(filter) {
|
||||
const container = document.getElementById('datasetTable');
|
||||
let rows = DATASETS_WITH_STATUS.filter(ds => {
|
||||
if (filter === 'all') return true;
|
||||
if (filter === 'ready') return ds.run_status === 'ready' || (!ds.run_status || ds.run_status === 'ready');
|
||||
if (filter === 'analyzing') return ['pending', 'loading', 'profiling', 'aggregating', 'clustering', 'extracting'].includes(ds.run_status);
|
||||
if (filter === 'completed') return ds.run_status === 'completed';
|
||||
if (filter === 'failed') return ds.run_status === 'failed';
|
||||
return true;
|
||||
});
|
||||
|
||||
if (!rows.length) {
|
||||
container.innerHTML = '<p style="text-align:center;color:#999;padding:1rem;">无匹配的数据集</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
let html = '<table class="ds-table"><thead><tr><th></th><th>ID</th><th>状态</th><th>行数</th><th>文件数</th><th>来源</th></tr></thead><tbody>';
|
||||
rows.forEach(ds => {
|
||||
const label = ds.status_label || '待分析';
|
||||
const badgeClass = (ds.run_status === 'completed') ? 'completed' :
|
||||
(ds.run_status === 'failed') ? 'failed' :
|
||||
(ds.run_status && ds.run_status !== 'ready') ? 'analyzing' : 'ready';
|
||||
const dispId = ds.display_id || ds.dataset_id;
|
||||
const rowCount = ds.row_count != null ? ds.row_count.toLocaleString() : '?';
|
||||
const fileCount = ds.file_count != null ? ds.file_count : '?';
|
||||
const source = ds.sqlite_table ? ('SQLite: ' + ds.sqlite_table) : (ds.csv_glob || '?');
|
||||
const checked = (selectedRunId === String(ds.dataset_id)) ? 'checked' : '';
|
||||
html += '<tr data-ds-id="' + ds.dataset_id + '" onclick="selectRow(\'' + ds.dataset_id + '\')" class="' + (selectedRunId === String(ds.dataset_id) ? 'selected' : '') + '">' +
|
||||
'<td><input type="radio" name="auto_run_id" value="' + ds.dataset_id + '" ' + checked + ' onclick="event.stopPropagation();selectRow(\'' + ds.dataset_id + '\')"></td>' +
|
||||
'<td style="font-weight:600;">#' + dispId + '</td>' +
|
||||
'<td><span class="status-badge ' + badgeClass + '">' + label + '</span></td>' +
|
||||
'<td>' + rowCount + '</td>' +
|
||||
'<td>' + fileCount + '</td>' +
|
||||
'<td style="max-width:180px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">' + escapeHtml(source) + '</td>' +
|
||||
'</tr>';
|
||||
});
|
||||
html += '</tbody></table>';
|
||||
container.innerHTML = html;
|
||||
}
|
||||
|
||||
function selectRow(dsId) {
|
||||
selectedRunId = String(dsId);
|
||||
// Update radio buttons and row highlight
|
||||
document.querySelectorAll('#datasetTable input[name="auto_run_id"]').forEach(r => {
|
||||
r.checked = (r.value === String(dsId));
|
||||
});
|
||||
document.querySelectorAll('#datasetTable tr').forEach(tr => tr.classList.remove('selected'));
|
||||
const row = document.querySelector('#datasetTable tr[data-ds-id="' + dsId + '"]');
|
||||
if (row) row.classList.add('selected');
|
||||
}
|
||||
|
||||
function setDatasetFilter(filter) {
|
||||
activeFilter = filter;
|
||||
document.querySelectorAll('#statusFilter .filter-btn').forEach(btn => {
|
||||
btn.classList.toggle('active', btn.dataset.filter === filter);
|
||||
});
|
||||
buildDatasetTable(filter);
|
||||
}
|
||||
|
||||
// Init filter tabs
|
||||
document.querySelectorAll('#statusFilter .filter-btn').forEach(btn => {
|
||||
btn.addEventListener('click', function() {
|
||||
setDatasetFilter(this.dataset.filter);
|
||||
});
|
||||
});
|
||||
buildDatasetTable('all');
|
||||
|
||||
// ── Timeline rendering ──
|
||||
let statusPoll = null;
|
||||
|
||||
function escapeHtml(str) {
|
||||
@@ -154,10 +259,6 @@ function toggleStep(el) {
|
||||
if (body) body.classList.toggle('open');
|
||||
}
|
||||
|
||||
function formatTimestamp() {
|
||||
return new Date().toLocaleTimeString('zh-Hans', {hour:'2-digit',minute:'2-digit',second:'2-digit'});
|
||||
}
|
||||
|
||||
function buildTimeline(llmThinking, toolCalls) {
|
||||
// Build tool entries from tool_calls_json (no separate thought entries)
|
||||
const entries = (toolCalls || []).map(tc => ({
|
||||
@@ -195,22 +296,19 @@ function renderTimeline(entries) {
|
||||
if (!entries || entries.length === 0) return;
|
||||
|
||||
panel.style.display = 'block';
|
||||
// Clear and re-render entire timeline each poll
|
||||
container.innerHTML = '';
|
||||
let lastStep = -1;
|
||||
|
||||
for (const entry of entries) {
|
||||
// Step header
|
||||
if (entry.step !== lastStep) {
|
||||
const header = document.createElement('div');
|
||||
header.style.cssText = 'font-weight:700;font-size:0.85rem;color:#4361ee;padding:0.6rem 0 0.3rem 0;border-top:1px solid #eee;margin-top:0.4rem;';
|
||||
header.textContent = `步骤 ${entry.step}`;
|
||||
header.textContent = '步骤 ' + entry.step;
|
||||
container.appendChild(header);
|
||||
lastStep = entry.step;
|
||||
}
|
||||
|
||||
if (entry.type === 'tool') {
|
||||
// Tool call — collapsible with I/O; thinking shown when output is null
|
||||
const card = document.createElement('div');
|
||||
card.style.cssText = 'border:1px solid #e0e0e0;border-radius:6px;margin-bottom:0.3rem;overflow:hidden;background:#fff;';
|
||||
const inputStr = prettyJson(entry.input);
|
||||
@@ -244,8 +342,17 @@ function renderTimeline(entries) {
|
||||
}
|
||||
|
||||
async function startAuto() {
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
const runId = params.get('run_id');
|
||||
// Read run_id from radio buttons (primary method)
|
||||
const radio = document.querySelector('input[name="auto_run_id"]:checked');
|
||||
let runId;
|
||||
if (radio) {
|
||||
runId = radio.value;
|
||||
} else {
|
||||
// Fallback: URL parameter
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
runId = params.get('run_id');
|
||||
}
|
||||
|
||||
if (!runId) {
|
||||
document.getElementById('noRunMessage').style.display = 'block';
|
||||
return;
|
||||
@@ -258,6 +365,7 @@ async function startAuto() {
|
||||
document.getElementById('noRunMessage').style.display = 'none';
|
||||
document.getElementById('autoProgress').style.display = 'block';
|
||||
document.getElementById('cancelBtn').style.display = 'inline-block';
|
||||
document.getElementById('startAutoBtn').disabled = true;
|
||||
timelinePanel.style.display = 'none';
|
||||
timelineContainer.innerHTML = '';
|
||||
|
||||
@@ -275,11 +383,11 @@ async function startAuto() {
|
||||
|
||||
statusPoll = setInterval(async () => {
|
||||
try {
|
||||
const r = await fetch(`/runs/${runId}/status/`);
|
||||
const r = await fetch('/runs/' + runId + '/status/');
|
||||
const s = await r.json();
|
||||
const pct = s.progress_pct || 0;
|
||||
const msg = s.progress_msg || s.status;
|
||||
document.getElementById('autoStatus').textContent = `${msg} (${pct}%)`;
|
||||
document.getElementById('autoStatus').textContent = msg + ' (' + pct + '%)';
|
||||
document.getElementById('autoBar').style.width = pct + '%';
|
||||
|
||||
// Build and render unified timeline
|
||||
@@ -310,13 +418,15 @@ function cancelAnalysis() {
|
||||
if (statusPoll) { clearInterval(statusPoll); statusPoll = null; }
|
||||
document.getElementById('autoStatus').textContent = '已取消';
|
||||
document.getElementById('cancelBtn').style.display = 'none';
|
||||
document.getElementById('timelinePanel').innerHTML += '\n--- 分析已取消 ---\n';
|
||||
document.getElementById('startAutoBtn').disabled = false;
|
||||
}
|
||||
|
||||
// Auto-start if run_id is provided via URL parameter
|
||||
(function() {
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
if (params.get('run_id')) {
|
||||
const rid = params.get('run_id');
|
||||
if (rid) {
|
||||
selectRow(rid);
|
||||
startAuto();
|
||||
}
|
||||
})();
|
||||
|
||||
@@ -346,15 +346,50 @@ function collectParams(stepIdx) {
|
||||
return params;
|
||||
}
|
||||
|
||||
// ── Validate required params, show inline errors ──
|
||||
function validateParams(stepIdx) {
|
||||
const step = workflow.find(s => s.idx === stepIdx);
|
||||
if (!step) return true;
|
||||
const required = step.tool.inputSchema?.required || [];
|
||||
// Clear previous inline errors
|
||||
document.querySelectorAll(`.validation-error`).forEach(el => el.remove());
|
||||
let valid = true;
|
||||
required.forEach(key => {
|
||||
const el = document.querySelector(`.p-${stepIdx}[data-key="${key}"]`);
|
||||
const val = el ? el.value : '';
|
||||
if (!val || (typeof val === 'string' && val.trim() === '')) {
|
||||
valid = false;
|
||||
if (el) {
|
||||
el.style.border = '2px solid #dc3545';
|
||||
el.style.background = '#fff5f5';
|
||||
const errSpan = document.createElement('span');
|
||||
errSpan.className = 'validation-error';
|
||||
errSpan.style.cssText = 'color:#dc3545;font-size:0.75rem;display:block;margin-top:2px;';
|
||||
errSpan.textContent = `必填字段 "${key}" 不能为空`;
|
||||
el.parentNode.appendChild(errSpan);
|
||||
}
|
||||
}
|
||||
});
|
||||
if (!valid) showError('请填写所有必填参数（红色标注）');
|
||||
return valid;
|
||||
}
|
||||
|
||||
// ── Execute single step ──
|
||||
async function executeStep(idx) {
|
||||
const step = workflow.find(s => s.idx === idx);
|
||||
if (!step) return;
|
||||
|
||||
// Collect params from DOM BEFORE re-rendering (renderSteps destroys/recreates DOM)
|
||||
const params = collectParams(idx);
|
||||
|
||||
// Frontend validation: check required fields
|
||||
if (!validateParams(idx)) return;
|
||||
|
||||
step.params = params;
|
||||
step.status = 'running'; renderSteps();
|
||||
const card = document.getElementById(`step-${idx}`);
|
||||
const resultDiv = document.getElementById(`result-${idx}`);
|
||||
resultDiv.style.display = 'none';
|
||||
const params = collectParams(idx);
|
||||
|
||||
// Raw row-level clustering: skip entity aggregation entirely
|
||||
if (step.tool.name === 'build_entity_profiles') {
|
||||
@@ -708,5 +743,24 @@ async function applyFilter() {
|
||||
onDatasetChange();
|
||||
}
|
||||
})();
|
||||
|
||||
// ── Real-time param sync: delegated change listener ──
|
||||
document.getElementById('workflowSteps').addEventListener('change', function(e) {
|
||||
const el = e.target;
|
||||
if (!el.classList) return;
|
||||
// Find step index from class "p-N"
|
||||
const match = el.className && el.className.match(/\bp-(\d+)\b/);
|
||||
if (!match) return;
|
||||
const stepIdx = parseInt(match[1]);
|
||||
const step = workflow.find(s => s.idx === stepIdx);
|
||||
if (!step) return;
|
||||
step.params = collectParams(stepIdx);
|
||||
// Clear validation styling if user fixes a field
|
||||
el.style.border = '';
|
||||
el.style.background = '';
|
||||
const errSpan = el.parentNode && el.parentNode.querySelector('.validation-error');
|
||||
if (errSpan) errSpan.remove();
|
||||
});
|
||||
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -13,7 +13,9 @@
|
||||
<div style="font-size:2rem;margin-bottom:0.5rem;">📂</div>
|
||||
<p style="margin-bottom:1rem;color:#666;">拖拽 CSV 文件到此处</p>
|
||||
<input type="file" name="files" multiple accept=".csv,.zip" style="display:none;" id="fileInput">
|
||||
<input type="file" directory webkitdirectory style="display:none;" id="dirInput">
|
||||
<button type="button" class="btn btn-primary" id="fileSelectBtn">📄 选择CSV文件</button>
|
||||
<button type="button" class="btn" style="background:#43aa8b;color:#fff;" id="dirSelectBtn">📁 选择整个目录</button>
|
||||
</div>
|
||||
<div id="submitBar" style="margin-top:1rem;display:none;">
|
||||
<button type="submit" class="btn btn-primary" id="submitBtn">开始上传</button>
|
||||
@@ -59,6 +61,7 @@
|
||||
<script>
|
||||
const dropZone = document.getElementById('dropZone');
|
||||
const fileInput = document.getElementById('fileInput');
|
||||
const dirInput = document.getElementById('dirInput');
|
||||
const fileList = document.getElementById('fileList');
|
||||
const submitBar = document.getElementById('submitBar');
|
||||
const submitBtn = document.getElementById('submitBtn');
|
||||
@@ -69,6 +72,8 @@ dropZone.addEventListener('dragleave', () => dropZone.style.borderColor = '#ccc'
|
||||
dropZone.addEventListener('drop', e => { e.preventDefault(); handleFiles(e.dataTransfer.files); });
|
||||
document.getElementById('fileSelectBtn').addEventListener('click', () => fileInput.click());
|
||||
fileInput.addEventListener('change', () => handleFiles(fileInput.files));
|
||||
document.getElementById('dirSelectBtn').addEventListener('click', () => dirInput.click());
|
||||
dirInput.addEventListener('change', function(e) { handleFiles(this.files); });
|
||||
|
||||
function handleFiles(files) {
|
||||
selectedFiles = [...files];
|
||||
|
||||
+1
-1
@@ -39,7 +39,7 @@ if os.path.exists(db_path):
|
||||
# Restore previously uploaded datasets whose CSV files still exist.
|
||||
try:
|
||||
from analysis.session_store import SessionStore
|
||||
count = SessionStore().restore_from_disk()
|
||||
count = SessionStore().start_async_restore()
|
||||
if count > 0:
|
||||
import logging
|
||||
logging.getLogger('django').info('Restored %d datasets from disk', count)
|
||||
|
||||
Reference in New Issue
Block a user