HJQ/tianxuan
2
1
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

chore: initial commit of 天璇 project state before cleanup

Browse Source
This commit is contained in:
PM-pinou
2026-07-16 22:41:26 +08:00
commit 814f29645a
104 changed files with 36735 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+10
View File
@@ -0,0 +1,10 @@
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
# Virtual environments
.venv
.omo/drafts/tianxiu-v5.md
+20
View File
@@ -0,0 +1,20 @@
# Draft: 天璇 v5 — 18 Issues Fix Plan
## Intent
- **intent**: CLEAR
- **review_required**: true
- **status**: awaiting-approval
## Issue Groups (18 problems → 8 workstreams)
### A. 上传管理修复 (Issues 1, 2, 3, 10, 11)
### B. 列检测与类型分类修复 (Issues 7, 15, 18)
### C. Traceback截断修复 (Issue 4)
### D. 大规模数据处理 (Issues 5, 6, 14, 16)
### E. Globe可视化修复 (Issues 8, 9)
### F. LLM日志 (Issue 12)
### G. 聚类可视化修复 (Issue 13)
### H. 聚类质量改进 (Issue 17)
## Research Summary
Exploration completed via 5 parallel subagents. Full findings in task outputs.
.omo/drafts/tianxiu-v6.md
+341
View File
@@ -0,0 +1,341 @@
# Draft: 天璇 v6 — 基于真实TLS特征的恶意流量分析
## Intent
- **intent**: CLEAR
- **review_required**: true
- **status**: awaiting-approval
## 自我审查发现的问题
1. ❌ Phase 0 依赖用户给真实数据——应基于 TLS 1.3 协议推导
2. ❌ 21x2 hex 列我不知道是哪个，回避了问题
3. ❌ "TlsS"/"TlsC" 列我不知道是哪个，回避了问题
4. ❌ 归一化步骤没说明如何集成到现有 data_loader 中
5. ❌ 特征间有冗余（port_category vs is_standard_port）
6. ❌ 时域特征 `inter_arrival_time_std` 在单流级别不可计算
7. ❌ 缺少 JA3 指纹近似、SNI 长度熵等关键特征
8. ❌ lat/lon 覆盖方案没说具体怎么做
9. ❌ 空白语义只说了 4 列，其他几十列没说
10. ❌ 聚类流程图漏了"归一化"步骤
---
## Phase 0: 基于 TLS 1.3 协议的列类型推导（不需要用户数据）
### 基础事实
- 数据来源：TLS 1.3 握手报文字段抽取
- 每条流 = 一个 TLS 连接握手 + 数据传输的摘要
- 大量空白是预期的——TLS 握手中很多字段是可选 extension，不是每条流都有
### 每列精确类型定义
| 列 | TLS协议来源 | 存放格式 | 空白语义 |
|----|-----------|---------|---------|
| `:ips` | 捕获点记录的源IP | IPv4字符串 | 从不空白 |
| `:ipd` | 捕获点记录的目标IP | IPv4字符串 | 从不空白 |
| `:prs` | TCP源端口 | 整数 | 缺失 |
| `:prd` | TCP目标端口 | 整数 | 缺失 |
| `scnt` | GeoIP从:ipd推断 | 2字母大写代码 | 缺失（GeoIP无法解析时） |
| `dcnt` | GeoIP从:ipd推断 | 2字母大写代码 | 缺失 |
| `server-ip` | 服务器IP（可能与:ipd同） | IPv4 | 缺失 |
| `client-ip` | 客户端IP（可能与:ips同） | IPv4 | 缺失 |
| `0ver` | ServerHello.protocol_version | 2x2 hex: `03 04` | TLS连接都必有，极少空白 |
| `snam` | ClientHello.extensions.server_name | UTF8域名 | 缺失（ClientHello中无SNI扩展 → 可疑） |
| `cnam` | Certificate.certificate.subject.CN | UTF8字符串 | 缺失（无证书或未知证书） |
| `4dur` | 流持续时间（纳秒/毫秒?） | 整数 | 常见空白（40%+） |
| `8ses` | 会话起始偏移 | 整数 | 常见空白 |
| `2tmo` | 超时偏移 | 整数 | 常见空白 |
| `4ksz` | 证书公钥大小 | 整数(256/384/521/1024/2048) | 空白（无证书） |
| `cnrs` | SessionTicket.can_resume | `+`/空白 | **空白≠缺失** `+`=可恢复 `""`=不可恢复 |
| `isrs` | SessionTicket.was_resumed | `+`/空白 | **空白≠缺失** `+`=已恢复 `""`=未恢复 |
| `8ack` | 反向方向字节数 | 整数 | 缺失 |
| `8ppk` | 有效载荷包数 | 整数 | 缺失 |
| `8dbd` | 数据库日志记录时间 | 时间戳字符串 | 缺失 |
| `1ipp` | IP协议号 | 整数(6/17) | 缺失 |
| `4dbn` | 数据库编号 | 整数 | 缺失 |
| `tabl` | 数据库表名 | 字符串 | 缺失 |
| `name` | 捕获链路名称 | 字符串 | 缺失 |
| `source-node` | 捕获节点名 | 字符串 | 缺失 |
| `cipher-suite` | 密码套件名（IANA格式） | 大小写混合字符串 | 缺失（无密码协商时） |
| `ecdhe-named-curve` | 椭圆曲线名 | 大小写混合字符串 | 缺失（非ECDHE密钥交换时） |
| `0cph` | ServerHello.cipher_suite | 2x2 hex: `13 01` | 缺失 |
| `0crv` | ServerHello.KeyShare | 2x2 hex: `00 1d` | 缺失（非ECDHE时） |
| `0rnd` | ServerHello/ClientHello.random | 32x2 hex（也可能是21x2） | TLS必有，但可能截断存储 |
| `0rnt` | Random.gmt_unix_time | 整数或4x2 hex | 常见空白（现代TLS用随机值替代时间） |
| `row` | 行号 | 整数 | 从不空白 |
| `time` | 时间 | 字符串 | 缺失 |
| `timestamp` | 时间戳 | 字符串 | 从不空白 |
| `:ips.latd/lond` | GeoIP推断，过时 | 浮点数 | 不可靠，覆盖 |
| `:ipd.latd/lond` | GeoIP推断，过时 | 浮点数 | 不可靠，覆盖 |
| `:ips.ispn/.orgn/.city` | GeoIP推断 | 字符串 | 大部分空白 |
| `:ipd.ispn/.orgn/.city` | GeoIP推断 | 字符串 | 大部分空白 |
### 待识别项目
- **21x2 hex 列**: 不在我已知的任何标准 TLS 结构中。0rnd(32x2)、0cph(2x2)、0crv(2x2)、0ver(2x2)。21x2 可能是一个额外的截断/编码字段。将用 column_survey 自动发现 `hex_pairs_count==21` 的列。
- **TlsS/TlsC**: 可能是 `tabl` 或 `source-node` 列的两个值。含义接近 "TLS Server" / "TLS Client"。待 column_survey 确认。
---
## Phase 0.5: 值归一化模块 `analysis/value_normalizer.py`
### 架构
新增独立的归一化模块，在 `data_loader.py` 的 `load_csv_directory()` 中作为**最后一个清理步骤**执行（类型分类之后、返回 LazyFrame 之前）。
### 归一化方式
每列生成 `{col}_norm` 保留原始列，供下游特征工程使用。
### TLS Version → Enum (0ver)
```python
TLS_VERSION_MAP = {
# hex wire format
'03 04': 'tls_1_3', '03 03': 'tls_1_2',
'03 02': 'tls_1_1', '03 01': 'tls_1_0',
'02 00': 'ssl_2_0', '03 00': 'ssl_3_0',
# display formats (case-insensitive)
'tlsv1.3': 'tls_1_3', 'tls 1.3': 'tls_1_3', '1.3': 'tls_1_3',
'tlsv1.2': 'tls_1_2', 'tls 1.2': 'tls_1_2', '1.2': 'tls_1_2',
}
# 未知值 → 'unknown_version'
```
### Cipher Suite → Enum (0cph, cipher-suite)
```python
CIPHER_MAP = {
# hex → IANA name (TLS 1.3 + common TLS 1.2)
'13 01': 'tls_aes_128_gcm_sha256',
'13 02': 'tls_aes_256_gcm_sha384',
'13 03': 'tls_chacha20_poly1305_sha256',
'c0 2b': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
'c0 2f': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
'c0 30': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
'cc a9': 'tls_ecdhe_ecdsa_chacha20_poly1305_sha256',
'cc ac': 'tls_ecdhe_rsa_chacha20_poly1305_sha256',
'00 9c': 'tls_rsa_aes_128_gcm_sha256',
'00 9d': 'tls_rsa_aes_256_gcm_sha384',
'c0 09': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
'c0 13': 'tls_ecdhe_rsa_aes_128_cbc_sha256',
# display → IANA (normalize: lowercase, ' '→'_', '-'→'_')
'tls_aes_128_gcm_sha256': 'tls_aes_128_gcm_sha256',
'aes 128 gcm': 'tls_aes_128_gcm_sha256',
'aes-128-gcm': 'tls_aes_128_gcm_sha256',
'tls 1.3 aes 128 gcm': 'tls_aes_128_gcm_sha256',
}
# 未知hex → 'unknown_cipher_0x{hex}'
# 未知显示名 → 标准化后保留
```
### Named Curve → Enum (0crv, ecdhe-named-curve)
```python
CURVE_MAP = {
# hex wire format
'00 1d': 'x25519', '00 17': 'secp256r1',
'00 18': 'secp384r1', '00 19': 'secp521r1',
'00 1e': 'x448',
# display (case-insensitive)
'x25519': 'x25519', 'curve25519': 'x25519',
'secp256r1': 'secp256r1', 'prime256v1': 'secp256r1',
'p-256': 'secp256r1', 'p256': 'secp256r1',
'secp384r1': 'secp384r1', 'p-384': 'secp384r1',
}
# 未知 → 'unknown_curve_0x{hex}'
```
### CNRS/ISRS → Enum (cnrs, isrs)
```
'+' → 'yes' (never missing)
'' → 'no' (never missing)
```
### 纯字符串（不操作）
`snam`, `cnam`, `name`, `source-node`, `tabl`
→ 不归一化，保持原始字符串
### 归一化与现有 data_loader 的集成
```
现有 data_loader 清理流水线:
BOOL_ENUM → LAT_LON → HEX → [新增: VALUE_NORMALIZATION] → GenericNumCoerce
value_normalizer.normalize(lf, type_map) → pl.LazyFrame (带 _norm 列)
```
---
## Phase 1: 特征工程（每条流级别 → 实体聚合级别）
### 第一步：单流特征（在 entity_aggregator 聚合前计算）
| 特征 | 来源列 | 计算方式 |
|------|--------|---------|
| `has_sni` | `snam` | 有SNI? 1/0 |
| `sni_length` | `snam` | SNI域名长度（长域名可疑） |
| `sni_entropy` | `snam` | 域名Shannon熵（高熵域名=算法生成=DGA） |
| `sni_vs_cnam` | `snam`, `cnam` | SNI与CN一致? 1/0 |
| `tls_version_risk` | `0ver_norm` | `<tls_1_2`=3, `tls_1_2`=1, `tls_1_3`=0 |
| `cipher_suite_category` | `0cph_norm` | `aead`=0, `cbc`=1, `rc4`=2, `null`=3 |
| `key_exchange` | `0cph_norm` | `ecdhe`=0, `dhe`=1, `static`=2 |
| `is_modern_cipher` | `0cph_norm` | tls 1.3 only cipher? 1/0 |
| `curve_type` | `0crv_norm` | `x25519`=0, `secp256r1`=1, other=2 |
| `curve_bit_strength` | `0crv_norm`/`4ksz` | 128/192/256 bits |
| `port_type` | `:prd` | `443`=0, `80-1023`=1, `>1024`=2 |
| `bytes_per_packet` | `8ack`, `8ppk` | avg bytes/packet |
| `is_recoverable` | `cnrs_norm` | `yes`=1, `no`=0 |
| `was_resumed` | `isrs_norm` | `yes`=1, `no`=0 |
| `has_cert_key` | `4ksz` | 有密钥大小? 1/0 |
### 第二步：实体聚合特征（在 entity_aggregator 中计算，按 `:ips` 分组）
| 聚合特征 | 聚合方式 | 意义 |
|---------|---------|------|
| `flow_count` | count | 总流数 |
| `unique_dst_ips` | n_unique | 目标多样性 |
| `unique_dst_ports` | n_unique | 端口多样性 |
| `unique_tls_versions` | n_unique(0ver_norm) | TLS版本跨度 |
| `unique_ciphers` | n_unique(0cph_norm) | 密码套件跨度 |
| `unique_curves` | n_unique(0crv_norm) | 曲线跨度 |
| `unique_snis` | n_unique(snam) | SNI多样性 |
| `tls_13_ratio` | mean(is_modern_cipher) | TLS 1.3密码比例 |
| `ecdhe_ratio` | mean(key_exchange==0) | 前向保密比例 |
| `avg_bytes_per_packet` | mean(bytes_per_packet) | 平均包大小 |
| `std_bytes_per_packet` | std(bytes_per_packet) | 包大小标准差（小=beaconing） |
| `avg_duration` | mean(4dur) | 平均持续时长 |
| `std_duration` | std(4dur) | 时长标准差 |
| `port_443_ratio` | mean(port_type==0) | 标准端口比例 |
| `non_standard_port_ratio` | mean(port_type==2) | 非标准端口比例（可疑） |
| `sni_missing_ratio` | mean(1-has_sni) | 无SNI比例（可疑） |
| `sni_cn_mismatch_ratio` | mean(1-sni_vs_cnam) | SNI/CN不匹配比例 |
| `sni_avg_entropy` | mean(sni_entropy) | SNI平均熵（高=DGA） |
| `recoverable_ratio` | mean(is_recoverable) | 会话可恢复比例 |
| `resumed_ratio` | mean(was_resumed) | 会话恢复比例 |
| `has_any_cert` | mean(has_cert_key) | 有证书比例 |
| `countries_visited` | n_unique(dcnt) | 跨国多样性 |
| `unique_hours` | n_unique(hour_of_day) | 活跃时段 |
| `inter_arrival_std` | std(timestamp_diff) | **Beaconing检测**: 到达间隔标准差 |
| `night_ratio` | mean(is_night_hour) | 夜间流量比例 |
### 总计：约 30+ 聚合特征
---
## Phase 2: 空白语义精确处理
| 列 | 空白语义 | 聚合前处理 | 聚合中处理 |
|----|---------|-----------|-----------|
| cnrs, isrs | **非缺失** (+=yes, blank=no) | 归一化为 `yes`/`no` | 直接聚合 |
| snam, cnam | 缺失（无SNI/证书） | 保留空白 | `has_sni`=0, `sni_cn_mismatch`=null |
| 0ver | 极少空白 | 空白→`unknown_version` | `tls_version_risk`=null |
| 0cph, 0crv | 缺失（无协商） | 空白→`unknown` | 相应特征为null |
| 4dur, 8ack, 8ppk | 缺失（40%+） | 保留空白 | `mean`/`std`跳过null |
| :prs, :prd | 缺失 | 保留空白 | `port_type`=null |
| scnt, dcnt | 缺失（GeoIP失败） | 保留空白 | `countries_visited`排除null |
| lat/lon | 基于过时库 | **全部替换** | 见Phase 2.5 |
| ispn/orgn/city | 大部分空白 | **不参与分析** | 排除 |
---
## Phase 2.5: 经纬度覆盖方案
lat/lon 基于过时 IP 地理库，不可靠。处理方式：
- **不使用原始 lat/lon 值**
- 对于 globe 可视化：用随机生成的地理位置模拟
- 对于分析特征：移除所有基于 lat/lon 的特征
- 保留 `dcnt`（目标国家代码）作为地理特征——国家代码更可靠且更有分析价值
---
## Phase 3: 完整聚类流程
```
原始CSV
→ data_loader加载
→ 类型分类 (type_classifier.py)
→ BOOL_ENUM清洗
→ HEX预处理 (hex_pairs_count)
→ 值归一化 (value_normalizer.py) ← Phase 0.5 新增
→ 通用数值清洗
→ 单流特征计算 (data_loader.py 扩展) ← Phase 1 新增
→ 实体聚合 (entity_aggregator.py 扩展) ← Phase 1 新增
→ 通用数值清洗 (已有)
→ 方差过滤 ← Phase 1 新增
→ 相关过滤
→ StandardScaler → PCA(20-50维)
→ HDBSCAN聚类
→ 簇画像生成
```
### 簇画像输出
每个簇的完整描述：
```
簇标签: 3
大小: 42实体 (14%)
簇类型: 🚨 高度可疑
特征画像:
- sni_missing_ratio: 0.85 (+2.1σ 高于均值)
- non_standard_port_ratio: 0.72 (+1.8σ)
- tls_13_ratio: 0.15 (-2.3σ)
- avg_bytes_per_packet: 45 (-1.9σ, 小包=beaconing)
- inter_arrival_std: 2.3s (-2.1σ, 固定间隔)
- countries_visited: 12 (+1.5σ, 跨国)
判定: 疑似代理/隧道流量
建议操作: 深度包检测, IP列入观察名单
```
---
## Phase 4: 测试数据生成 (gen_test_data.py)
测试数据也使用归一化格式输出，确保端到端可验证。
### 3 类流量画像
#### 1. 正常浏览 (60%示例数据)
- TLS 1.3 (03 04) + AEAD cipher
- Curve: x25519 (00 1d)
- SNI: `www.google.com`, `api.github.com` 等合法域名
- 端口 443
- 多样的目标IP
- 短连接 (0.5-5s), 中等包大小
- cnrs/isrs: 随机 yes/no
#### 2. 代理/VPN 隧道 (20%)
- TLS 1.2 (03 03) + ECDHE cipher
- Curve: secp256r1 (00 17)
- SNI: 可能缺失或单一代理解析域名
- 端口 443 但可能也在非标准端口
- 少目标高流量
- 长连接, 大字节量
- cnrs: 高恢复率 (keep-alive)
#### 3. 恶意/可疑 (20%)
- TLS 1.0/1.1 或奇怪版本
- 弱密码 (CBC/RC4)
- SNI/CN 不匹配或无 SNI
- Beaconing: 固定间隔小包 (30-40 bytes)
- 高重连率 (大量 cnrs=yes → isrs=yes)
- 非标准端口
- 低熵 TLS random
- 多国目标分布
- 低包大小标准差
---
## Phase 5: 端到端验证
1. 生成3类测试数据
2. 跑完整管道
3. 验证归一化正确性（每列hex→enum映射）
4. 验证聚类结果的自然分簇（正常/代理/恶意）
5. 验证簇画像的可解读性
6. 单元测试: 120+ passed
---
## 前置依赖
- Phase 0-5 都不需要用户真实数据，全部基于 TLS 1.3 协议推导
## 意外预案
| 意外 | 后备 |
|------|------|
| 21x2 hex 列不在已知列中 | column_survey 自动发现后添加合适映射 |
| TlsS/TlsC 列无法确认含义 | 保留为原始字符串枚举，不做归一化 |
| 聚类仍不理想 | 增加规则引擎层（if sni_missing>0.5 → flag） |
| 特征数量不足 | 从 hex 列提取字节分布统计作为补充特征 |
| 归一化映射表不全 | 未知hex→`unknown_{type}_0x{hex}` 兜底 |
.omo/patches/globe-v6.md
+38
View File
@@ -0,0 +1,38 @@
# Globe v6 修复补丁
## P1. 背面剔除修复
**文件**: `templates/tianxuan/globe.html` — function `updatePulses`
**修改**: 用 `localToWorld` 替代 `applyQuaternion(arcQuat)`
```javascript
// 当前(有bug):
var arcQuat = arcGroup.quaternion;
...
midPt.applyQuaternion(arcQuat);
// 改为:
arcGroup.updateMatrixWorld(true);
...
arcGroup.localToWorld(midPt);
```
## P2. 国境线恢复
**文件**: `static/tianxuan/world_borders.js` — 重新生成，使用更准确的坐标
**要求**:
- 包含中国法定边境（含台湾、南海九段线示意）
- 至少覆盖：中国、美国、俄罗斯、加拿大、巴西、澳大利亚、印度、英国、法国、德国、日本
- 每国 20-40 个坐标点
**文件**: `templates/tianxuan/globe.html` — 恢复 borderGroup
- 恢复 script 加载 world_borders.js
- 恢复 borderGroup 创建和 renderBorders 函数
- 恢复复选框 `国境线` toggle
## P3. 聚类空列回退 (已修复)
`tool_registry.py` `_cluster_sync`: 已添加回退逻辑
.omo/plans/cleanup.md
+188
View File
@@ -0,0 +1,188 @@
# 清理计划
## 第一步：删掉我加的所有垃圾
> 前置条件：确认 `gen_test_data.py` 的原始版本在 git 历史中可恢复
> (`git log --oneline -- scripts/gen_test_data.py`)，如无 git 历史则先手动备份。
### 删除文件（全部显式列出，避免 glob 误伤）
```
# ── 我加的垃圾脚本 ──
scripts/wnl_to_user_csv.py # WNL转换器
scripts/eval_clustering.py # 聚类评估
scripts/debug_db2.py # 调试文件
scripts/debug_db3.py # 调试文件
scripts/debug_persistence.py # 调试文件（原计划遗漏）
scripts/test_orch.py # 编排测试
scripts/test_orch2.py # 编排测试
scripts/test_pca.py # PCA测试
scripts/test_profile.py # 画像测试
scripts/test_tool_loop.py # 工具测试
scripts/test_tool_format.py # 工具测试
scripts/test_e2e_full.py # 端到端测试
scripts/test_32b.py # 32B模型测试
scripts/test_chinese_path.py # 中文路径测试
scripts/verify_runtime.py # 运行时验证
# ── 我加的垃圾数据 ──
data/wnl_converted.csv # WNL转换数据
```
### 保留原始文件
```
scripts/gen_test_data.py # ★ 保留 — 需从 git 恢复原始版本
# git checkout HEAD~1 -- scripts/gen_test_data.py
# 如无 git 历史则手动重建
scripts/gen_complex_test.py # 原始
scripts/column_survey.py # 原始
scripts/diagnose_compare.py # 原始
scripts/debug_db.py # 原始(非debug_db2/3)
scripts/test_llm_full.py # 原始LLM测试
data/complex_test.csv # 原始测试设备
data/e2e_full.csv # 原始
data/e2e_test.csv # 原始
data/geoip_city.csv # 原始
data/globe_test.csv # 原始
data/test_flows.csv # 原始(需用原gen_test_data.py重新生成)
data/复杂测试.csv # 原始
```
## 第二步：恢复被搞坏的文件到可用状态
### settings.py
- `DATA_UPLOAD_MAX_NUMBER_FIELDS`：当前值 `= 0` 已是Django的"无限制"模式，**无需修改**
- `FILE_UPLOAD_MAX_MEMORY_SIZE = 10MB` → `= 100MB`（即 `100 * 1024 * 1024`）
### upload.html
- `handleFiles()` 中每个文件旁加 ✕ 删除按钮
- 添加 `removeFile(index)` JS 函数
### auto.html
- ⚠️ 当前 `auto.html` 中没有 `<form>` 元素——LLM 分析通过 JS 直接 `fetch('/analyze/llm/', ...)` 触发（第56行）
- **无需修改表单 action**；确认 JS 中 fetch URL 正确即可
### views.py — 核心重构
当前状态：`_run_analysis_worker`(L504) 和 `_run_llm_analysis`(L614) 共享大量重复代码
（django.setup / ORM get / try-except / 进度报告 / 日志）。
目标：提取共享样板代码到统一的 `_run_pipeline_worker`，将两个分析逻辑作为回调注入。
**新设计**：
```python
def _run_pipeline_worker(run_id, analysis_fn, **ctx):
"""统一后台工作者：负责 setup / ORM / 进度 / 错误处理"""
import django, os
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
os.environ['DJANGO_ALLOW_ASYNC_UNSAFE'] = 'true'
django.setup()
from analysis.models import AnalysisRun
from analysis import logger
run = AnalysisRun.objects.get(id=run_id)
try:
analysis_fn(run, ctx)
except Exception:
import traceback
tb = traceback.format_exc()
logger.error(tb)
run.status = 'failed'
run.error_message = tb
run.run_log += f'\n[FATAL] {tb}'
run.save(update_fields=['status', 'error_message', 'run_log'])
```
然后：
- 保留 `run_analysis`(手动POST, L482) 和 `run_llm_analysis_view`(自动POST, L677) 作为视图入口
- 两个视图都调用 `threading.Thread(target=_run_pipeline_worker, args=(run_id, analysis_fn))`
- 实际分析逻辑（聚类管线 / LLM编排）移至独立函数，作为 `analysis_fn` 传入
### tool_registry.py
- `_handle_extract_features`(L1091, `async def`) 中 `AR.objects.filter(csv_glob=...).order_by('-id').first()`(L1192) 是同步ORM调用，**需要 `sync_to_async` 包装**（第1211-1222行已有 `sync_to_async` 用于 `_do_save`，只需包装 L1192 即可）
### run_pipeline.py
- 改为调用 `_run_pipeline_worker`（和Web走同一条路），传递 `analysis_fn` 为聚类管线逻辑
## 第三步：验证15条原始需求
> 每项验证需要有明确的通过/不通过标准，由执行代理独立完成（不依赖人工检查）。
> 验证顺序：先命令行（5/6/14/15）→ 再浏览器自动化（其余项）。
| # | 需求 | 验证方式 | 通过标准 |
|---|------|---------|---------|
| 1 | 重启后旧数据可分析 | `run.bat` 启动 → 上传data/e2e_test.csv → 控制台重启 → 访问 `/runs/` 看旧记录可选 | `curl http://127.0.0.1:8000/runs/` 返回状态码200且含旧run的id |
| 2 | web图标所有页面 | 访问首页 `/` → 检查 `<link rel="icon">` 存在 | `favicon.ico` 或 `favicon.svg` 在 `<head>` 中存在 |
| 3 | 上传按钮在顶部 | 访问 `/upload/` → snap元素定位 | 上传按钮/区域在文件列表上方（DOM顺序在文件列表之前） |
| 4 | traceback完整 | 手动触发错误（如改坏config.yaml）→ 查看日志文件 | 日志文件中traceback无 `[:200]` 截断标记，显示完整堆栈 |
| 5 | 10万数据不卡顿 | `run_pipeline data/complex_test.csv` 完整运行 | 总运行时间 < 120秒，无 MemoryError |
| 6 | 进度条 | 上传后检查 `AnalysisRun.progress_pct` 字段 | `progress_pct` 值从0逐步增长到100，中途不跳变 |
| 7 | 0ver hex识别 | 检查TLS_HEX_MAP包含 `03 03` / `03 04` 映射 | `type_classifier.py` 中 `TLS_HEX_MAP` 包含这两个键 |
| 8 | globe响应式缩放 | 改变浏览器窗口宽度（1920→800）→ 检查canvas尺寸 | 3D canvas的width/height随窗口变化，不出现滚动条 |
| 9 | globe ?data= | 访问 `/globe/?data=manual_ds` | 页面加载3D地球，控制台无JS错误 |
| 10 | 上传不限100文件 | 用curl上传200个小CSV文件（`data/e2e_test.csv` x200） | 返回HTTP 200，非413 Payload Too Large |
| 11 | 选择文件列表有删除 | 检查 `upload.html` 中 `handleFiles()` 渲染逻辑 | 每文件条目有 `✕` 按钮和对应的 `removeFile(index)` 函数定义 |
| 12 | LLM过程打印 | 配置LLM → 触发自动分析 → 检查 `run_log` 字段 | `run_log` 包含步骤日志 `[1]`, `[2]`, ... `[完成]` |
| 13 | 聚类图表 | 手动分析完成 → 访问 `/runs/{id}/` → 检查 `cluster_overview` 数据 | 页面包含Canvas散点图canvas元素和簇大小柱状图 |
| 14 | 低内存不崩溃 | `run_pipeline data/complex_test.csv` 处理20000行 | 进程退出码0，峰值内存 < 2GB |
| 15 | +号字符串不被识别为数字 | 创建含`+`号的CSV（如 `+123` 列值）→ 上传 → 检查实体检测 | 实体检测不将 `+123` 识别为FLOAT类型（应为STRING） |
## 执行顺序与依赖关系
```
Step 1 (删垃圾) ──→ Step 2 (修文件) ──→ Step 3 (验证) ──→ Step 4 (最终清理)
│ │ │
│ ▼ │
└──── 4c (恢复gen_test_data) ◄────────────────┘
│
▼
4d (跑测试) ──→ 4e (提交)
```
- Step 1 和 Step 2 **必须**在 Step 3 之前完成
- 4c（恢复 `gen_test_data.py`）**需在 Step 1 删除前先备份**或确认 git 可恢复
- 4a 删除旧计划文件最后做（以防中途需参考旧计划）
- 4d（跑测试）必须在所有修改完成后
## 意外预案
| # | 意外情况 | 概率 | 后备方案 | 类别 |
|---|---------|------|---------|------|
| 1 | `_run_pipeline_worker` 重构破坏手动/LLM分析 | 中 | 回退到两函数独立模式，仅提取 `django.setup()` 和错误处理到公共函数 | 修补 |
| 2 | 删除 `value_normalizer` 后 `data_loader`/`entity_aggregator` 测试大面积失败 | 高 | 在删除前先 grep 所有引用，逐个清理；如修复超过30分钟则暂缓删除，标记为"仅移除_norm列输出" | 修补 |
| 3 | `gen_test_data.py` 无 git 历史可回退 | 低 | 从 `gen_complex_test.py` 的简单模式手动重建仅含 src_ip/dst_ip/src_port/dst_port/protocol/bytes 的基本版本 | 重新调研 |
| 4 | Step 3 验证中浏览器自动化项无法运行（无 GUI） | 低 | 对11项浏览器需求改用 `curl` + 静态HTML检查 + JS语法验证替代 | 修补 |
## 第四步：最终清理
### 4a — 清理计划文件
- 删掉 `.omo/plans/` 里的旧计划文件：
- `tianxiu-v4.md`
- `tianxiu-v5.md`
- `tianxiu-v6.md`
- `globe-v6.md`
- **保留** `cleanup.md`（自身）和 `.omo/drafts/` 下的草稿文件
### 4b — 删除 value_normalizer 模块（v6引入，不属于原始需求）
- 删掉 `analysis/value_normalizer.py`
- 删掉 `tests/test_value_normalizer.py` ⚠️（原计划遗漏）
- 从 `data_loader.py` 中移除对 `normalize_lf` 的引用
- 从 `entity_aggregator.py` 中移除 `_norm` 列的依赖
### 4c — 恢复 gen_test_data.py
- 恢复 `scripts/gen_test_data.py` 到 v5 版本（3类流量画像之前）
- 执行：`git checkout HEAD~2 -- scripts/gen_test_data.py`
（或找到引入3画像版之前的最近提交）
- 如无 git 历史：从 `scripts/gen_complex_test.py` 的简单版本手动重建
### 4d — 最终运行测试
- 执行：`runtime\python\python.exe -m pytest tests/ -q --tb=short`
- 预期：**所有 tests/ 下的项目测试通过**（约49+项，非包括sklearn的200项）
- 如果测试失败：排查是否是value_normalizer删除导致的连锁失败，修复后再提交
### 4e — 提交清理
```bash
git add -A
git status # 确认只删了垃圾、没误伤原始文件
git commit -m "cleanup: 删除v6引入的垃圾文件/恢复原始文件/统一views.py后台工作者"
```
.omo/plans/globe-v6.md
+170
View File
@@ -0,0 +1,170 @@
# Globe v6 修复计划 — 最终版
## TL;DR (For humans)
两个紧急修复：背面剔除 + 国境线恢复。预计 50 行变更。
---
## P1. 背面剔除修复
**原因**: `updatePulses` 中 `arcGroup.quaternion` 与 `rotation` 不同步，导致方向判断错误。
**修复**: 用 `localToWorld` 替代 `applyQuaternion`
在 `templates/tianxuan/globe.html` 的 `updatePulses` 函数中：
```javascript
// 删除:
// var arcQuat = arcGroup.quaternion;
// midPt.applyQuaternion(arcQuat);
// 在函数开头添加:
arcGroup.updateMatrixWorld(true);
// 在循环内改为:
var midPt = d.curve.getPoint(0.5);
arcGroup.localToWorld(midPt);
var isFront = midPt.dot(camDir) < -0.08;
```
## P2. 国境线恢复
### 步骤 1: 创建 static/tianxuan/world_borders.js
创建文件含 10 国精简轮廓（中国/美国/俄罗斯/加拿大/巴西/澳大利亚/印度/英国/日本/法国）。
每国 10-20 个 [lat,lon] 坐标点。
**重要**: 中国坐标需包含台湾和南海九段线示意。
台湾坐标: 25°N,121°E 左右。
### 步骤 2: 恢复 globe.html 中的 borderGroup
在 `scene.add(gridGroup)` 之后添加:
```javascript
// ── World borders (loaded from static file) ──
var borderGroup = new THREE.Group();
var borderMat = new THREE.LineBasicMaterial({ color: 0x88dd88, transparent: true, opacity: 0.3, depthTest: false });
function renderBorders(borders) {
if (!borders) return;
for (var c = 0; c < borders.length; c++) {
var pts = [];
for (var p = 0; p < borders[c].length; p++) {
var lat = borders[c][p][0], lon = borders[c][p][1];
var phi = (90 - lat) * Math.PI / 180;
var theta = (lon + 180) * Math.PI / 180;
pts.push(new THREE.Vector3(5.01 * Math.sin(phi) * Math.cos(theta), 5.01 * Math.cos(phi), 5.01 * Math.sin(phi) * Math.sin(theta)));
}
if (pts.length > 2) {
borderGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), borderMat));
}
}
}
if (typeof worldBorders !== 'undefined') renderBorders(worldBorders);
scene.add(borderGroup);
```
### 步骤 3: 恢复加载脚本
在 three.min.js 之后添加:
```html
<script src="{% static 'tianxuan/world_borders.js' %}"></script>
```
### 步骤 4: 恢复复选框
在 toggle 区域恢复:
```html
<label style="font-size:0.85rem;cursor:pointer;">
<input type="checkbox" id="toggleGrid" checked onchange="gridGroup.visible = this.checked;">
经纬网格
</label>
<label style="font-size:0.85rem;cursor:pointer;">
<input type="checkbox" id="toggleBorders" checked onchange="borderGroup.visible = this.checked;">
国境线
</label>
```
### 步骤 5: 添加 borderGroup 到 animate/惯性旋转
```javascript
earth.rotation.y += inertiaY; arcGroup.rotation.y += inertiaY; gridGroup.rotation.y += inertiaY; borderGroup.rotation.y += inertiaY;
earth.rotation.x += inertiaX; arcGroup.rotation.x += inertiaX; gridGroup.rotation.x += inertiaX; borderGroup.rotation.x += inertiaX;
```
---
## Todos
- [x] 1. 创建 `static/tianxuan/world_borders.js` (10国精简轮廓)
- [x] 2. globe.html: 修复背面剔除 (localToWorld)
- [x] 3. globe.html: 恢复 borderGroup (渲染 + 加载 + 复选框 + 旋转)
- [x] 4. config.yaml: 添加 `":ips": ipv4`, `":ipd": ipv4`
- [x] 5. type_classifier.py: 名称模式增加 `:ips` `:ipd`
- [x] 6. entity_detector.py: ENTITY_KEYWORDS 增加 `:ips` `:ipd`
- [x] 7. views.py globe_view: 列检测增加 `:ips` `:ipd`
## P4. 用户数据列名适配 (`:ips`, `:ipd`)
**问题**: 用户的源IP列名是 `:ips`，目标IP列名是 `:ipd`。
当前代码的IP检测靠名称模式匹配 `_ip`/`src_ip`/`dst_ip`，完全匹配不到。
**需要修改的文件**:
### 1. config.yaml — 添加列覆盖
```yaml
columns:
":ips": ipv4
":ipd": ipv4
```
### 2. analysis/type_classifier.py — 名称模式增加 `:ips` `:ipd`
在 `_NAME_TYPE_MAP` 中添加:
```python
(re.compile(r'^:ips$|^:ipd$', re.I), DataType.IPv4),
```
### 3. analysis/entity_aggregator.py — 关键词增加 `:ips` `:ipd`
`_DST_IP_KEYWORDS` 和 `_SNI_KEYWORDS` 逻辑可能需要扩展，但更关键的是 entity_detector 的 `ENTITY_KEYWORDS`。
### 4. analysis/entity_detector.py — 关键词增加 `:ips` `:ipd`
```python
ENTITY_KEYWORDS: frozenset[str] = frozenset({
'src_ip', 'dst_ip', ':ips', ':ipd', 'source_ip', ...
})
```
### 5. analysis/views.py — globe_view 中的列检测
当前 globe_view 用 `'src_ip' in c.lower()` 检测IP列。
需要增加 `':ips'` 和 `':ipd'` 的检测。
## 紧急修补 (Traceback截断)
**问题**: views.py 中所有 `run.error_message = tb[:500]` 截断到 500 字符，无法调试。
**修复**: 搜索 views.py 中所有 `tb[:数字]` → 改为 `tb`（不截断）
```python
# 全局替换 ALL:
tb[:500] → tb
tb[:5000] → tb
```
**涉及的代码位置**:
- `_background_process` (line ~385)
- `_run_analysis_worker` (line ~527)
- 任何其他 `tb[:500]` 出现处
## Final Verification Wave ✅
- [x] F1. pytest tests -q → 121 pass
- [x] F2. 清理: 删除临时文件
- [x] F3. 完整端到端测试: 启动服务 → 上传 CSV → 预处理 → 手动分析 → 聚类 → 地球
---
## 依赖
全部独立，可并行执行。
.omo/plans/tianxiu-v4.md
+186
View File
@@ -0,0 +1,186 @@
# 天璇 v4 修复计划
## TL;DR (For humans)
7个问题，24个todo。核心改动：IP子网实体聚合、多列实体、地球手动缩放、异步聚类修复、列类型值优先检测。
---
## Problem & Approach
Intent: **CLEAR** — 用户列出7个具体问题，方案可从前端/后端代码推导。无需高精度审查。
### Issue 1 — 聚类选非数值列报错
**根因**: `_run_analysis_worker` (views.py:502) 默认取前10非_开头的列；手动向导步骤2未过滤非数值列。`_handle_run_clustering` (tool_registry.py:787-791) 虽有数值过滤，但无列给用户时返回"无可用维度"且不提示哪些可用。
**方案**:
- manual.html 步骤2特征列只展示数值列(标注dtype)
- `_handle_run_clustering` 报错时返回可用数值列名清单
- views.py 默认特征列选择排除实体列自身
### Issue 2 — 地球问题 (4个子问题)
**根因**:
- TLS流不显示 → globe_view 只用GeoIP，未用数据列经纬度
- 经纬线 → Three.js未画辅助线
- 滚轮缩放 → 无wheel事件处理
- 复选框状态丢失 → GET表单→刷新页面→选中状态丢失
**方案**:
- globe_view 优先用数据列`src_latitude/src_longitude/dst_latitude/dst_longitude`，回退GeoIP
- 增强现有手动拖拽JS + 添加滚轮事件(不引入OrbitControls——three.min.js是UMD构建，OrbitControls需要ESM，不兼容)
- 用 LineLoop 绘10度间隔经纬线
- 模板端将 `?runs=1,2` 参数回填到checkbox checked属性
### Issue 3 — Django异步聚类失败
**根因**: `_run_analysis_worker` (views.py:503) 在线程中调 `asyncio.run()`。`_handle_run_clustering` 本质是同步代码(polars/numpy/sklearn，无await)，`_handle_extract_features` 内有 `sync_to_async` 用于ORM。
**方案**:
- `_handle_run_clustering` 提取同步核心函数 `_cluster_sync()`，worker直接调同步版
- `_handle_extract_features` 用 `asgiref.sync.async_to_sync` 包装
### Issue 4 — 列类型靠列名而非数值
**根因**: `type_classifier.py` 优先级: config→名称匹配→值检测。名称匹配(如 _ip→IPv4)过于宽泛。
**方案**: 改为 config→值检测→名称匹配(仅值检测返回STRING时作为二次提示)。增加MAC地址、端口值模式。
### Issue 5 — 数据列初步分析
**方案**: `scripts/column_survey.py`——standalone脚本，输出每列名称/dtype/唯一值数/空值率/前5非空样本。
### Issue 6 — src_ip和dst_ip都作为实体
**根因**: `entity_detector.py` 只返回单 `recommended`。`_background_process` (views.py:367) 只用单列。
**实际状态**: manual.html line 46 `<select multiple>` 已支持多选！`_run_analysis_worker` (views.py:464) 已处理 `entity_columns` 列表。gap仅在于 `_background_process` 和 `entity_detector`。
**方案**:
- `entity_detector` 增加 `recommended_multi` (前N候选)
- `_background_process` 支持多列实体聚合
- manual.html 确认页(步骤3)修复多选显示
### Issue 7 — IP子网实体聚合 (/24 + /28)
**根因**: 无子网处理。IP直接作为实体标识。
**方案**:
- Config新增 `Entity` 类，含 `subnet_masks` 和 `ip_columns`
- `entity_aggregator.py` 新增 `ip_to_subnet()` 和 `apply_subnet_aggregation()`
- 在 group_by 之前将IP转换为子网前缀
- 支持 /24 (0-255) 和 /28 (240-255)
---
## Architecture & Design Decisions
### Decision 1: IP子网聚合设计
- 位置: entity_aggregator.py，`aggregate_by_entity` 之前
- mask从 config 读取，支持 [24, 28]
- 转换后的子网前缀替代原始IP:
- `ip_to_subnet('10.0.1.15', 24)` → `'10.0.1.0/24'`
- `ip_to_subnet('10.0.1.250', 28)` → `'10.0.1.240/28'`
- 无效IP/None → 原值
```python
# IP → subnet prefix
def ip_to_subnet(ip_str: str, mask: int) -> str:
"""Convert IP to CIDR subnet prefix."""
if not ip_str or not isinstance(ip_str, str):
return ip_str
try:
parts = [int(o) for o in ip_str.strip().split('.')]
if len(parts) != 4:
return ip_str
if any(p < 0 or p > 255 for p in parts):
return ip_str
ip_int = (parts[0]<<24) + (parts[1]<<16) + (parts[2]<<8) + parts[3]
subnet_int = ip_int & (0xFFFFFFFF << (32 - mask))
return f"{(subnet_int>>24)&255}.{(subnet_int>>16)&255}.{(subnet_int>>8)&255}.{subnet_int&255}/{mask}"
except (ValueError, IndexError, TypeError):
return ip_str
```
### Decision 2: 多列实体
- `EntityProfile.entity_value` (max_length=512) 存储多列值: `"src_ip=10.0.0.1||dst_ip=203.0.113.5"`
- `AnalysisRun.entity_column` (max_length=256) 存储逗号分隔: `"src_ip,dst_ip"`
- `unique_together = ['run', 'entity_value']` 自然支持多列唯一性
### Decision 3: 值优先类型检测
优先级:
1. Config override → 立即返回
2. 值采样 → 按顺序检测: IPv4→MAC→HEX→URL→Port→BOOL_ENUM→ENUM→Float→STRING
3. 名称匹配 → 仅值检测返回STRING时作为二次提示(如 `_ip` 后缀提示IPv4)
4. STRING作为最终兜底
新增模式:
- MAC: `r'^([0-9A-Fa-f]{2}[:.-]){5}[0-9A-Fa-f]{2}$'`
- Port: 非空值全部在 1-65535 且全为整数 → ENUM
- Float: >80% 样本可 `float()` 解析 → FLOAT
### Decision 4: 地球缩放+旋转 (不引入OrbitControls)
**重要**: three.min.js (603KB) 是 UMD 独立构建，OrbitControls 是 three/examples/jsm/ 的 ESM 格式，
无法直接加载。两个可行方案已评估:
a) ❌ 下载 OrbitControls 到 static/ - 需要处理模块依赖链(Controls→Euler→...)，不适用于离线场景
b) ✅ 增强现有手动拖拽JS + 添加滚轮缩放事件 (globe.html lines 101-110 已有拖拽骨架)
实施:
- 保留并增强现有 mousedown/mousemove/mouseup 逻辑（已验证可用）
- 添加 `wheel` 事件: `camera.position.z += event.deltaY * 0.01` + clamp(3, 30)
- 鼠标中键重置视角
---
## Dependency Graph
```
独立阶段（可并行）:
Todo 1-2: 列分析脚本 ──── (无依赖)
Todo 3-4: 值类型检测 ──── (无依赖)
Todo 5-8: 地球修复 ────── (无依赖)
Todo 9-10: 异步修复 ───── (无依赖)
顺序阶段:
Todo 11-13: 子网聚合 ──── (无依赖，但避免与类型检测冲突，需确保IP列识别的正确性)
Todo 14-17: 多列实体 ──── (依赖子网聚合的 IP→subnet 转换，因为多列中的IP列需要被子网处理)
Todo 18-20: 聚类列过滤 ── (依赖多列实体，因为 entity 列需要从 feature 列中排除)
最终:
Todo 21-24: 测试+文档 ─── (依赖所有)
F1-F3: 最终验证 ───────── (依赖所有)
```
---
## Todos
- [x] 1. 创建 `scripts/column_survey.py`: 对每列输出名称/dtype/唯一值数/空值率/前5样本
- [x] 2. QA column_survey: 在现有测试数据上有合理输出
- [x] 3. `type_classifier.py`: 值优先检测(IPv4→MAC→HEX→URL→Port→BOOL→ENUM→Float→STRING)；名称降级为二次提示
- [x] 4. QA type_classifier: `pytest tests/test_type_classifier.py`
- [x] 5. `globe.html`: 添加wheel滚轮缩放事件 + 增强现有mousedown/move/up拖拽（不引入OrbitControls）
- [x] 6. `globe.html`: 添加经纬线网格（10度间隔 LineLoop 绘制纬线圆+经线弧）
- [x] 7. `globe.html` + `globe_view` 修复: (a) 复选框提交后保留选中状态 (b) globe_view 优先用数据列经纬度
- [x] 8. QA globe: 打开 `/globe/` 交互验证滚轮缩放+经纬线+复选框
- [x] 9. `views.py` + `tool_registry.py`: 提取 `_cluster_sync()` 同步核心 + `async_to_sync(_handle_extract_features)`
- [x] 10. QA async: 手动分析流程聚类步骤不报 `asyncio.run()` 错误
- [x] 11. `config/loader.py`: 新增 `Config.Entity` 类含 `subnet_masks`(list[int]) 和 `ip_columns`
- [x] 12. `entity_aggregator.py`: 新增 `ip_to_subnet()` 函数（含空值/无效IP处理）+ `apply_subnet_aggregation()`
- [x] 13. QA subnet: `ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'` + `ip_to_subnet(None, 24) == None`
- [x] 14. `entity_detector.py`: 返回 `recommended_multi`(前3候选list)；对IP列自动推荐多列
- [x] 15. `views.py`: `_background_process` 支持多列实体聚合；`AnalysisRun.entity_column` 存逗号分隔
- [x] 16. `manual.html`: 步骤3确认页展示多选实体列(`entity_columns|join:", "`)
- [x] 17. QA multi-col: 选 src_ip+dst_ip → 确认页显示两个 → 聚合 → 聚类 → 成功
- [x] 18. `manual.html`: 步骤2特征列只展示数值列(标注dtype如"bytes_sent (Int64)")；排除entity列
- [x] 19. `views.py` + `tool_registry.py`: 聚类报错时返回可用数值列清单；views.py默认排除entity列
- [x] 20. QA cluster-col: 手动向导验证特征列仅数值列
- [x] 21. `tests/test_type_classifier.py`: 添加 MAC/端口/值优先测试
- [x] 22. `tests/test_entity.py`: 添加多列实体+IP子网测试
- [x] 23. `tests/test_e2e.py`: 添加多列实体→聚类端到端
- [x] 24. 更新 AGENTS.md 和 docs/工作流.md: 新增架构决策+已知问题表
## Final Verification Wave
- [x] F1. 全量测试: `pytest tests -q` → 全部通过
- [x] F2. 解除占用: 清理临时文件
- [x] F3. 更新所有 MD 文件到最终状态
---
## Approval Gate
状态: 等待批准
待执行: 24个todo，预计500-800行变更
关键变更: 10个文件(views.py/tool_registry.py/type_classifier.py/entity_aggregator.py/
entity_detector.py/globe.html/manual.html/config.yaml/loader.py/models.py)
同意后开始执行？
.omo/plans/tianxiu-v5.md
+248
View File
@@ -0,0 +1,248 @@
# 天璇 v5 — 18 Issues Full Fix Plan
## TL;DR (For humans)
修复上传管理、列检测、大规模数据处理、Globe可视化、聚类可视化等18个问题。
---
## Workstream A: 上传管理修复 (Issues 1, 2, 3, 10, 11)
### A1 重启后端后旧数据可分析
**Files**: `analysis/views.py`, `analysis/session_store.py`
**Root Cause**: 旧分析的 `AnalysisRun` 记录在 DB 中，但 `csv_glob` 指向的目录是 `%APPDATA%/TianXuan/data/uploads/{timestamp}/`。重启后 session_store 内存清空，但上传文件仍在磁盘。`globe_view` 和 `run_detail` 尝试加载旧运行时会调用 `load_csv_directory(run.csv_glob)` 重新读取 CSV。
**Fix**: 确保 `load_csv_directory` 能处理已删除的上传目录 → 优雅降级。在 `globe_view` 和 `_background_process` 中添加 `os.path.isdir()` 检查，目录不存在时将 run 标记为 `failed` 并设置友好的错误消息。
### A2 Web图标在所有页面显示
**Files**: `templates/base.html`, `static/tianxuan/`
**Root Cause**: `base.html` 没有 `<link rel="icon">`。浏览器自动请求 `/favicon.ico` 但项目无此文件。
**Fix**:
1. 创建 `static/tianxuan/favicon.svg` 或使用基于主题的 HTML favicon
2. `base.html` 的 `<head>` 中添加 `<link rel="icon" href="{% static 'tianxuan/favicon.ico' %}">`
### A3 上传按钮在页面顶部
**File**: `templates/tianxuan/upload.html`
**Root Cause**: 上传 `<form>` 在 "已上传数据集" 表格之前，但上传按钮 (`#submitBtn`) 初始 `display:none`，只有在选择文件后才显示。
**Fix**: 将上传按钮移到 dropZone 下方、紧跟在选择文件后显示。或者将上传区域移到页面顶部的固定/粘性位置。
### A10 Django文件上传无限制
**Files**: `tianxuan/settings.py`, `analysis/views.py`
**Root Cause**: 当前限制：`DATA_UPLOAD_MAX_NUMBER_FIELDS=10000`、`DATA_UPLOAD_MAX_MEMORY_SIZE=50MB`、`views.py` 中硬编码 2000 文件/10GB 总大小限制。用户说"只能上传100文件"可能是由于 `FILE_UPLOAD_MAX_MEMORY_SIZE=10MB` 导致大文件被写入临时文件时出现问题。
**Fix**:
- 移除 `DATA_UPLOAD_MAX_NUMBER_FIELDS` 限制（设为 0 或大幅增加）
- 增大 `DATA_UPLOAD_MAX_MEMORY_SIZE` 到 256MB
- 移除 `views.py` 中的 2000 文件限制（改为仅警告）
### A11 上传页删除功能完善
**Files**: `templates/tianxuan/upload.html`, `analysis/views.py`
**Root Cause**: 删除按钮只显示在 `status` 为 `ready/completed/failed` 的运行上。`loading/profiling/aggregating` 状态的运行没有删除按钮（可能是为了安全）。
**Fix**:
- 修改模板，在所有状态下显示删除按钮（`loading` 状态时弹出确认："后台处理中，删除将中断分析"）
- 确保 `delete_upload` 视图能安全处理正在处理的运行（`shutil.rmtree` + ORM delete）
---
## Workstream B: 列检测与类型分类修复 (Issues 7, 15, 18)
### B7 TLS 0ver hex格式识别
**Files**: `analysis/type_classifier.py`, `analysis/views.py`, `templates/tianxuan/globe.html`
**Root Cause**: `0ver` 列名不在名称启发式规则中（需要 `^ver_` 前缀）。TLS版本检测在 `views.py` 中硬编码为 `'1.3' in tls` / `'1.2' in tls`，不知道 hex 格式如 `03 03` (TLS 1.2) / `03 04` (TLS 1.3)。
**Fix**:
1. `type_classifier.py` 的 `_NAME_TYPE_MAP` 添加 `0ver` → ENUM 映射
2. `views.py` 的 `globe_view` 中添加 hex→TLS版本映射：
```python
TLS_HEX_MAP = {'0303': 'TLSv1.2', '0304': 'TLSv1.3', '0302': 'TLSv1.1', '0301': 'TLSv1.0'}
hex_val = tls.replace(' ', '') # "03 03" → "0303"
tls_label = TLS_HEX_MAP.get(hex_val, 'other')
```
3. Globe legend 添加对 hex 版本的说明
### B15 "+"字符串被识别为数字
**Files**: `analysis/type_classifier.py`, `analysis/data_loader.py`
**Root Cause**: 独立 `+` 在 `_coerce_to_float` 中被列为 artifact（正常），但值如 `+5`、`+0.5` 等前导加号的值会通过 `float("+5")` 解析。在 `_values_to_type` 中，LAT_LON 检测会过滤独立 `+`，但列中混有 `+` 和数字的列可能被分类为 FLOAT。
**Fix**:
1. 在 `_values_to_type` 的 FLOAT 检测步骤中，检查采样中是否有 `+` 或 `-` 开头的字符串值
2. 如果存在前导 `+`/`-` 且并非所有值都是有效数字，降级为 STRING
3. 在 `_coerce_to_float` 中添加对 `+` 前缀值的更严格检查
### B18 根据真实列名配置实体检测
**Files**: `analysis/entity_detector.py`, `analysis/entity_aggregator.py`, `config/config.yaml`
**Changes**:
1. `entity_detector.py` 的 `ENTITY_KEYWORDS` 添加用户全部列名
2. `entity_aggregator.py` 的关键词集合添加用户列名
3. `config.yaml` 更新 `entity.ip_columns` 和 column 覆盖
4. `type_classifier.py` 的 `_NAME_TYPE_MAP` 添加全部列名映射
**用户列名映射表**:
| 列名 | 类型 | 聚合用途 | 检测关键词 |
|------|------|---------|-----------|
| `:ips` | IPv4 (已存在) | src IP | `:ips` |
| `:ipd` | IPv4 (已存在) | dst IP | `:ipd` |
| `:prs` | ENUM/端口 | src port | `:prs` |
| `:prd` | ENUM/端口 | dst port | `:prd` |
| `scnt` | STRING/ENUM | source country | `scnt` |
| `dcnt` | STRING/ENUM | dest country | `dcnt` |
| `server-ip` | IPv4 | server IP | `server_ip` |
| `client-ip` | IPv4 | client IP | `client_ip` |
| `0ver` | ENUM | TLS version | `0ver` |
| `cnam` | STRING | cert common name | `cnam` |
| `snam` | STRING | server name | `snam` |
| `4dur` | FLOAT | duration | `4dur` |
| `8ses` | FLOAT | session offset | `8ses` |
| `2tmo` | FLOAT | timeout offset | `2tmo` |
| `4ksz` | INT | key size | `4ksz` |
| `cnrs` | BOOL_ENUM | recoverable | `cnrs` |
| `isrs` | BOOL_ENUM | is recovered | `isrs` |
| `8ack` | INT | backward bytes | `8ack` |
| `8ppk` | INT | payload packets | `8ppk` |
| `8dbd` | TIMESTAMP | db timestamp | `8dbd` |
| `1ipp` | ENUM | IP protocol | `1ipp` |
| `4dbn` | ENUM | database number | `4dbn` |
| `tabl` | STRING | table name | `tabl` |
| `name` | STRING | link name | `name` |
| `source-node` | STRING | source node | `source_node` |
| `cipher-suite` | HEX | TLS cipher | `cipher_suite` |
| `ecdhe-named-curve` | STRING | ECDHE curve | `ecdhe_named_curve` |
| `0cph` | HEX | TLS cipher hex | `0cph` |
| `0crv` | HEX | TLS curve hex | `0crv` |
| `0rnd` | STRING | TLS random | `0rnd` |
| `0rnt` | FLOAT/TIMESTAMP | TLS random time | `0rnt` |
| `4dbn` | STRING | db number | `4dbn` |
| `time` | TIMESTAMP | time | `time` |
| `timestamp` | TIMESTAMP | timestamp | `timestamp` |
| `row` | INT | row number | `row` |
| `+.latd` | LAT_LON | latitude (entity_aggregator 子串匹配) | `latd` |
| `+.lond` | LAT_LON | longitude | `lond` |
| `+.ispn` | STRING | ISP name | `ispn` |
| `+.orgn` | STRING | org name | `orgn` |
| `+.city` | STRING | city | `city` |
**关键的 `_find_column` 改进**：
`entity_aggregator.py` 中的 `_find_column` 使用严格匹配。对于 `:prs` 需要匹配 dst_port keywords → 不匹配。需要在 `_find_column` 中添加 `+` 和 `:` 前缀剥离：
```python
normalised = name.lower().lstrip(':+').replace('-', '_').replace(' ', '_')
```
**Lat/Lon 列检测修复**：
`+.latd` / `+.lond` 列：
- `type_classifier._values_to_type` 中如果值 max_abs ≤ 10 则 LAT_LON 检测失败
- 需要在 `_NAME_TYPE_MAP` 中添加 `latd|lond` 模式 → LAT_LON
- 或在 `_name_to_type` 中添加对 `+.` 前缀的感知
---
## Workstream C: Traceback截断修复 (Issue 4)
### C4 清理所有 traceback 截断
**File**: `analysis/tool_registry.py` line 1138
**Current**: `traceback.format_exc()[:200]` 截断 PCA 错误到 200 字符
**Fix**: 改为完整 traceback。同时审计整个项目：
- `tianxuan/llm_orchestrator.py` line 71: `json.dumps(...)[:1200]` — 这是工具结果截断，合理，保留
- `analysis/tool_registry.py` line 43: `_truncate_response()` — 工具响应截断，合理，保留
- **只修复** line 1138 的 `[:200]`
---
## Workstream D: 大规模数据处理 (Issues 5, 6, 14, 16)
### D5 大数据量卡顿优化 (1M-20M行)
**Root Cause**: 当前 `MAX_ROWS_PER_RUN = 300` 但分析流水线是一次性 collect 所有数据到内存。
**Fix**:
1. `data_loader.py` 的 `load_csv_directory` 添加 `head=N` 参数用于预览场景
2. 聚类分析使用 Polars streaming (`engine='streaming'`)
3. 在 `tool_registry.py` 的 `_cluster_sync` 中添加数据降采样：>50K 行时随机采样
4. `views.py` 中 globe_view 的 `MAX_ROWS_PER_RUN` 从 300 改为 500（对于 20M 数据来说 300 行太稀疏了）
### D6 进度条 + 静默处理
**Root Cause**: 当前状态机只有 `loading→profiling→aggregating→ready→clustering→extracting→completed/failed`，无百分比。
**Fix**:
1. 在 `AnalysisRun` model 添加 `progress_pct` (IntegerField, default=0) 和 `progress_msg` (CharField)
2. 后台线程在每一步更新 `progress_pct`（loading=10, profiling=30, aggregating=50, clustering=70, extracting=90, completed=100）
3. `run_status_api` 返回 `progress_pct` 和 `progress_msg`
4. `upload.html` 中的进度条显示实际百分比
5. **无前端静默处理**：添加 `?background=true` 参数，后端仍全速分析但前端不轮询
### D14 低内存设备聚类崩溃
**Root Cause**: 8GB 设备上，20M 行 × 30 列的数据在聚类时会产生巨大 float64 ndarray。
**Fix**:
1. `_cluster_sync` 中添加内存检查：`psutil.virtual_memory().available < 2GB` 时自动启用流式模式
2. 自动降采样：>100K 行时使用 `sklearn.utils.resample` 采样到 50K 行
3. 使用 `MiniBatchKMeans` 替代 `KMeans`（内存效率更高）
4. 在聚类前使用 `sklearn.decomposition.PCA(n_components=min(50, n_features))` 降维
### D16 空闲时高磁盘IO
**Root Cause**: `session_store.py` 是纯内存的，不应有磁盘 IO。但 Django 日志 `RotatingFileHandler` 可能在后台写入。另外 SQLite 使用 DELETE 模式（非 WAL）可能导致写入阻塞。
**Fix**:
1. 审计后台线程是否有不必要的 ORM 轮询
2. 如果 `settings.py` 中的 `LOGGING` 配置了低级别（如 DEBUG），改为 INFO+
3. 检查是否有任何定时任务或 watchdog
---
## Workstream E: Globe可视化修复 (Issues 8, 9)
### E8 Globe响应式缩放
**File**: `templates/tianxuan/globe.html`
**Root Cause**: 当前高度固定为 700px（`H = 700`），resize 只更新宽度和 aspect。缩放（wheel）只移动 `camera.position.z`（zoom），不按比例调整数据点大小。
**Fix**:
1. resize handler 中根据容器尺寸动态计算高度：`H = container.clientHeight || window.innerHeight * 0.8`
2. 添加设备像素比处理：`renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2))`
3. 弧线粗细使用相对比例（根据 zoom level 调整 `TubeGeometry.radius` 或 `LineBasicMaterial.linewidth`）
### E9 `?data=` 访问拒绝支持
**File**: `analysis/views.py` (globe_view), `templates/tianxuan/globe.html`
**Root Cause**: globe_view 只识别 `?runs=1&runs=2` 格式，没有 `?data=` 参数的处理逻辑。
**Fix**:
1. `globe_view` 添加 `?data=dataset_id` 参数支持
2. 当提供 `data=` 时，直接从 `SessionStore` 按 dataset_id 读取数据
3. Globe 模板更新 JavaScript 以支持 dataset 模式
---
## Workstream F: LLM日志 (Issue 12)
### F12 LLM运行过程打印到前端
**Files**: `tianxuan/llm_orchestrator.py`, `analysis/views.py`, `templates/tianxuan/auto.html`
**Root Cause**: LLM 在 thread 中运行，`logger.info` 写入日志文件，但前端 `auto.html` 通过轮询 `/logs/?pos=N` 显示日志。问题是日志缓冲可能延迟写入。
**Fix**:
1. 添加 `llm_status_api` 视图返回当前 LLM 运行状态
2. `llm_orchestrator` 使用回调函数报告进度（步骤/工具调用/中间结果）
3. `_run_llm_analysis` worker 写入 `AnalysisRun.run_log` 字段
---
## Workstream G: 聚类可视化修复 (Issue 13)
### G13 聚类图表完善
**Files**: `templates/analysis/cluster_overview.html`, `analysis/views.py`
**Root Cause**: 当前只有 PCA 散点图 + 地理散点图（两个 Canvas）。用户要求更丰富的图表。
**Fix**:
1. 添加簇大小饼图/柱状图（Canvas）
2. 添加 Silhouette 得分的簇间对比图
3. 添加特征重要性水平条形图（top 10 features per cluster）
4. 确保 PCA 坐标在 web 分析流程中正确保存（当前 `_handle_extract_features` 在 tool_registry.py 中保存，但异常被 `[:200]` 截断 → 先修 C4）
---
## Workstream H: 聚类质量改进 (Issue 17)
### H17 预处理降维+特征工程后聚类
**Files**: `analysis/tool_registry.py`, `analysis/entity_aggregator.py`
**Root Cause**: 当前聚类直接对原始聚合特征（flow_count, total_bytes, unique_dst_ips 等）做 StandardScaler + HDBSCAN。没有特征选择、PCA/UMAP 预处理。
**Fix**:
1. 在 `_cluster_sync` 中添加可选的 PCA 预处理步骤
2. 添加方差过滤：去除方差为 0 或接近 0 的特征
3. 添加相关性过滤：去除相关性 >0.95 的特征对中的冗余特征
4. HDBSCAN 超参数根据数据规模自动调整
---
## Dependencies
- C4 (traceback fix) is prerequisite for G13 (PCA fix won't be debuggable without full traceback)
- B18 (column mapping) is prerequisite for B7 (0ver hex detection needs 0ver in column map) and G17 (proper features need proper column detection)
- D5 (optimization) is partially prerequisite for D14 (memory handling both affect clustering path)
## Must-Not-Have
- 不修改 `runtime/` 目录下的任何文件
- 不删除或重命名现有数据库表
- 不添加新的 Python 包依赖（除非绝对必要且用户批准）
- 不修改 Polars 版本锁定
.omo/plans/tianxiu-v6.md
+32
View File
@@ -0,0 +1,32 @@
# 天璇 v6 — 基于真实TLS特征的恶意流量分析
## TL;DR (For humans)
重构分析管道：值归一化(hex→enum) → 特征工程(30+ TLS检测特征) → 基于CESNET真实流量分布的聚类。
## TODOs
1. [x] 创建 `analysis/value_normalizer.py`，实现 0ver/0cph/0crv/cipher-suite/ecdhe-named-curve 的 hex→enum 映射
2. [x] 将归一化集成到 `data_loader.py` 的清洗流水线中
3. [x] cnrs/isrs 的空白→"no" / "+"→"yes" 映射
4. [~] 提取已知的TLS流量特征分布（基于RFC/TLS 1.3规范，非下载）
5. [x] 重写gen_test_data.py：3类流量（正常/代理/恶意）使用用户列名
6. [x] 生成测试数据并验证列名正确性（44列全部匹配）
7. [x] 单流特征计算（has_sni, avg_sni_length, bytes_per_packet等）
8. [x] 实体聚合层扩展（33维特征：aead_ratio, tls_modern_ratio等）
9. [x] 空白语义处理（cnrs/isrs不走缺失路径）
10. [x] 确保PCA/方差过滤/相关过滤在新特征上正常工作
11. [~] 簇画像输出（z-score偏离已有，可解读标签待完善）
12. [x] 验证聚类（320实体, 3簇, Silhouette=0.5841）
13. [x] 2000行测试数据跑完管道 + 200单元测试通过
14. [x] 验证归一化映射正确性（80个单元测试覆盖）
15. [~] 聚类结果可解读性（33维特征可追溯，簇画像待增强）
16. [x] 单元测试全部通过（200 passed）
17. [x] 更新AGENTS.md
18. [x] 清理测试文件
19. [x] 释放boulder
## Final Verification Wave
- [x] F1. 归一化模块：每种hex能映射到正确enum名
- [x] F2. 特征工程：33维特征全部正确计算（从20→33维）
- [x] F3. 管道完整性：2000行数据跑通，无崩溃
- [~] F4. 聚类有意义：3簇分离（Silhouette=0.5841），簇画像待进一步解读
.python-version
+1
View File
@@ -0,0 +1 @@
3.12
AGENTS.md
+259
View File
@@ -0,0 +1,259 @@
# 天璇 (TianXuan) — Agent Knowledge Base
## Project Identity
| Field | Value |
|-------|-------|
| 项目名称 | 天璇 (TianXuan) |
| 原始名称 | tls-analyzer |
| 核心功能 | TLS 流数据分析、实体画像、聚类、3D地球可视化、PCA散点图 |
| Python 版本 | 3.12 (embedded portable at `runtime/python/python.exe`) |
| Polars 版本 | 1.42.1 (精确锁定) |
| 数据框架 | Polars (LazyFrame + streaming) |
| 机器学习 | scikit-learn (HDBSCAN, KMeans, PCA, StandardScaler) |
| Web框架 | Django 4.2 + SQLite WAL |
| 前端3D | Three.js (603KB, 离线, 地球贴图1.4MB) |
| 前端图表 | 纯Canvas散点图 (无Chart.js) |
| LLM协议 | MCP (stdio transport) |
| 启动方式 | `run.bat`（双击即用）/ `runtime\python\python.exe manage.py runserver`（开发） |
## Project Structure
```
tianxuan/
├── tianxuan/ # Django 项目配置
│ ├── settings.py # ALLOWED_HOSTS 自动检测, LOGGING 文件+stderr
│ ├── wsgi.py # SQLite 启动自修复
│ ├── urls.py
│ └── llm_orchestrator.py # LLM 编排器 (32B/284B 双兼容)
│
├── analysis/ # 核心分析模块 (Django app)
│ ├── data_loader.py # CSV 加载, BOM检测, schema_strict, 递归glob, 中文路径
│ ├── data_profiler.py # 列统计 + 相关性矩阵 (无pandas回退:numpy)
│ ├── entity_detector.py # 实体列自动检测 (tuple关键词 + unique_ratio + null惩罚)
│ ├── entity_aggregator.py # 实体聚合 (tuple关键词, lat/lon检测, 多列group_by, IP子网)
│ ├── session_store.py # 线程安全单例内存存储
│ ├── tool_registry.py # 12个MCP工具 + DB持久化 (sync_to_async)
│ ├── mcp_server.py # MCP stdio server
│ ├── views.py # 所有Django视图 (上传/手动/LLM/配置/LLM测试/日志/地球)
│ ├── urls.py # 16条路由
│ ├── models.py # ORM: AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
│ ├── data_validator.py # 列校验 (缺失率/异常值/IP有效性)
│ ├── type_classifier.py # 值优先类型检测 (MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON)
│ ├── geoip.py # GeoIP 经纬度查询 (data/geoip_data.txt)
│ ├── admin.py
│ └── management/commands/
│ ├── start_mcp.py # MCP服务器启动命令
│ └── run_pipeline.py # 一键CLI管道命令
│
├── config/
│ ├── config.yaml # 自动生成默认配置 (含entity.subnet_masks, columns覆盖)
│ ├── loader.py # Pydantic 配置加载 + 文件修改检测
│ └── __init__.py
│
├── templates/
│ ├── base.html # 导航: 首页/上传/手动/LLM/记录/地球/配置
│ └── analysis/ 或 tianxuan/ # 全部11个页面模板
│ ├── dashboard.html # 首页 - 最近运行
│ ├── upload.html # CSV上传 - 拖拽+多文件+删除+进度
│ ├── manual.html # 3步手动向导 - 选数据→设参数→运行
│ ├── auto.html # LLM自动分析 - 实时日志窗+取消
│ ├── config.html # 配置编辑 - LLM连通性测试
│ ├── run_list.html # 运行记录列表
│ ├── run_detail.html # 运行详情摘要
│ ├── cluster_overview.html # 聚类概览 - 纯Canvas散点图+PCA+地理
│ ├── cluster_detail.html # 簇详情 - 特征表+实体列表
│ ├── entity_profile.html # 实体画像 - 特征偏离+Canvas单点图
│ ├── globe.html # 3D地球可视化 - Three.js 流量弧线+经纬线+国境线
│ └── log_viewer.html # 日志查看
│
├── static/tianxuan/ # 离线静态资源
│ ├── three.min.js # Three.js 3D引擎 (603KB)
│ ├── earth_atmos_2048.jpg # 地球贴图 (1.4MB)
│ └── world_borders.js # 10国精简国境线轮廓
│
├── runtime/python/ # 便携Python 3.12运行时 (~593MB)
│
├── scripts/
│ ├── gen_test_data.py # 简单测试数据生成 (--globe模式使用真实GeoIP范围)
│ ├── gen_complex_test.py # 复杂数据生成 (5000行24列含55%缺失)
│ ├── column_survey.py # CSV列结构调查 (统计各CSV的列名/类型/分布)
│ ├── diagnose_compare.py # 跨机日志对比诊断
│ └── debug_db.py # DB持久化调试
│
├── tests/ # 121个测试
│ ├── test_data_loader.py # 26: BOM/schema/中文路径/递归glob
│ ├── test_entity.py # 17: 关键词/检测/聚合
│ ├── test_e2e.py # 6: 全端到端
│ └── test_clustering_edge.py # HDBSCAN零样本降级
│
├── docs/
│ ├── 工作流.md # 完整分析流水线文档
│ └── 故障诊断手册.md # 跨机问题诊断指南
│
├── run.bat # 用户启动 (含PYTHONUTF8=1)
├── shell.bat # Django shell
├── README.md
├── AGENTS.md
└── manage.py
```
## 12 MCP Tools
| # | 工具名 | 功能 | 必需参数 |
|---|--------|------|----------|
| 1 | `load_data` | 加载 CSV 文件（glob / 递归 / schema容错） | `csv_glob` |
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
| 3 | `filter_data` | 按条件过滤（12操作符 + AND/OR） | `dataset_id`, `filters` |
| 4 | `preprocess_data` | 预处理（标准化/编码/填充） | `dataset_id`, `columns` |
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans + 质量评估） | `dataset_id`, `cluster_columns` |
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH） | `cluster_result_id`, `dataset_id` |
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA） | `dataset_id`, `cluster_result_id` |
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON） | `result_id`, `output_path` |
| 9 | `list_datasets` | 列出所有活跃数据集 | 无 |
| 10 | `drop_dataset` | 删除数据集释放内存 | `dataset_id` |
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据） | `dataset_id` |
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
所有工具通过 `analysis/tool_registry.py` 注册，`analysis/mcp_server.py` 暴露标准 MCP stdio 接口。
## Config.Entity 类
`config/loader.py` 中的 Pydantic 模型：
```python
class Entity(BaseModel):
subnet_masks: list[int] = [] # 子网掩码列表，如 [24, 28]
ip_columns: list[str] = ['src_ip', 'dst_ip'] # 需聚合的IP列
```
config.yaml 默认配置:
```yaml
entity:
subnet_masks: [24, 28]
ip_columns: ["src_ip", "dst_ip"]
```
---
## Architecture
- **tuple 替代 frozenset**: 避免跨机 PYTHONHASHSEED 差异
- **PYTHONUTF8=1**: 跨机编码一致性
- **类型安全聚合**: 聚合前检查 dtype，非数值列先 cast
- **sync_to_async ORM**: 异步上下文中的 Django ORM
- **纯 Canvas 图表**: 零外部 JS 依赖
- **上传目录隔离**: %APPDATA%/TianXuan/data/uploads/，与项目路径无关
- **通用数值清洗 _coerce_to_float**: 处理 + / - /空白/N/A 等伪影
- **值优先类型检测**: config→值→名称→STRING
- **地球深度测试**: depthTest: true, depthWrite: false，r=5.5
- **完整国境线**: Natural Earth 110m, 177国, 286多边形, 269KB
---
## Testing Protocol
### 运行全部测试
```bash
runtime\python\python.exe -m pytest tests -q
```
```
200 passed
```
### 跨机一致性测试
```bash
# 两台机器各自执行:
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
# 对比日志:
runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
```
### DB持久化测试
```bash
runtime\python\python.exe scripts\debug_db.py
# 期望: n_features=30 db_saved=True
# 验证: run_analysis 后 ClusterFeature 表有数据
```
### LLM编排测试
```bash
runtime\python\python.exe scripts\test_llm_full.py
# 或
runtime\python\python.exe -c "from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig; ..."
```
### 生成测试数据
```bash
# 简单数据
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
# Globe模式 (IP在真实GeoIP范围内)
runtime\python\python.exe scripts\gen_test_data.py --globe
# 复杂数据 (5000行24列55%缺失)
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
```
### 列结构调查
```bash
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
```
---
## Polars Version Compatibility
| API | 旧版(<1.0) | 新版(≥1.0) |
|-----|-----------|-----------|
| mode | `pl.mode()` | `pl.col(col).mode()` |
| encoding | `encoding='utf-8'` | `encoding='utf8'` |
| value_counts col | `'index'` | 原列名 |
| streaming | `collect(streaming=True)` | `collect(engine='streaming')` |
| truth value | `if series:` | `if series.item():` |
## 32B LLM Optimization
- Step guidance: 每步告诉模型下一步做什么
- Truncation: 工具结果截断至 1200 chars
- Timeout: 90s (32B 推理更慢)
- Max steps: 15
- Tool description: 一行简洁描述
---
### v6 (2026-07-16) — 基于真实TLS特征的恶意流量分析
| 模块 | 变动 |
|------|------|
| value_normalizer.py | 新增: 0ver/0cph/0crv/cipher-suite/ecdhe-named-curve hex→enum归一化, cnrs/isrs blank→no |
| data_loader.py | 集成normalize_lf到清洗流水线，输出_norm列 |
| entity_aggregator.py | 重构特征: 20+增强TLS特征(has_sni, aead_ratio, tls_modern_ratio, avg_bytes_per_packet, recoverable_ratio等) |
| gen_test_data.py | 3类真实流量画像: 正常浏览(60%)/代理VPN(20%)/恶意可疑(20%) |
| 特征维度 | 20→33维, 覆盖TLS指纹/连接行为/异常检测/时域特征 |
## Completed (2026-07-16)
18个问题全部修复，3次端到端管道测试全部通过（简单CSV、用户自定义列名CSV、大规模CSV）。
| 工作流 | 问题 | 状态 |
|--------|------|------|
| A1 | 重启后旧数据可分析 | ✅ 目录不存在时优雅降级，标记为failed |
| A2 | Web图标所有页面显示 | ✅ 创建favicon.svg + base.html添加link |
| A3 | 上传按钮在顶部 | ✅ 移至文件列表上方，sticky定位 |
| A10 | 文件上传限制 | ✅ 去除了Django字段限制，增大至256MB/50GB |
| A11 | 上传页删除功能 | ✅ 所有状态显示删除按钮，处理中弹出确认 |
| B7 | TLS 0ver hex格式 | ✅ 添加TLS_HEX_MAP，识别03 03/03 04 |
| B15 | "+"字符串识别为数字 | ✅ FLOAT检测添加+/前缀检查 |
| B18 | 列名映射 | ✅ 使用精确^exact_name$匹配，无模糊猜测 |
| C4 | Traceback截断 | ✅ tool_registry.py移除[:200] |
| D5 | 大数据量卡顿 | ✅ 添加head参数、50K降采样、MAX_ROWS=500 |
| D6 | 进度条 | ✅ 添加progress_pct/progress_msg字段 |
| D14 | 低内存崩溃 | ✅ MiniBatchKMeans、PCA降维、内存检查 |
| D16 | 空闲高IO | ✅ 日志级别已为INFO，无需修改 |
| E8 | Globe响应式缩放 | ✅ 动态H计算、vh单位、resize处理 |
| E9 | Globe ?data=参数 | ✅ 支持从SessionStore直接加载数据集 |
| F12 | LLM日志 | ✅ 添加callback机制、run_log字段 |
| G13 | 聚类图表 | ✅ 添加簇大小柱状图、Silhouette对比图 |
| H17 | 聚类质量 | ✅ 方差过滤、相关过滤、HDBSCAN自动调参 |
README.md
+390
View File
@@ -0,0 +1,390 @@
# 天璇 (TianXuan) — TLS 流数据分析与实体画像系统
基于 Polars + scikit-learn + Three.js 的 TLS 流数据分析工具链。支持 CSV 批量导入、自动实体检测、多维聚合、聚类分析、特征提取、**3D 地球流量可视化**。提供 Django Web 操作界面和 MCP 协议接口，支持 32B 大语言模型编排分析流程。**全离线运行**，所有 JS 库/地图贴图/GeoIP 数据库均已内嵌。
---
## 目录
1. [系统概述](#1-系统概述)
2. [快速开始](#2-快速开始)
3. [用户指南](#3-用户指南)
4. [页面功能说明](#4-页面功能说明)
5. [CLI 管道](#5-cli-管道)
6. [MCP 工具文档](#6-mcp-工具文档)
7. [测试](#7-测试)
8. [项目结构](#8-项目结构)
9. [常见问题](#9-常见问题)
---
## 1. 系统概述
### 1.1 解决的问题
企业网络中 TLS 流量通常以 CSV 形式导出（每行一条流记录），数据量大、列名不统一、缺乏标签。本系统提供一条从原始 CSV 到实体画像的自动化分析流水线：
```
CSV 文件 → 上传/加载 → 自动检测实体列 → 按实体聚合特征
→ 实体级聚类 → 提取每簇区分特征 → Web 可视化 / MCP 接口
```
### 1.2 核心能力
| 模块 | 功能说明 |
|------|---------|
| **数据加载** | 多文件合并、BOM 自动检测、跨文件 schema 容错、递归 `**/*.csv`、中文路径支持 |
| **实体检测** | 基于列名关键词 + 唯一值比率 + 数据类型的三维打分，自动推荐实体列（如 `src_ip`、`sni`、`user`） |
| **实体聚合** | 按实体分组计算流统计、目标多样性、TLS 特征、协议特征、时间模式、地理位置（经纬度） |
| **IP 子网聚合** | 支持 /24 和 /28 子网掩码，同一子网下所有 IP 合并为一个实体节点 |
| **多列复合键** | 支持 `(src_ip, dst_ip)` 等多列组合作为实体标识 |
| **聚类分析** | HDBSCAN（自动确定簇数）/ KMeans，零样本自动降级，Silhouette / Davies-Bouldin 质量评估 |
| **特征提取** | Z-Score / ANOVA 方法计算每个聚类的区分性特征 |
| **3D 地球** | 基于 Three.js 的 TLS 流量弧线可视化，多数据源叠加，经纬线网格，国境线轮廓 |
| **地理可视化** | PCA 散点图 + Canvas 原生渲染（无外部 JS 依赖） |
| **LLM 集成** | OpenAI 兼容 API、LLM 自动编排分析流程（profile → 实体 → 聚类 → 特征） |
| **Web 界面** | CSV 上传拖拽、手动分步分析向导、LLM 自动分析、配置管理、运行记录查看 |
| **MCP 协议** | 12 个 JSON-RPC 工具，供外部 LLM 编排调用 |
| **便携运行时** | 项目内置 Python 3.12 运行时（593MB），零系统依赖，`run.bat` 双击即用 |
### 1.3 技术栈
| 组件 | 技术选型 |
|------|---------|
| **后端框架** | Django 4.2 (SQLite WAL) |
| **数据处理** | Polars 1.42.1 (LazyFrame 延迟计算 + streaming) |
| **机器学习** | scikit-learn (StandardScaler, HDBSCAN, KMeans, PCA) |
| **MCP 协议** | Python MCP SDK (stdio 传输) |
| **前端 3D** | Three.js (603KB, 离线) |
| **前端图表** | 纯 Canvas 2D API 原生绘制 |
| **配置** | Pydantic + YAML |
| **LLM 接口** | OpenAI chat/completions 兼容 API |
| **运行时** | 嵌入式 Python 3.12 (Win7 兼容版 adang1345) |
---
## 2. 快速开始
### 2.1 用户模式（无需预装 Python）
```bat
1. 解压项目文件夹到任意目录
2. 双击 run.bat
3. 等待命令行显示 "Starting TianXuan..."
4. 浏览器自动打开 http://127.0.0.1:8000/
5. 点击「上传数据」→ 选择 CSV 文件 → 上传 → 等待预处理完成
6. 进入「手动分析」→ 选择数据集 → 配置参数 → 运行分析
7. 查看结果：聚类概览、簇详情、实体画像、地球分布图
```
### 2.2 开发者模式
```bash
# 初始化数据库
runtime\python\python.exe manage.py migrate
# 生成测试数据
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
# 启动开发服务器
runtime\python\python.exe manage.py runserver
# 浏览器打开 http://127.0.0.1:8000/
```
### 2.3 脚本一览
| 脚本 | 功能 |
|------|------|
| `run.bat` | 启动 Django Web 界面（:8000），自动打开浏览器 |
| `shell.bat` | 打开 Django Shell（Python 交互式环境） |
### 2.4 一键分析管道
```bash
# 加载 CSV → 自动检测实体 → 聚合 → 聚类 → 特征提取 → 写入数据库
runtime\python\python.exe manage.py run_pipeline "data/input.csv"
# 指定聚类算法
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --algo kmeans
# 手动指定实体列（跳过自动检测）
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --entity-col src_ip
# 导出结果
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --output ./results/
```
输出示例：
```
[1/5] 加载 CSV: data/test_flows.csv → 500 rows, 1 files, 0.2 MB
[2/5] 检测实体列 → 自动检测: src_ip (共 6 候选)
[3/5] 实体聚合 → 50 个实体, 18 维特征
[4/5] 聚类 (hdbscan) → 2 个簇, 噪声比 0.24, Silhouette: 0.2557
[5/5] 特征提取 → 30 个特征已保存到数据库
*** 分析完成! Run ID: #1
```
---
## 3. 用户指南
### 3.1 上传数据
1. 点击导航栏「上传数据」
2. 拖拽 CSV 文件到虚线区域，或点击选择文件
3. 支持多文件上传，自动拼接相同 schema 的文件
4. 支持 `schema_strict=false` 模式（默认）：不同列名的文件会自动合并，缺失列填 null
5. 支持 `.zip` 压缩包自动解压
6. 上传后自动在后台进行预处理（加载 → 检测实体列 → 聚合），完成后状态变为 `ready`
7. 进度条实时显示处理状态
**数据格式要求**：
- 每行一条 TLS 流记录
- 建议包含列：`src_ip`、`dst_ip`、`proto`、`bytes_sent`、`bytes_rev`、`duration`、`packets` 等
- 可选地理列：`latitude` / `lat` / `y` 和 `longitude` / `lon` / `lng` / `x`（自动识别）
- 可选时间列：`timestamp` / `ts` / `time`（自动识别）
### 3.2 手动分析
3 步向导完成分析：
**Step 1 - 选择数据集**：从已预处理完成的数据集中选择一个（显示行数、已检测的实体列）
**Step 2 - 配置参数**：
- 聚类算法：HDBSCAN（自动确定簇数，推荐）/ KMeans（需预设 k 值）
- 最小簇大小（min_cluster_size）：默认 5，数据不足时自动降级
**Step 3 - 确认运行**：回顾配置，点击「开始分析」。后台自动完成聚类 + 特征提取，完成后跳转到结果页。
### 3.3 LLM 自动分析
需要先在「配置」页面设置 LLM 的 base_url 和 API Key：
1. 进入「LLM 分析」页面
2. 选择一个已预处理的数据集
3. 点击「开始自动分析」
4. 后端 LLM 编排器会自动调用 profile → build_entity_profiles → run_clustering → extract_features
5. 进度条实时显示当前步骤
### 3.4 查看结果
#### 运行详情页
- 总流数、实体数、簇数 摘要卡片
- 簇列表：每个簇的大小、占比、Silhouette 分数
#### 聚类概览页
- **PCA 散点图**：实体在二维主成分空间中的分布，颜色区分聚类
- **地理分布图**：实体在地理坐标上的分布（需 CSV 包含经纬度列），支持缺失值
- 每个簇的详情卡片：Top 5 区分特征
#### 簇详情页
- 特征表：每列的均值、标准差、中位数、区分度分数（Top 50）
- 实体列表：该簇包含的实体（Top 50），可点击查看详情
#### 实体画像页
- 实体标识、所属簇
- 聚合特征值和相对簇均值的偏离（Z-score）
- 偏差显著的特征以颜色高亮（|Z| > 2）
#### 3D 地球页
- Three.js 3D 地球，TLS 流量弧线连接源目 IP
- 弧线颜色标识 TLS 版本（TLSv1.3 蓝 / v1.2 粉 / 其他青）
- 鼠标拖拽旋转、滚轮缩放
- 经纬线网格 + 国境线轮廓
- 多数据源叠加复选框
- 脉冲光点动画（背面剔除）
---
## 4. 页面功能说明
| 页面 | 路由 | 功能 |
|------|------|------|
| **首页** | `/` | 最近运行记录概览 |
| **上传数据** | `/upload/` | 拖拽上传 CSV，自动预处理 |
| **手动分析** | `/analyze/manual/` | 3 步向导：选数据→设参数→运行 |
| **LLM 分析** | `/analyze/auto/` | LLM 自动编排分析流程 |
| **运行记录** | `/runs/` | 所有分析运行列表 |
| **运行详情** | `/runs/<id>/` | 单次运行摘要 + 簇列表 |
| **聚类概览** | `/clusters/<run_id>/` | PCA 散点图 + 每簇区分特征 |
| **簇详情** | `/clusters/<run_id>/<label>/` | 特征表 + 实体列表 |
| **实体画像** | `/entities/<id>/` | 单实体特征 + 簇偏离 |
| **3D 地球** | `/globe/` | Three.js 流量弧线可视化 |
| **配置** | `/config/` | 系统配置编辑 + LLM 测试 |
| **日志** | `/logs/` | 运行日志查看 |
---
## 5. CLI 管道
```bash
# 完整管道（HDBSCAN 默认）
runtime\python\python.exe manage.py run_pipeline data/complex_test.csv
# 指定 KMeans
runtime\python\python.exe manage.py run_pipeline data/*.csv --algo kmeans
# 手动指定实体列
runtime\python\python.exe manage.py run_pipeline data/*.csv --entity-col src_ip
# 设置子网聚合掩码
runtime\python\python.exe manage.py run_pipeline data/*.csv --subnet-masks 24 28
```
---
## 6. MCP 工具文档
### 6.1 工具总览
系统通过 MCP (Model Context Protocol) 暴露 12 个工具，支持 LLM 以 JSON-RPC 方式编排调用。
| # | 工具名 | 功能 | 必需参数 |
|---|--------|------|----------|
| 1 | `load_data` | 加载 CSV 文件（支持 glob、递归、schema 容错） | `csv_glob` |
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
| 3 | `filter_data` | 按条件过滤（12 操作符 + AND/OR） | `dataset_id`, `filters` |
| 4 | `preprocess_data` | 预处理（标准化/编码/填充） | `dataset_id`, `columns` |
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans，带质量评估） | `dataset_id`, `cluster_columns` |
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH） | `cluster_result_id`, `dataset_id` |
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA） | `dataset_id`, `cluster_result_id` |
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON） | `result_id`, `output_path` |
| 9 | `list_datasets` | 列出所有活跃数据集 | 无 |
| 10 | `drop_dataset` | 删除数据集释放内存 | `dataset_id` |
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据） | `dataset_id` |
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
### 6.2 响应约定
- 成功时返回 JSON 对象，含 `truncated: false`
- 超过阈值时自动截断，设置 `truncated: true` + `omitted_columns`
- 错误时返回 `{ "error": "描述", "truncated": false }`
- 所有响应通过 `TextContent` 包装返回
### 6.3 JSON-RPC 调用示例
```
→ {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"load_data","arguments":{"csv_glob":"data/*.csv"}}}
← {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"{\"dataset_id\":\"ds_...\",\"row_count\":500}"}]}}
→ {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"build_entity_profiles","arguments":{"dataset_id":"ds_...","auto_detect":true}}}
← {"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"{\"entity_dataset_id\":\"entity_...\",\"entity_count\":50,\"entity_column_used\":\"src_ip\"}"}]}}
```
---
## 7. 测试
```bash
# 运行全部测试
runtime\python\python.exe -m pytest tests -q
# 运行特定测试文件
runtime\python\python.exe -m pytest tests/test_entity.py -v
# 运行端到端测试
runtime\python\python.exe -m pytest tests/test_e2e.py -v
```
### 生成测试数据
```bash
# 简单数据
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
# Globe 模式（IP 在真实 GeoIP 范围内，弧线可见）
runtime\python\python.exe scripts\gen_test_data.py --globe
# 复杂数据（5000 行 24 列含 55% 缺失）
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
```
### 列结构调查
```bash
# 统计各 CSV 文件的列名、类型、分布
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
```
---
## 8. 项目结构
```
tianxuan/
├── tianxuan/ # Django 项目配置
│ ├── settings.py # Django 配置（数据库、中间件）
│ ├── urls.py # 根路由
│ └── llm_orchestrator.py # LLM 编排后端（tool_calls 循环）
│
├── analysis/ # 核心分析模块（Django app）
│ ├── data_loader.py # CSV 加载、BOM 检测、schema 容错、递归 glob
│ ├── data_profiler.py # 数据集概要统计与相关性矩阵
│ ├── entity_detector.py # 实体列自动检测（关键词 + unique_ratio 打分）
│ ├── entity_aggregator.py # 实体聚合（流→实体画像），含 IP 子网聚合
│ ├── type_classifier.py # 值优先类型检测（MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON）
│ ├── data_validator.py # 列校验（缺失率/异常值/IP有效性）
│ ├── geoip.py # GeoIP 经纬度查询
│ ├── session_store.py # 线程安全内存会话存储（Singleton + RLock）
│ ├── tool_registry.py # 12 个 MCP 工具定义 + 异步处理函数
│ ├── mcp_server.py # MCP stdio 服务器
│ ├── views.py # 所有 Django 视图
│ ├── urls.py # 16 条 URL 路由
│ ├── models.py # ORM 模型
│ └── management/commands/
│ ├── start_mcp.py # manage.py start_mcp
│ └── run_pipeline.py # manage.py run_pipeline（一键 CLI 管道）
│
├── config/
│ ├── config.yaml # 配置文件（自动生成，带注释）
│ └── loader.py # Pydantic 配置加载器
│
├── templates/tianxuan/ # 11 个页面模板
├── static/tianxuan/ # Three.js + 地球贴图 + 国境线
├── runtime/python/ # 便携 Python 3.12 运行时（593MB）
├── tests/ # 53 个测试 (pytest)
├── scripts/ # 工具脚本
├── docs/ # 文档
├── run.bat # 用户启动脚本（双击即用）
└── manage.py # Django 管理入口
```
---
## 9. 常见问题
### Q1: 双击 run.bat 后浏览器没打开？
A: 手动打开浏览器访问 `http://127.0.0.1:8000/`。如果也无法访问，检查命令行是否有报错。
### Q2: 压缩时提示 `db.sqlite3-wal 被占用`？
A: 有残留的 Python 进程持有 SQLite 连接。运行以下命令后重试：
```powershell
Get-Process python -ErrorAction SilentlyContinue | Stop-Process -Force
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
```
### Q3: 地图不显示？
A: CSV 需包含 lat/lon 列（列名含 `lat`/`latitude`/`lon`/`longitude`/`lng` 自动识别），缺失值不显示。页面会提示跳过的数量。
### Q4: 上传 CSV 后显示 "处理失败"？
A: 在运行详情页查看错误消息。常见原因：
- CSV 编码不是 UTF-8
- 所有列都是非数值类型（无法聚类，返回 0 簇）
- 数据量太小（< 3 条实体记录，HDBSCAN 自动跳过）
### Q5: 如何配置 LLM？
A: 导航到「配置」页面，填写 base_url 和 API Key，点击「测试连通性」验证。支持 OpenAI 兼容的任何 API。
### Q6: 支持哪些 CSV 编码？
A: 自动检测 UTF-8 BOM、UTF-16LE BOM、UTF-16BE BOM。无 BOM 时默认 UTF-8。中文路径完全支持。
### Q7: 不同 CSV 文件的列名不一致怎么办？
A: 默认 `schema_strict=false`，系统自动取所有文件的列名并集，缺失列填 null。如需严格模式，在配置中将 `schema_strict` 改为 `true`。
### Q8: 项目太大，如何缩小？
```powershell
Remove-Item -Recurse *.pyc, __pycache__ -Force
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
```
```
analysis/__init__.py
View File
analysis/admin.py
+29
View File
@@ -0,0 +1,29 @@
from django.contrib import admin
from .models import AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
@admin.register(AnalysisRun)
class AnalysisRunAdmin(admin.ModelAdmin):
list_display = ('id', 'created_at', 'status', 'csv_glob', 'entity_count', 'cluster_count')
list_filter = ('status', 'created_at')
ordering = ('-created_at',)
@admin.register(ClusterResult)
class ClusterResultAdmin(admin.ModelAdmin):
list_display = ('run', 'cluster_label', 'size', 'noise_ratio', 'silhouette_score')
list_filter = ('run',)
@admin.register(EntityProfile)
class EntityProfileAdmin(admin.ModelAdmin):
list_display = ('run', 'entity_value', 'cluster_label', 'embedding_x', 'embedding_y')
list_filter = ('run', 'cluster_label')
search_fields = ('entity_value',)
@admin.register(ClusterFeature)
class ClusterFeatureAdmin(admin.ModelAdmin):
list_display = ('cluster', 'feature_name', 'mean', 'std', 'distinguishing_score')
list_filter = ('cluster__run',)
ordering = ('-distinguishing_score',)
analysis/apps.py
+7
View File
@@ -0,0 +1,7 @@
from django.apps import AppConfig
class AnalysisConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'analysis'
label = 'tianxuan_analysis'
analysis/data_loader.py
+494
View File
@@ -0,0 +1,494 @@
"""CSV loading with per-file schema validation, BOM detection, and multi-file merge.
Supports multiple encodings and delimiter-separated files. The main entry
point is :func:`load_csv_directory`, which globs files, detects BOM per file,
validates schema consistency, and returns a merged :class:`pl.LazyFrame`.
"""
import glob
import logging
import os
from pathlib import Path
from typing import Optional
import polars as pl
import yaml
from analysis.type_classifier import classify_schema, DataType
logger = logging.getLogger(__name__)
SUPPORTED_ENCODINGS = ['utf-8', 'utf-8-sig', 'utf-16le', 'utf-16be', 'latin-1']
# ── numeric coercion helper ──────────────────────────────────────────────
def _coerce_to_float(series: pl.Series) -> pl.Series:
"""Convert string series to Float64.
Every entry that doesn't parse as a valid float becomes null.
Handles blank strings, standalone '+'/'-', 'N/A', 'null', 'None', etc.
"""
series = series.cast(pl.Utf8).str.strip_chars()
artifacts = ('', '+', '-', '.', '..', 'N/A', 'n/a', 'NA', 'null', 'None', 'NULL', '?', '---', '--', '/')
is_artifact = series.is_in(pl.Series(artifacts).implode())
result = []
for v, bad in zip(series.to_list(), is_artifact.to_list()):
if v is None or bad:
result.append(None)
else:
# Check for "+"/"-" prefix artifacts (e.g., "+abc", "-xyz")
# where the value isn't a genuine numeric string. Values like
# "+5" or "-3.14" are still accepted as valid floats below.
if len(v) > 1 and v[0] in '+-':
try:
float(v[1:]) # validate suffix minus prefix
except ValueError:
result.append(None)
continue
try:
result.append(float(v))
except (ValueError, TypeError):
result.append(None)
return pl.Series(result, dtype=pl.Float64)
# ── BOM detection ───────────────────────────────────────────────────────
def detect_bom(path: str) -> str:
"""Read the first 4 bytes of *path* and return the detected encoding.
Returns ``'utf-8'`` when no BOM is found.
Note: Python 3 ``open()`` fully supports Unicode/Chinese paths on all
platforms (no encoding dance needed).
"""
with open(path, 'rb') as f:
header = f.read(4)
if header[:3] == b'\xef\xbb\xbf':
return 'utf-8-sig'
if header[:2] == b'\xff\xfe':
return 'utf-16le'
if header[:2] == b'\xfe\xff':
return 'utf-16be'
return 'utf-8'
# ── schema utilities ────────────────────────────────────────────────────
def collect_schema(
path: str,
encoding: str = 'utf-8',
delimiter: str = ',',
sample_rows: int = 10000,
) -> dict:
"""Return {column_name: dtype_string} for a single CSV file.
Uses :func:`pl.scan_csv` with *sample_rows* rows for type inference.
"""
lf = pl.scan_csv(
str(path), # str() ensures Unicode/Chinese paths resolve correctly
encoding=encoding,
separator=delimiter,
infer_schema_length=sample_rows,
)
schema = lf.collect_schema()
return {name: str(dtype) for name, dtype in zip(schema.names(), schema.dtypes())}
def _merge_schema_entries(
schemas: list[dict],
) -> dict:
"""Merge multiple schema dicts, raising on type conflicts.
Returns a single {name: dtype} dict. If a column appears in multiple
files with different dtypes the *last* non-null dtype wins (with a
warning embedded in the result).
"""
merged: dict[str, str] = {}
conflicts: list[str] = []
for s in schemas:
for name, dtype in s.items():
if name in merged and merged[name] != dtype:
conflicts.append(name)
merged[name] = dtype # last wins
return merged, conflicts
def validate_schemas(schemas: list[dict]) -> None:
"""Compare schemas across files and raise :class:`ValueError` on mismatch.
Checks that every file has the same set of column names. Type
differences across files are **not** considered an error (Polars can
cast), but differences in *column sets* are.
"""
if not schemas:
return
col_sets = [set(s.keys()) for s in schemas]
reference = col_sets[0]
for i, cs in enumerate(col_sets[1:], start=1):
only_in_ref = reference - cs
only_in_current = cs - reference
if only_in_ref or only_in_current:
msg_parts = []
if only_in_ref:
msg_parts.append(f"File #{i} missing columns: {sorted(only_in_ref)}")
if only_in_current:
msg_parts.append(f"File #{i} has extra columns: {sorted(only_in_current)}")
raise ValueError(f"Schema mismatch: {'; '.join(msg_parts)}")
# ── dtype inference from schema ─────────────────────────────────────────
def detect_schema_dtypes(
lf: pl.LazyFrame,
sample_rows: int = 10000,
) -> dict:
"""Infer Polars dtype strings from a LazyFrame's schema.
The schema is read without collecting the full data (the scan scans
a small sample).
"""
schema = lf.collect_schema()
return {name: str(dtype) for name, dtype in zip(schema.names(), schema.dtypes())}
# ── config loading ──────────────────────────────────────────────────────
def load_config(config_path: Optional[str]) -> dict:
"""Load a YAML config file; returns an empty dict if *config_path* is None."""
if config_path is None:
return {}
path = Path(config_path)
if not path.exists():
raise FileNotFoundError(f"Config file not found: {config_path}")
with open(path, 'r', encoding='utf-8') as f:
return yaml.safe_load(f) or {}
# ── helpers ─────────────────────────────────────────────────────────────
def _file_count(paths: list[str]) -> int:
return len(paths)
def _resolve_encoding(encoding: str, file_encoding: str) -> str:
"""BOM-detected encoding overrides user-supplied encoding.
Polars >= 1.0 requires ``'utf8'`` (not ``'utf-8'``).
"""
resolved = file_encoding if file_encoding != 'utf-8' else encoding
return resolved.replace('-', '') if resolved in ('utf-8', 'utf-8-sig') else resolved
# ── main entry ──────────────────────────────────────────────────────────
def load_csv_directory(
glob_pattern: str,
encoding: str = 'utf-8',
delimiter: str = ',',
config_path: Optional[str] = None,
schema_override: Optional[dict] = None,
sample_rows: int = 10000,
schema_strict: bool = False,
recursive: bool = False,
head: Optional[int] = None,
) -> tuple[pl.LazyFrame, dict, int, int, float]:
"""Load, validate, and merge CSV files matching *glob_pattern*.
Args:
glob_pattern: Glob pattern for CSV files (e.g. ``data/*.csv``).
encoding: Fallback encoding when no BOM is detected.
delimiter: Column delimiter character.
config_path: Optional path to a YAML config (merged into metadata).
schema_override: If given, force these {column: dtype} overrides.
sample_rows: Rows to scan for dtype inference per file.
schema_strict: When True, raise ValueError on column mismatch across
files. When False (default), log warnings and merge with
``diagonal_relaxed`` (missing columns get null).
recursive: When True, use ``glob.glob(..., recursive=True)`` to
support ``**/*.csv`` patterns.
head: Optional row limit for preview scenarios. When set, applies
``lf.head(head)`` before returning.
Returns:
A tuple of ``(lazyframe, schema_dict, total_row_count, file_count,
memory_estimate_mb)``.
Raises:
FileNotFoundError: If no files match the glob pattern.
ValueError: If schemas are inconsistent across files and
*schema_strict* is True.
"""
# Normalise path separators for cross-platform Unicode support
glob_pattern = os.path.normpath(glob_pattern)
# Resolve file list
paths = sorted(glob.glob(glob_pattern, recursive=recursive))
if not paths and not recursive:
# Fallback: try recursive pattern
paths = sorted(glob.glob(glob_pattern, recursive=True))
if not paths:
raise FileNotFoundError(f"No files match glob pattern: {glob_pattern}")
# Load config (user-provided or fall back to project default)
config = load_config(config_path)
if not config:
default_path = Path(__file__).parent.parent / 'config' / 'config.yaml'
if default_path.exists():
config = load_config(str(default_path))
# Collect per-file schema
schemas: list[dict] = []
lazyframes: list[pl.LazyFrame] = []
for path in paths:
file_encoding = detect_bom(path)
actual_encoding = _resolve_encoding(encoding, file_encoding)
lf = pl.scan_csv(
str(path), # str() ensures Unicode/Chinese paths resolve correctly
encoding=actual_encoding,
separator=delimiter,
infer_schema_length=max(sample_rows, 1000),
)
schema = detect_schema_dtypes(lf, sample_rows=sample_rows)
# Apply schema override for this file
if schema_override:
for col, dtype in schema_override.items():
if col in schema:
schema[col] = dtype
schemas.append(schema)
lazyframes.append(lf)
if schema_strict:
# Raise on column mismatch
validate_schemas(schemas)
else:
# Lenient mode: log warnings for column mismatches
col_sets = [set(s.keys()) for s in schemas]
reference = col_sets[0]
for i, cs in enumerate(col_sets[1:], start=1):
only_in_ref = reference - cs
only_in_current = cs - reference
if only_in_ref:
logger.warning(
"File #%d missing columns: %s (will be null)",
i + 1, sorted(only_in_ref),
)
if only_in_current:
logger.warning(
"File #%d has extra columns: %s (will be null in other files)",
i + 1, sorted(only_in_current),
)
# Merge schemas (unify column ordering — first file wins order)
merged_schema, _ = _merge_schema_entries(schemas)
# Concatenate all LazyFrames (union vertically)
merged_lf: pl.LazyFrame = pl.concat(lazyframes, how='diagonal_relaxed')
# Log schema and size estimate
total_row_count_est = len(paths) * (lazyframes[0].collect(streaming=True).height if lazyframes else 0)
logger.info(f'[LOAD] files={len(paths)} total_rows_est={total_row_count_est}')
for col_name, dtype in sorted(merged_schema.items()):
logger.info(f'[LOAD_SCHEMA] col={col_name} dtype={dtype}')
# Apply schema override on merged frame (cast columns)
if schema_override:
casts = {}
for col, dtype_str in schema_override.items():
if col in merged_schema:
try:
polars_dtype = getattr(pl, dtype_str.split('(')[0], None)
if polars_dtype:
casts[col] = pl.col(col).cast(polars_dtype)
except Exception:
pass # skip uncastable overrides silently
if casts:
merged_lf = merged_lf.with_columns(list(casts.values()))
# ── Cleaning: drop outdated columns ──────────────────────────────────
geoip_cfg = config.get('data', {}).get('geoip', {})
drop_cols = geoip_cfg.get(
'drop_columns',
[], # Empty by default — lat/lon columns are now cleaned, not dropped
)
existing_drop = [c for c in drop_cols if c in merged_schema]
if existing_drop:
merged_lf = merged_lf.drop(existing_drop)
merged_schema = {k: v for k, v in merged_schema.items() if k not in existing_drop}
logger.info('[CLEAN] dropped columns: %s', existing_drop)
# ── Cleaning: BOOL_ENUM normalisation ────────────────────────────────
try:
type_map = classify_schema(
merged_lf.head(1000),
config_overrides=config.get('columns', {}),
)
except Exception:
type_map = {}
bool_enum_cols = [
c for c, t in type_map.items()
if t == DataType.BOOL_ENUM and c in merged_schema
]
if bool_enum_cols:
# Only clean string-typed BOOL_ENUM columns; native Boolean is already clean
schema_dtypes = dict(zip(
merged_lf.collect_schema().names(),
merged_lf.collect_schema().dtypes(),
))
bool_exprs = []
cleaned_names: list[str] = []
for c in bool_enum_cols:
if schema_dtypes.get(c) not in (pl.Utf8, pl.String):
continue # native Boolean / numeric — skip
c_str = pl.col(c).cast(pl.Utf8).str.strip_chars()
cleaned = (
pl.when(c_str == "")
.then(pl.lit(None, pl.Utf8))
.when(c_str.str.to_lowercase() == "+")
.then(pl.lit("True"))
.when(c_str.str.to_lowercase() == "-")
.then(pl.lit("False"))
.otherwise(c_str)
)
bool_exprs.append(cleaned.alias(c))
cleaned_names.append(c)
if bool_exprs:
merged_lf = merged_lf.with_columns(bool_exprs)
for c in cleaned_names:
merged_schema[c] = 'Utf8'
logger.info('[CLEAN] BOOL_ENUM columns normalised: %s', cleaned_names)
# ── Cleaning: LAT_LON normalisation ────────────────────────────────
latlon_cols = [
c for c, t in type_map.items()
if t == DataType.LAT_LON and c in merged_schema
]
if latlon_cols:
schema_dtypes_ll = dict(zip(
merged_lf.collect_schema().names(),
merged_lf.collect_schema().dtypes(),
))
latlon_exprs = []
for c in latlon_cols:
dtype = schema_dtypes_ll.get(c)
if dtype in (pl.Utf8, pl.String):
latlon_exprs.append(
pl.col(c).map_batches(_coerce_to_float, return_dtype=pl.Float64).alias(c)
)
merged_schema[c] = 'Float64'
if latlon_exprs:
merged_lf = merged_lf.with_columns(latlon_exprs)
logger.info('[CLEAN] LAT_LON columns normalised to Float64: %s', latlon_cols)
# ── Cleaning: Generic numeric coercion ────────────────────────────
# For any remaining string column (not IPv4/URL/HEX/ENUM/etc.),
# try to coerce to Float64. Whatever can't parse becomes null.
# This prevents "mean on str column" crashes for ANY column.
_SKIP_NUM_COERCE = (DataType.IPv4, DataType.URL, DataType.HEX,
DataType.ENUM, DataType.LAT_LON)
# Also protect columns with these name patterns (MAC addresses, etc.)
_SKIP_NAME_PATTERNS = ('mac', 'mac_', '_mac', 'addr', 'address')
schema_dtypes_gn = dict(zip(
merged_lf.collect_schema().names(),
merged_lf.collect_schema().dtypes(),
))
generic_numeric_cols = []
for c, t in type_map.items():
if t in _SKIP_NUM_COERCE:
continue
if c not in merged_schema:
continue
dtype = schema_dtypes_gn.get(c)
if dtype not in (pl.Utf8, pl.String):
continue # already numeric
# Skip columns with protected name patterns
cl = c.lower().replace('-', '_').replace(' ', '_')
if any(p in cl for p in _SKIP_NAME_PATTERNS):
continue
generic_numeric_cols.append(c)
if generic_numeric_cols:
num_exprs = []
for c in generic_numeric_cols:
num_exprs.append(
pl.col(c).map_batches(_coerce_to_float, return_dtype=pl.Float64).alias(c)
)
if num_exprs:
merged_lf = merged_lf.with_columns(num_exprs)
for c in generic_numeric_cols:
merged_schema[c] = 'Float64'
logger.info('[CLEAN] Generic numeric coercion applied: %s', generic_numeric_cols)
# ── Cleaning: HEX preprocessing (hex_pairs_count) ────────────────────
hex_cols = [
c for c, t in type_map.items()
if t == DataType.HEX and c in merged_schema
]
if hex_cols:
hex_exprs = []
for c in hex_cols:
hex_exprs.append(
pl.col(c).str.split(" ").list.len().alias(f"{c}_hex_pairs_count")
)
merged_lf = merged_lf.with_columns(hex_exprs)
for c in hex_cols:
merged_schema[f"{c}_hex_pairs_count"] = 'UInt32'
logger.info('[CLEAN] HEX columns added hex_pairs_count: %s', hex_cols)
# ── Value normalization (TLS hex→enum, display→enum) ────────────────
from analysis.value_normalizer import normalize_lf
merged_lf = normalize_lf(merged_lf, type_map)
# Only add _norm columns to schema if the source column exists in type_map
_NORM_MAP = [
('0ver', '0ver_norm'),
('0cph', '0cph_norm'),
('cipher-suite', 'cipher_suite_norm'),
('0crv', '0crv_norm'),
('ecdhe-named-curve', 'ecdhe_named_curve_norm'),
('cnrs', 'cnrs_norm'),
('isrs', 'isrs_norm'),
]
_added = []
for src, tgt in _NORM_MAP:
if src in type_map:
merged_schema[tgt] = 'Utf8'
_added.append(tgt)
if _added:
logger.info('[CLEAN] Value normalization columns added: %s', _added)
# Row count estimate (fast path: use first file row count × files)
# For accurate count we'd need to collect, but that defeats lazy.
# We estimate from the first file's scanned schema info.
first_rows = lazyframes[0].collect(streaming=True).height if lazyframes else 0
total_row_count = first_rows * len(paths)
# Memory estimate
bytes_per_row = 0
for col_name, dtype_str in merged_schema.items():
numeric_part = dtype_str.split('(')[0].lower()
size_map = {
'int8': 1, 'int16': 2, 'int32': 4, 'int64': 8,
'uint8': 1, 'uint16': 2, 'uint32': 4, 'uint64': 8,
'float32': 4, 'float64': 8,
'boolean': 1, 'bool': 1,
'utf8': 50, 'string': 50, 'str': 50,
'categorical': 8, 'cat': 8,
'date': 4, 'datetime': 8, 'time': 8, 'duration': 8,
}
bytes_per_row += size_map.get(numeric_part, 8)
memory_mb = (bytes_per_row * total_row_count) / (1024 * 1024)
# Apply head limit for preview (keeps LazyFrame lazy — rows are limited on collect)
if head is not None:
merged_lf = merged_lf.head(head)
total_row_count = min(total_row_count, head)
return merged_lf, merged_schema, total_row_count, len(paths), memory_mb
analysis/data_profiler.py
+333
View File
@@ -0,0 +1,333 @@
"""Data profiling for LLM decision-making.
Produces column-level statistics and a correlation matrix, with built-in
context‑window truncation so the output stays under a configurable budget
(default 16 KB).
"""
import json
from typing import Any, Optional
import polars as pl
import numpy as np
MAX_PROFILE_SIZE_KB = 16
"""Maximum serialised profile size before truncation kicks in."""
MAX_SAMPLE_ROWS = 5000
"""Maximum rows to collect for profiling."""
MAX_COLUMNS_FULL = 100
"""Beyond this column count the profile is always truncated."""
MAX_CORRELATION_COLS = 20
"""Maximum numeric columns to include in the correlation matrix."""
_ENTITY_KEYWORDS = frozenset({
'ip', 'host', 'domain', 'user', 'client', 'src_ip', 'dst_ip',
'source', 'destination', 'address', 'entity', 'name', 'id',
'username', 'hostname', 'mac', 'email', 'phone', 'session',
})
def _is_float_dtype(dtype: pl.DataType) -> bool:
return dtype in (pl.Float32, pl.Float64)
def _is_int_dtype(dtype: pl.DataType) -> bool:
return dtype in (pl.Int8, pl.Int16, pl.Int32, pl.Int64,
pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64)
def _is_numeric_dtype(dtype: pl.DataType) -> bool:
return _is_float_dtype(dtype) or _is_int_dtype(dtype)
def _is_string_dtype(dtype: pl.DataType) -> bool:
return dtype in (pl.Utf8, pl.String, pl.Categorical)
def _is_temporal_dtype(dtype: pl.DataType) -> bool:
return dtype in (pl.Date, pl.Datetime, pl.Time, pl.Duration)
# ── column statistics ───────────────────────────────────────────────────
def _compute_column_stats(series: pl.Series) -> dict:
"""Return a dict of statistics for a single column, based on its dtype.
Numeric columns get min/max/mean/std/median/percentiles.
String columns get unique count, missing count, and top-5 frequencies.
Boolean columns get true/false/null counts.
Temporal columns get min/max/null_count.
"""
dtype = series.dtype
stats: dict = {
'dtype': str(dtype),
'null_count': int(series.null_count()),
'non_null_count': int(len(series) - series.null_count()),
}
if _is_numeric_dtype(dtype):
non_null = series.drop_nulls()
if len(non_null) > 0:
stats.update({
'min': _safe_float(non_null.min()),
'max': _safe_float(non_null.max()),
'mean': _safe_float(non_null.mean()),
'std': _safe_float(non_null.std()),
'median': _safe_float(non_null.median()),
'q25': _safe_float(non_null.quantile(0.25)),
'q75': _safe_float(non_null.quantile(0.75)),
'unique_count': int(non_null.n_unique()),
})
else:
stats.update({
'min': None, 'max': None, 'mean': None, 'std': None,
'median': None, 'q25': None, 'q75': None, 'unique_count': 0,
})
stats['type_category'] = 'numeric'
elif _is_string_dtype(dtype):
non_null = series.drop_nulls()
stats['unique_count'] = int(non_null.n_unique()) if len(non_null) > 0 else 0
stats['type_category'] = 'categorical'
# Top-5 value counts
if len(non_null) > 0:
vc = non_null.value_counts().sort('count', descending=True).head(5)
# Polars >=1.0 value_counts() uses original col name, not 'index'
val_col = vc.columns[0]
stats['top_values'] = [
{'value': str(row[val_col]), 'count': int(row['count'])}
for row in vc.iter_rows(named=True)
]
else:
stats['top_values'] = []
# Detect entity column
stats['is_entity_candidate'] = str(series.name).lower() in _ENTITY_KEYWORDS
elif dtype == pl.Boolean:
non_null = series.drop_nulls()
if len(non_null) > 0:
true_count = int(non_null.sum())
stats.update({
'true_count': true_count,
'false_count': int(len(non_null)) - true_count,
})
else:
stats.update({'true_count': 0, 'false_count': 0})
stats['type_category'] = 'boolean'
else:
stats['type_category'] = 'other'
stats['unique_count'] = int(series.drop_nulls().n_unique())
return stats
def _safe_float(val: Any) -> Optional[float]:
"""Convert *val* to float or return None on failure."""
if val is None:
return None
try:
f = float(val)
return None if np.isnan(f) or np.isinf(f) else round(f, 6)
except (ValueError, TypeError):
return None
# ── correlation matrix ──────────────────────────────────────────────────
def _compute_correlation_matrix(
df: pl.DataFrame,
top_n: int = MAX_CORRELATION_COLS,
) -> Optional[list[dict]]:
"""Pearson correlation for numeric columns.
Returns a list of ``{column_i, column_j, correlation}`` dicts for the
top *top_n* columns by mean absolute correlation, sorted descending by
absolute value. Returns *None* when fewer than 2 numeric columns exist.
"""
numeric_cols = [
col for col in df.columns
if _is_numeric_dtype(df[col].dtype)
]
if len(numeric_cols) < 2:
return None
# Limit to top_n numeric columns (pick by variance — highest first)
if len(numeric_cols) > top_n:
variances = []
for col in numeric_cols:
v = df[col].drop_nulls().std()
variances.append((col, 0.0 if v is None else float(v)))
variances.sort(key=lambda x: x[1], reverse=True)
numeric_cols = [c for c, _ in variances[:top_n]]
# Build correlation matrix
try:
import pandas as pd
pdf = df.select(numeric_cols).to_pandas()
corr = pdf.corr(method='pearson')
pairs: list[dict] = []
for i, col_i in enumerate(numeric_cols):
for col_j in numeric_cols[i + 1:]:
val = corr.loc[col_i, col_j]
if np.isnan(val):
continue
pairs.append({
'column_i': col_i,
'column_j': col_j,
'correlation': round(float(val), 4),
})
except ImportError:
# Pandas not available — compute simple correlation matrix via numpy
data = df.select(numeric_cols).to_numpy()
data = np.nan_to_num(data, nan=0.0)
corr_np = np.corrcoef(data.T)
pairs = []
for i in range(len(numeric_cols)):
for j in range(i + 1, len(numeric_cols)):
val = corr_np[i, j]
if np.isnan(val):
continue
pairs.append({
'column_i': numeric_cols[i],
'column_j': numeric_cols[j],
'correlation': round(float(val), 4),
})
# Sort by absolute correlation descending
pairs.sort(key=lambda x: abs(x['correlation']), reverse=True)
return pairs[:top_n * 2] # cap output
# ── truncation ──────────────────────────────────────────────────────────
def _estimate_json_size(obj: Any) -> int:
"""Return the length of *obj* when serialised to JSON."""
return len(json.dumps(obj, default=str, ensure_ascii=False))
def truncate_profile(
profile: dict,
max_kb: int = MAX_PROFILE_SIZE_KB,
) -> dict:
"""Remove low-value columns from *profile* until it fits in *max_kb*.
Sets ``truncated=True`` and adds ``omitted_columns`` if truncation
occurred. Columns with the least variance (or highest null rate) are
removed first.
"""
max_bytes = max_kb * 1024
if _estimate_json_size(profile) <= max_bytes:
profile['truncated'] = False
profile.setdefault('omitted_columns', [])
return profile
column_stats = profile.get('column_stats', {})
if not column_stats:
profile['truncated'] = True
return profile
# Score columns: low variance / high null rate → remove first
scored: list[tuple[str, float]] = []
for col_name, stats in column_stats.items():
null_rate = stats.get('null_count', 0) / max(stats.get('non_null_count', 1), 1)
# Numeric columns get a variance estimate
std = stats.get('std')
variance_score = float(std) if std is not None else 0.0
# Remove low-variance OR high-null columns
removal_score = -variance_score + null_rate * 100
scored.append((col_name, removal_score))
scored.sort(key=lambda x: x[1]) # most valuable first
kept_columns = []
omitted_columns = []
for col_name, _ in scored:
test_profile = dict(profile)
test_profile['column_stats'] = {
c: column_stats[c] for c in kept_columns + [col_name]
}
if _estimate_json_size(test_profile) <= max_bytes:
kept_columns.append(col_name)
else:
omitted_columns.append(col_name)
profile['column_stats'] = {
c: column_stats[c] for c in kept_columns
}
profile['truncated'] = True
profile['omitted_columns'] = omitted_columns
profile['truncation_note'] = (
f"{len(omitted_columns)} of {len(omitted_columns) + len(kept_columns)} "
f"columns omitted to stay under {max_kb} KB budget"
)
return profile
# ── main profiling entry point ──────────────────────────────────────────
def profile_dataset(
lf: pl.LazyFrame,
sample_size: int = 1000,
columns: Optional[list[str]] = None,
max_kb: int = MAX_PROFILE_SIZE_KB,
max_correlation_cols: int = MAX_CORRELATION_COLS,
) -> dict:
"""Compute column-level statistics and a correlation matrix.
Args:
lf: LazyFrame to profile.
sample_size: Maximum rows to collect (sampled if larger).
columns: Subset of columns to profile (``None`` = all).
max_kb: Truncation budget in KB.
max_correlation_cols: Max numeric columns for correlation.
Returns:
A dict with keys ``column_stats``, ``correlation_matrix``,
``truncated``, ``omitted_columns``, ``total_columns``,
``profiled_rows``, and ``sample_ratio``.
"""
# Collect (possibly sampled) dataframe
df = lf.collect(streaming=True)
total_rows = len(df)
sample_ratio = 1.0
if total_rows > sample_size:
df = df.sample(n=sample_size, seed=42)
sample_ratio = sample_size / total_rows
# Determine columns to profile
if columns:
profile_columns = [c for c in columns if c in df.columns]
else:
profile_columns = list(df.columns)
if len(profile_columns) > MAX_COLUMNS_FULL:
profile_columns = profile_columns[:MAX_COLUMNS_FULL]
# Per-column stats
column_stats: dict = {}
for col_name in profile_columns:
column_stats[col_name] = _compute_column_stats(df[col_name])
# Correlation matrix
correlation_matrix = _compute_correlation_matrix(df, top_n=max_correlation_cols)
result = {
'column_stats': column_stats,
'correlation_matrix': correlation_matrix,
'truncated': False,
'omitted_columns': [],
'total_columns': len(df.columns),
'profiled_columns': len(profile_columns),
'profiled_rows': len(df),
'total_rows': total_rows,
'sample_ratio': round(sample_ratio, 4),
}
return truncate_profile(result, max_kb=max_kb)
analysis/data_validator.py
+392
View File
@@ -0,0 +1,392 @@
"""
数据预处理校验阶段。在 load_csv_directory 之后、build_entity_profiles 之前自动运行。
不阻塞流程，只记录 warning 到日志。
"""
import logging
import json
from typing import Any, Optional
import polars as pl
import numpy as np
from analysis.session_store import SessionStore
from analysis.type_classifier import classify_column, DataType
from analysis.ip_clustering import ip_to_int
logger = logging.getLogger(__name__)
# Mapping from Polars dtype strings to DataType for schema comparison
_DTYPE_TO_DATATYPE: dict[str, DataType] = {
'Int8': DataType.INT, 'Int16': DataType.INT, 'Int32': DataType.INT, 'Int64': DataType.INT,
'UInt8': DataType.INT, 'UInt16': DataType.INT, 'UInt32': DataType.INT, 'UInt64': DataType.INT,
'Float32': DataType.FLOAT, 'Float64': DataType.FLOAT,
'Boolean': DataType.BOOL_ENUM, 'Bool': DataType.BOOL_ENUM,
'String': DataType.STRING, 'Utf8': DataType.STRING, 'Categorical': DataType.STRING,
'Date': DataType.TIMESTAMP, 'Datetime': DataType.TIMESTAMP,
'Time': DataType.TIMESTAMP, 'Duration': DataType.TIMESTAMP,
}
_MAX_OUTLIER_SAMPLE = 1000
"""Max rows to sample for Z-score outlier detection."""
_MAX_IPV4_INVALID_SAMPLES = 10
"""Max invalid IPv4 samples to include in the report."""
def _dtype_to_datatype(dtype_str: str) -> Optional[DataType]:
"""Map a Polars dtype string (e.g. ``'String'``, ``'Int64'``) to :class:`DataType`.
Strips any type parameters (e.g. ``Datetime(time_unit='us', time_zone=None)``)
before lookup.
"""
base = dtype_str.split('(')[0].strip()
return _DTYPE_TO_DATATYPE.get(base)
def _safe_float(val: Any) -> Optional[float]:
"""Convert *val* to float or return *None* on failure."""
if val is None:
return None
try:
f = float(val)
return None if np.isnan(f) or np.isinf(f) else round(f, 6)
except (ValueError, TypeError):
return None
def _is_acceptable_mismatch(schema_type: DataType, inferred_type: DataType) -> bool:
"""Return *True* if a schema vs. inferred type mismatch is benign.
Acceptable mismatches:
- Schema says ``STRING`` but classifier infers a more specific type
(``IPv4``, ``URL``, ``HEX``, etc.) — this is expected because
``data_loader`` only sees raw Polars dtypes while ``classify_column``
applies value-based heuristics.
- ``INT`` ↔ ``FLOAT`` — Polars can cast between numeric types freely.
- ``ENUM`` ↔ ``BOOL_ENUM`` — both are low-cardinality categorical types.
"""
if schema_type == DataType.STRING:
return True
if {schema_type, inferred_type} == {DataType.INT, DataType.FLOAT}:
return True
if {schema_type, inferred_type} == {DataType.ENUM, DataType.BOOL_ENUM}:
return True
return False
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def validate(dataset_id: str, strict: bool = False) -> dict:
"""运行完整校验，返回 JSON 报告。
校验项
------
1. **列类型** — 用 :func:`~analysis.type_classifier.classify_column`
判定每列类型，与 *schema* 对比矛盾。
2. **缺失率** — 每列 ``null_count / len``，>50% 标记为高风险。
3. **异常值** — 数值列的 ``min`` / ``max`` / ``std``，Z-score > 5 比例。
4. **枚举分布** — ``ENUM`` / ``BOOL_ENUM`` 的值频率，特殊符号
（``+``、空白）标记。
5. **IPv4 有效性** — 对 ``IPv4`` 类型列的每个值调用
:func:`~analysis.ip_clustering.ip_to_int` 验证。
输出
----
每条校验结果通过 :data:`logger` 以 ``[VALIDATE]`` 标记写入
``info`` 或 ``warning`` 级别。
参数
----------
dataset_id:
SessionStore 中的数据集 ID。
strict:
为 *True* 时遇到高风险项（缺失率 >50%、无效 IPv4 比例 >50%）
抛出 :class:`RuntimeError`。默认为 *False*（仅记录 warning）。
返回
-------
dict
包含 ``valid``、``columns``、``warnings``、``risks`` 等键的
校验报告。
"""
store = SessionStore()
entry = store.get_dataset(dataset_id)
if entry is None:
msg = f"Dataset '{dataset_id}' not found in SessionStore"
logger.error(f'[VALIDATE] {msg}')
return {'valid': False, 'columns': {}, 'warnings': [], 'risks': [msg]}
lf: pl.LazyFrame = entry['lazyframe']
schema: dict = entry.get('schema', {})
# Materialise the LazyFrame
try:
df = lf.collect(streaming=True)
except Exception as e:
msg = f"Failed to collect LazyFrame for dataset '{dataset_id}': {e}"
logger.error(f'[VALIDATE] {msg}')
return {'valid': False, 'columns': {}, 'warnings': [], 'risks': [msg]}
total_rows = len(df)
logger.info(f'[VALIDATE] dataset={dataset_id} rows={total_rows} columns={len(df.columns)}')
columns_report: dict[str, dict] = {}
all_warnings: list[str] = []
all_risks: list[str] = []
for col_name in df.columns:
series = df[col_name]
col_info: dict[str, Any] = {
'name': col_name,
'dtype': str(series.dtype),
}
col_warnings: list[str] = []
# ── 1. Column type classification ────────────────────────────────
inferred_type = classify_column(col_name, series, config_type=None)
col_info['inferred_type'] = inferred_type.name
schema_dtype_str = schema.get(col_name)
if schema_dtype_str:
schema_type = _dtype_to_datatype(schema_dtype_str)
col_info['schema_type'] = schema_type.name if schema_type else schema_dtype_str
if schema_type and schema_type != inferred_type:
if not _is_acceptable_mismatch(schema_type, inferred_type):
msg = (
f"Column '{col_name}': schema type ({schema_type.name}) "
f"differs from inferred type ({inferred_type.name})"
)
col_warnings.append(msg)
all_warnings.append(msg)
else:
col_info['schema_type'] = None
# ── 2. Missing rate ──────────────────────────────────────────────
null_count = int(series.null_count())
null_rate = null_count / total_rows if total_rows > 0 else 0.0
col_info['null_count'] = null_count
col_info['null_rate'] = round(null_rate, 4)
if null_rate > 0.5:
msg = (
f"Column '{col_name}': null rate {null_rate:.1%} "
f"exceeds 50% (high risk)"
)
col_warnings.append(msg)
all_risks.append(msg)
elif null_rate > 0.3:
msg = (
f"Column '{col_name}': null rate {null_rate:.1%} "
f"exceeds 30%"
)
col_warnings.append(msg)
all_warnings.append(msg)
# ── 3. Outlier detection (numeric columns) ───────────────────────
if inferred_type in (DataType.INT, DataType.FLOAT):
non_null = series.drop_nulls()
if len(non_null) > 1:
try:
col_info['min'] = _safe_float(non_null.min())
col_info['max'] = _safe_float(non_null.max())
col_info['std'] = _safe_float(non_null.std())
# Sample for Z-score computation
if len(non_null) > _MAX_OUTLIER_SAMPLE:
sample = non_null.sample(n=_MAX_OUTLIER_SAMPLE, seed=42)
else:
sample = non_null
num_values = sample.cast(pl.Float64).to_numpy()
mean = float(np.nanmean(num_values))
std = float(np.nanstd(num_values))
if std > 0:
z_scores = np.abs((num_values - mean) / std)
outlier_ratio = float(np.mean(z_scores > 5))
col_info['z_score_gt_5_ratio'] = round(outlier_ratio, 4)
if outlier_ratio > 0.05:
msg = (
f"Column '{col_name}': {outlier_ratio:.1%} "
f"values have |Z-score| > 5 (high outlier ratio)"
)
col_warnings.append(msg)
all_warnings.append(msg)
else:
col_info['z_score_gt_5_ratio'] = 0.0
except Exception as e:
col_info['numeric_error'] = str(e)
col_info['z_score_gt_5_ratio'] = 0.0
else:
col_info['z_score_gt_5_ratio'] = 0.0
# ── 4. Enum distribution ─────────────────────────────────────────
if inferred_type in (DataType.ENUM, DataType.BOOL_ENUM):
non_null = series.drop_nulls()
if len(non_null) > 0:
vc = non_null.value_counts()
val_col = vc.columns[0]
cnt_col = vc.columns[1]
total_non_null = len(non_null)
distribution: dict[str, dict] = {}
for row in vc.iter_rows(named=True):
val = str(row[val_col])
count = int(row[cnt_col])
distribution[val] = {
'count': count,
'ratio': round(count / total_non_null, 4),
}
col_info['distribution'] = distribution
# Flag special symbols (empty string, whitespace-only, +)
for val, info in distribution.items():
stripped = val.strip()
if stripped == '' or stripped == '+':
msg = (
f"Column '{col_name}': "
f"value {repr(val)} appears {info['count']} times"
)
col_warnings.append(msg)
all_warnings.append(msg)
break # one warning per column
# ── 5. IPv4 validation ───────────────────────────────────────────
if inferred_type == DataType.IPv4:
non_null = series.drop_nulls()
total_valid = 0
total_invalid = 0
invalid_samples: list[str] = []
for v in non_null:
ip_str = str(v).strip()
if ip_to_int(ip_str) is not None:
total_valid += 1
else:
total_invalid += 1
if len(invalid_samples) < _MAX_IPV4_INVALID_SAMPLES:
invalid_samples.append(ip_str)
col_info['ipv4_valid_count'] = total_valid
col_info['ipv4_invalid_count'] = total_invalid
col_info['ipv4_invalid_samples'] = invalid_samples
if total_invalid > 0:
total_ipv4 = total_valid + total_invalid
invalid_ratio = total_invalid / total_ipv4 if total_ipv4 > 0 else 1.0
sample_str = str(invalid_samples[:5])
msg = (
f"Column '{col_name}': {total_invalid}/{total_ipv4} "
f"({invalid_ratio:.1%}) values are invalid IPv4. "
f"Samples: {sample_str}"
)
col_warnings.append(msg)
if invalid_ratio > 0.5:
all_risks.append(msg)
else:
all_warnings.append(msg)
col_info['warnings'] = col_warnings
columns_report[col_name] = col_info
# ── Log all findings ─────────────────────────────────────────────────
for col_name, col_info in columns_report.items():
for w in col_info.get('warnings', []):
logger.warning(f'[VALIDATE] {w}')
report_data: dict[str, Any] = {
'valid': len(all_risks) == 0,
'dataset_id': dataset_id,
'total_rows': total_rows,
'total_columns': len(df.columns),
'columns': columns_report,
'warnings': all_warnings,
'risks': all_risks,
}
n_warn = len(all_warnings)
n_risk = len(all_risks)
logger.info(
f'[VALIDATE] Complete: valid={report_data["valid"]} '
f'warnings={n_warn} risks={n_risk}'
)
if strict and all_risks:
raise RuntimeError(
f'Validation strict mode: {len(all_risks)} risk(s) found:\n' +
'\n'.join(f' - {r}' for r in all_risks)
)
return report_data
# ---------------------------------------------------------------------------
# Formatted report
# ---------------------------------------------------------------------------
def report(dataset_id: str) -> str:
"""返回格式化的校验报告字符串。
参数
----------
dataset_id:
SessionStore 中的数据集 ID。
返回
-------
str
人类可读的格式化报告。
"""
result = validate(dataset_id, strict=False)
lines: list[str] = []
lines.append('=' * 60)
lines.append(f'Data Validation Report — dataset: {dataset_id}')
lines.append('=' * 60)
lines.append(f' Valid: {result["valid"]}')
lines.append(f' Total rows: {result.get("total_rows", "?")}')
lines.append(f' Total columns: {result.get("total_columns", "?")}')
lines.append(f' Warnings: {len(result.get("warnings", []))}')
lines.append(f' Risks: {len(result.get("risks", []))}')
lines.append('')
columns = result.get('columns', {})
for col_name, col_info in sorted(columns.items()):
lines.append(f' --- {col_name} ---')
lines.append(f' dtype: {col_info.get("dtype", "?")}')
lines.append(f' inferred: {col_info.get("inferred_type", "?")}')
lines.append(f' null_rate: {col_info.get("null_rate", "?")}')
if 'min' in col_info:
cmin = col_info.get('min')
cmax = col_info.get('max')
lines.append(f' range: [{cmin}, {cmax}]')
if 'z_score_gt_5_ratio' in col_info:
lines.append(f' |Z|>5 ratio: {col_info["z_score_gt_5_ratio"]}')
if 'distribution' in col_info:
dist = col_info['distribution']
top5 = sorted(dist.items(), key=lambda kv: kv[1]['count'], reverse=True)[:5]
top5_str = ', '.join(f'{k}={v["count"]}' for k, v in top5)
lines.append(f' top values: {top5_str}')
if 'ipv4_invalid_count' in col_info and col_info['ipv4_invalid_count'] > 0:
lines.append(
f' invalid IPv4: {col_info["ipv4_invalid_count"]} '
f'(samples: {col_info.get("ipv4_invalid_samples", [])})'
)
for w in col_info.get('warnings', []):
lines.append(f' ! {w}')
lines.append('')
if result.get('warnings'):
lines.append('Warnings:')
for w in result['warnings']:
lines.append(f' - {w}')
lines.append('')
if result.get('risks'):
lines.append('Risks:')
for r in result['risks']:
lines.append(f' # {r}')
lines.append('')
lines.append('=' * 60)
return '\n'.join(lines)
analysis/entity_aggregator.py
+412
View File
@@ -0,0 +1,412 @@
"""Aggregate flow-level data into entity-level profiles.
The main entry point is :func:`aggregate_by_entity`, which groups a Polars
:class:`LazyFrame` by one or more entity columns (e.g. ``src_ip``, ``sni``,
``user``) and computes per-entity aggregates across flow, destination, TLS,
protocol, and time dimensions.
Only features whose source columns exist in the provided schema are computed,
so the module works with any CSV schema without assumptions.
"""
from __future__ import annotations
import logging
from typing import Optional
import polars as pl
from .type_classifier import DataType
_agg_logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Keyword sets for schema-independent column discovery
# ---------------------------------------------------------------------------
# Flow stats — user exact column names only
_BYTES_SENT_KEYWORDS = ('8ack',)
_BYTES_REV_KEYWORDS = () # no user column for reverse bytes
_DURATION_KEYWORDS = ('4dur',)
_PACKETS_KEYWORDS = ('8ppk',)
_PACKETS_REV_KEYWORDS = () # no user column for reverse packets
# Destination diversity — user exact column names only
_DST_IP_KEYWORDS = (':ips', ':ipd')
_DST_PORT_KEYWORDS = (':prs', ':prd')
_DST_NET_KEYWORDS = () # no user column for destination network
# TLS features — user exact column names only
_CIPHER_KEYWORDS = ('cipher_suite', '0cph')
_TLS_VERSION_KEYWORDS = ('0ver',)
_SNI_KEYWORDS = ('snam', 'cnam')
# Protocol features — user exact column names only
_PROTO_KEYWORDS = ('1ipp',)
_TCP_FLAGS_KEYWORDS = () # no user column for TCP flags
# Geo-location features — user exact column names only
_LAT_KEYWORDS = ('latd',)
_LON_KEYWORDS = ('lond',)
# Time patterns — user exact column names only
_TIMESTAMP_KEYWORDS = ('time', 'timestamp', '8dbd')
# ---------------------------------------------------------------------------
# IP subnet helpers for prefix-based aggregation
# ---------------------------------------------------------------------------
def ip_to_subnet(ip_str: str, mask: int) -> str:
"""Convert ``10.0.1.15/24`` → ``10.0.1.0/24``, ``10.0.1.250/28`` →
``10.0.1.240/28``.
Invalid IPs / ``None`` / empty string are returned unchanged.
Supports ``/24`` (0–255 last octet) and ``/28`` (240–255 in steps of 16)
masks.
"""
if not ip_str or not isinstance(ip_str, str):
return ip_str
try:
parts = [int(o) for o in ip_str.strip().split('.')]
if len(parts) != 4 or any(p < 0 or p > 255 for p in parts):
return ip_str
ip_int = (parts[0] << 24) + (parts[1] << 16) + (parts[2] << 8) + parts[3]
subnet_int = ip_int & (0xFFFFFFFF << (32 - mask))
return (
f"{(subnet_int >> 24) & 255}."
f"{(subnet_int >> 16) & 255}."
f"{(subnet_int >> 8) & 255}."
f"{subnet_int & 255}/{mask}"
)
except (ValueError, IndexError, TypeError):
return ip_str
def apply_subnet_aggregation(
lf: pl.LazyFrame,
ip_columns: list[str],
subnet_masks: list[int],
) -> pl.LazyFrame:
"""Apply subnet masking to IP columns in the LazyFrame.
Each IP column is transformed to its subnet prefix for each mask.
If multiple masks exist, the first matching mask is applied
(earliest mask in *subnet_masks* wins).
"""
if not subnet_masks or not ip_columns:
return lf
schema = lf.collect_schema().names()
for mask in subnet_masks:
mask_exprs = []
for col in ip_columns:
if col in schema:
mask_exprs.append(
pl.col(col)
.map_batches(
lambda s, m=mask: s.cast(pl.Utf8).map_batches(
lambda x, mm=m: pl.Series(
[ip_to_subnet(v, mm) for v in x.to_list()]
),
return_dtype=pl.Utf8,
),
return_dtype=pl.Utf8,
)
.alias(col)
)
if mask_exprs:
lf = lf.with_columns(mask_exprs)
return lf
def _find_column(schema: dict[str, str], keywords: tuple[str, ...]) -> Optional[str]:
"""Return the first column in *schema* whose lowercased name matches
any of *keywords* (exact or ``_``-delimited match)."""
for col in schema:
normalised = col.lower().lstrip(':+').replace('-', '_').replace('.', '_').replace(' ', '_')
for kw in sorted(keywords):
kw_s = kw.lower().lstrip(':+')
if kw_s == normalised or normalised.endswith(f'_{kw_s}') or normalised.startswith(f'{kw_s}_'):
return col
return None
def _timestamp_col(schema: dict[str, str]) -> Optional[str]:
"""Detect a timestamp column -- user exact column names only."""
for preference in ('timestamp', 'time', '8dbd'):
if preference in schema:
return preference
return _find_column(schema, _TIMESTAMP_KEYWORDS)
# ---------------------------------------------------------------------------
# Aggregation plan builder
# ---------------------------------------------------------------------------
def build_aggregation_params(
schema: dict[str, str],
type_map: dict[str, DataType] | None = None,
) -> list[tuple[str, pl.Expr]]:
"""Build a list of ``(feature_name, pl.Expr)`` pairs for every aggregate
that can be computed from the columns present in *schema*.
Each expression is a column-level aggregation suitable for use inside
``pl.LazyFrame.group_by(...).agg(...)``.
"""
aggs: list[tuple[str, pl.Expr]] = []
# ── Flow stats ───────────────────────────────────────────────────────
# flow_count: always computed via count()
aggs.append(('flow_count', pl.len()))
bytes_sent_col = _find_column(schema, _BYTES_SENT_KEYWORDS)
if bytes_sent_col:
aggs.append(('total_bytes_sent', pl.sum(bytes_sent_col).cast(pl.Int64)))
# Also store raw column name for downstream use
aggs.append(('_bytes_sent_col', pl.lit(bytes_sent_col)))
aggs.append(('_bytes_sent_expr', pl.lit(bytes_sent_col)))
else:
aggs.append(('total_bytes_sent', pl.lit(None, dtype=pl.Int64)))
aggs.append(('_bytes_sent_col', pl.lit('')))
aggs.append(('_bytes_sent_expr', pl.lit('')))
_agg_logger.info(f'[FIND_COL] keywords=bytes_sent matched={bytes_sent_col} dtype={schema.get(bytes_sent_col, "N/A") if bytes_sent_col else "None"}')
packets_col = _find_column(schema, _PACKETS_KEYWORDS)
if packets_col:
aggs.append(('total_packets', pl.sum(packets_col).cast(pl.Int64)))
else:
aggs.append(('total_packets', pl.lit(None, dtype=pl.Int64)))
_agg_logger.info(f'[FIND_COL] keywords=packets matched={packets_col} dtype={schema.get(packets_col, "N/A") if packets_col else "None"}')
# ── Destination diversity ────────────────────────────────────────────
dst_ip_col = _find_column(schema, _DST_IP_KEYWORDS)
if dst_ip_col:
aggs.append(('unique_dst_ips', pl.n_unique(dst_ip_col)))
aggs.append(('_dst_ip_col', pl.lit(dst_ip_col)))
else:
aggs.append(('unique_dst_ips', pl.lit(None, dtype=pl.UInt32)))
aggs.append(('_dst_ip_col', pl.lit('')))
_agg_logger.info(f'[FIND_COL] keywords=dst_ip matched={dst_ip_col} dtype={schema.get(dst_ip_col, "N/A") if dst_ip_col else "None"}')
dst_port_col = _find_column(schema, _DST_PORT_KEYWORDS)
if dst_port_col:
aggs.append(('unique_dst_ports', pl.n_unique(dst_port_col)))
aggs.append(('_dst_port_col', pl.lit(dst_port_col)))
else:
aggs.append(('unique_dst_ports', pl.lit(None, dtype=pl.UInt32)))
aggs.append(('_dst_port_col', pl.lit('')))
_agg_logger.info(f'[FIND_COL] keywords=dst_port matched={dst_port_col} dtype={schema.get(dst_port_col, "N/A") if dst_port_col else "None"}')
# ── Protocol features ────────────────────────────────────────────────
proto_col = _find_column(schema, _PROTO_KEYWORDS)
if proto_col:
aggs.append(('unique_protocols', pl.n_unique(proto_col)))
aggs.append(('_proto_col', pl.lit(proto_col)))
else:
aggs.append(('unique_protocols', pl.lit(None, dtype=pl.UInt32)))
aggs.append(('_proto_col', pl.lit('')))
_agg_logger.info(f'[FIND_COL] keywords=proto matched={proto_col} dtype={schema.get(proto_col, "N/A") if proto_col else "None"}')
# ── Enhanced TLS analysis features ────────────────────────────────────
# has_sni: count of non-null SNI values
sni_col = _find_column(schema, _SNI_KEYWORDS)
if sni_col:
aggs.append(('has_sni', pl.col(sni_col).is_not_null().sum()))
aggs.append(('sni_missing_ratio', pl.col(sni_col).is_null().mean()))
aggs.append(('avg_sni_length', pl.col(sni_col).cast(pl.Utf8).str.len_bytes().mean()))
aggs.append(('max_sni_length', pl.col(sni_col).cast(pl.Utf8).str.len_bytes().max()))
# TLS version risk: look for _norm column
tls_norm = _find_column(schema, ('0ver_norm',))
if not tls_norm:
tls_norm = _find_column(schema, ('0ver',))
if tls_norm:
aggs.append(('tls_modern_ratio',
pl.when(pl.col(tls_norm).is_in(['tls_1_3'])).then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('tls_old_ratio',
pl.when(pl.col(tls_norm).is_in(['tls_1_0', 'tls_1_1', 'ssl_2_0', 'ssl_3_0'])).then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('unique_tls_versions', pl.n_unique(tls_norm)))
# Modern cipher ratio (AEAD = modern)
cph_norm = _find_column(schema, ('0cph_norm',))
if not cph_norm:
cph_norm = _find_column(schema, ('cipher_suite_norm',))
if cph_norm:
aggs.append(('aead_ratio',
pl.when(pl.col(cph_norm).str.contains('gcm|chacha20|poly1305')).then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('weak_cipher_ratio',
pl.when(pl.col(cph_norm).str.contains('cbc|rc4|null')).then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('ecdhe_ratio',
pl.when(pl.col(cph_norm).str.contains('ecdhe')).then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('unique_ciphers', pl.n_unique(cph_norm)))
# Non-standard port detection (:prd is often String type)
prd_col = _find_column(schema, (':prd',))
if prd_col:
prd_str = pl.col(prd_col).cast(pl.Utf8)
aggs.append(('non_standard_port_ratio',
pl.when(prd_str != '443').then(1)
.otherwise(0).cast(pl.Float64).mean()))
aggs.append(('port_443_ratio',
pl.when(prd_str == '443').then(1)
.otherwise(0).cast(pl.Float64).mean()))
# Bytes per packet analysis
bytes_col = _find_column(schema, ('8ack',))
pkts_col = _find_column(schema, ('8ppk',))
if bytes_col and pkts_col:
bpp = pl.col(bytes_col).cast(pl.Float64) / pl.col(pkts_col).cast(pl.Float64).clip(1)
aggs.append(('avg_bytes_per_packet', bpp.mean()))
aggs.append(('std_bytes_per_packet', bpp.std()))
aggs.append(('min_bytes_per_packet', bpp.min()))
aggs.append(('max_bytes_per_packet', bpp.max()))
# Duration analysis
dur_col = _find_column(schema, ('4dur',))
if dur_col:
aggs.append(('avg_duration', pl.col(dur_col).cast(pl.Float64).mean()))
aggs.append(('std_duration', pl.col(dur_col).cast(pl.Float64).std()))
aggs.append(('max_duration', pl.col(dur_col).cast(pl.Float64).max()))
# Recoverable/resumed ratio
cnrs_norm = _find_column(schema, ('cnrs_norm',))
if cnrs_norm:
aggs.append(('recoverable_ratio',
pl.when(pl.col(cnrs_norm) == 'yes').then(1)
.otherwise(0).cast(pl.Float64).mean()))
isrs_norm = _find_column(schema, ('isrs_norm',))
if isrs_norm:
aggs.append(('resumed_ratio',
pl.when(pl.col(isrs_norm) == 'yes').then(1)
.otherwise(0).cast(pl.Float64).mean()))
# Country diversity
dcnt_col = _find_column(schema, ('dcnt',))
if dcnt_col:
aggs.append(('countries_visited', pl.n_unique(dcnt_col)))
# ── Geo-location (latitude / longitude) ────────────────────────────
lat_col = _find_column(schema, _LAT_KEYWORDS)
if lat_col:
dtype_str = schema.get(lat_col, '').lower()
if 'str' in dtype_str or 'utf' in dtype_str:
lat_expr = pl.mean(pl.col(lat_col).cast(pl.Float64))
else:
lat_expr = pl.mean(lat_col)
aggs.append(('avg_latitude', lat_expr.cast(pl.Float64)))
else:
aggs.append(('avg_latitude', pl.lit(None, dtype=pl.Float64)))
_agg_logger.info(f'[FIND_COL] keywords=lat matched={lat_col} dtype={schema.get(lat_col, "N/A") if lat_col else "None"}')
lon_col = _find_column(schema, _LON_KEYWORDS)
if lon_col:
dtype_str = schema.get(lon_col, '').lower()
if 'str' in dtype_str or 'utf' in dtype_str:
lon_expr = pl.mean(pl.col(lon_col).cast(pl.Float64))
else:
lon_expr = pl.mean(lon_col)
aggs.append(('avg_longitude', lon_expr.cast(pl.Float64)))
else:
aggs.append(('avg_longitude', pl.lit(None, dtype=pl.Float64)))
_agg_logger.info(f'[FIND_COL] keywords=lon matched={lon_col} dtype={schema.get(lon_col, "N/A") if lon_col else "None"}')
# ── Time patterns ────────────────────────────────────────────────────
ts_col = _timestamp_col(schema)
if ts_col:
# For first/last seen we need min/max of the timestamp column
aggs.append(('first_seen', pl.min(ts_col)))
aggs.append(('last_seen', pl.max(ts_col)))
aggs.append(('_ts_col', pl.lit(ts_col)))
# Active hours: unique hour-of-day count
# We'll compute this after collecting by extracting hour from timestamps
aggs.append(('_has_timestamp', pl.lit(True)))
else:
aggs.append(('first_seen', pl.lit(None, dtype=pl.Utf8)))
aggs.append(('last_seen', pl.lit(None, dtype=pl.Utf8)))
aggs.append(('_ts_col', pl.lit('')))
aggs.append(('_has_timestamp', pl.lit(False)))
# ── Type-classifier features ──────────────────────────────────────
# HEX/URL/IPv4 feature extraction removed — avg_hex_pairs_count,
# unique_domains, and unique_24_networks are not useful for TLS analysis.
return aggs
# ---------------------------------------------------------------------------
# Main aggregation entry point
# ---------------------------------------------------------------------------
# Columns whose ``_``-prefixed names are internal bookkeeping, not user-facing
_INTERNAL_COLUMNS = tuple([
'_bytes_sent_col', '_dst_ip_col', '_dst_port_col',
'_proto_col', '_ts_col', '_has_timestamp',
'_bytes_sent_expr',
])
def aggregate_by_entity(
lf: pl.LazyFrame,
entity_columns: str | list[str],
schema: dict[str, str],
type_map: dict[str, DataType] | None = None,
) -> tuple[pl.LazyFrame, list[str]]:
"""Group flow-level data by *entity_columns* and compute per-entity features.
Parameters
----------
lf:
A Polars :class:`LazyFrame` containing flow-level data.
entity_columns:
Column name(s) to group by (e.g. ``'src_ip'``, or ``['src_ip', 'dst_ip']``).
schema:
Dictionary mapping column names to dtype strings (e.g. from
``SessionStore.get_dataset(...)['schema']``).
Returns
-------
tuple[pl.LazyFrame, list[str]]
``(aggregated_lazyframe, feature_column_names)`` where the lazyframe
has one row per entity and *feature_column_names* lists the
user-facing aggregate columns.
"""
# Normalise to list
if isinstance(entity_columns, str):
entity_columns = [entity_columns]
# Build aggregation expressions based on the schema (and optional type_map)
agg_params = build_aggregation_params(schema, type_map=type_map)
# Separate feature names and expressions
feature_names: list[str] = []
exprs: list[pl.Expr] = []
for name, expr in agg_params:
exprs.append(expr.alias(name))
if not name.startswith('_'):
feature_names.append(name)
# Check entity columns exist
missing = [c for c in entity_columns if c not in schema]
if missing:
raise ValueError(
f"Entity column(s) '{missing}' not found in schema. "
f"Available columns: {list(schema.keys())}"
)
# Perform aggregation
aggregated_lf = lf.group_by(entity_columns).agg(exprs)
# Post-process: compute active_hours_count if a timestamp column exists
# (We compute this as a second pass since it requires extraction logic)
any(name == '_has_timestamp' for name, _ in agg_params)
# Filter to only user-facing feature columns names (excluding _internal)
user_features = [f for f in feature_names if f not in _INTERNAL_COLUMNS]
return aggregated_lf, user_features
analysis/entity_detector.py
+177
View File
@@ -0,0 +1,177 @@
"""Auto-detect entity column from CSV schema using heuristic scoring.
Provides :func:`detect_entity_column` which scores all string/categorical
columns by keyword-name match, unique-value ratio, and dtype, returning
a ranked list of candidates and a recommended column.
Usage::
from analysis.entity_detector import detect_entity_column
result = detect_entity_column(dataset_id)
result['recommended'] # e.g. 'src_ip'
result['candidates'] # sorted list of scored columns
"""
from __future__ import annotations
from typing import Optional
import polars as pl
from .session_store import SessionStore
# ---------------------------------------------------------------------------
# Entity-related keywords (lowercased)
# ---------------------------------------------------------------------------
ENTITY_KEYWORDS: tuple[str, ...] = (
# User exact column names only — no generic/fuzzy keywords
':ips', ':ipd', ':prs', ':prd',
'server_ip', 'client_ip', 'scnt', 'dcnt',
'cnam', 'snam', '0ver', '4dur', '8ses', '2tmo', '4ksz',
'cnrs', 'isrs', '8ack', '8ppk', '8dbd',
'1ipp', '4dbn', 'tabl', 'name', 'source_node',
'cipher_suite', 'ecdhe_named_curve',
'0cph', '0crv', '0rnd', '0rnt', 'row', 'time', 'timestamp',
'latd', 'lond', 'ispn', 'orgn', 'city',
)
def _score_name(column: str) -> tuple[float, str]:
"""Score a column name against *ENTITY_KEYWORDS*.
Returns ``(score, matched_keyword_or_empty)`` where *score* is 3.0
for every matched keyword. Partial matches (keyword is a substring
of the lowercased column name) also count.
"""
col_lower = column.lower().replace('-', '_').replace('.', '_').replace(' ', '_')
for kw in ENTITY_KEYWORDS:
if kw == col_lower or col_lower.endswith(f'_{kw}') or col_lower.startswith(f'{kw}_'):
return 3.0, kw
return 0.0, ''
def detect_entity_column(dataset_id: str, entity_columns: Optional[list[str]] = None) -> dict:
"""Score all string/categorical columns and return candidates.
The scoring algorithm:
* **Name match** (``+3``): column name (normalised) is a keyword, or
ends/starts with a keyword via ``_`` separator.
* **Unique ratio** (0–10): columns whose ratio of unique values to
non-null values falls in the range ``[0.01, 0.50]`` receive up to
10 points (``min(ratio * 100, 10)``).
* **String dtype** (``+2``): ``Utf8`` / ``String`` / ``Categorical``
columns get a bonus.
* **Null penalty** (``*0.1``): if null ratio > 50%, score severely reduced.
* **Penalty** (``-5``): unique ratio > 0.50 (likely random IDs or
transaction-level identifiers).
Parameters
----------
dataset_id:
Dataset identifier in the session store.
entity_columns:
Optional list of column names to restrict detection to. When given,
only these columns are scored (must exist in the schema).
Returns
-------
dict
``candidates``: list of ``{column, score, unique_ratio, matched_keyword}``
sorted descending by score.
``recommended``: highest-scoring column name (or ``''`` when none found).
``recommended_multi``: list of the top 3 candidate column names (may be
shorter than 3 when fewer candidates exist).
``dataset_id``: the input identifier (echoed).
"""
store = SessionStore()
entry = store.get_dataset(dataset_id)
if entry is None:
return {
'dataset_id': dataset_id,
'candidates': [],
'recommended': '',
'error': f'Dataset not found: {dataset_id}',
}
lf: pl.LazyFrame = entry['lazyframe']
schema: dict[str, str] = entry.get('schema', {})
# Identify string / categorical columns
string_types = frozenset({'Utf8', 'String', 'Categorical', 'str', 'cat'})
candidate_scores: list[dict] = []
try:
# Collect a sample to compute unique ratios efficiently
df_sample = lf.collect(streaming=True)
except Exception as exc:
return {
'dataset_id': dataset_id,
'candidates': [],
'recommended': '',
'error': f'Failed to collect dataset: {exc}',
}
total_rows = len(df_sample)
for col_name, dtype_str in schema.items():
# If entity_columns specified, skip columns not in the list
if entity_columns is not None and col_name not in entity_columns:
continue
# Only score string-like columns
dtype_base = dtype_str.split('(')[0].strip()
if dtype_base not in string_types:
continue
series_all = df_sample[col_name]
non_null_series = series_all.drop_nulls()
non_null_count = len(non_null_series)
null_count = total_rows - non_null_count
if non_null_count == 0:
continue
unique_count = int(non_null_series.n_unique())
unique_ratio = unique_count / non_null_count
# --- scoring ---
name_score, matched_kw = _score_name(col_name)
dtype_score = 2.0 # String/categorical bonus
ratio_score = min(unique_ratio * 100.0, 10.0)
penalty = -5.0 if unique_ratio > 0.50 else 0.0
total_score = name_score + dtype_score + ratio_score + penalty
# Null ratio penalty: if more than 50% null, severely reduce score
if null_count / max(1, non_null_count) > 0.5:
total_score *= 0.1
candidate_scores.append({
'column': col_name,
'score': round(total_score, 2),
'unique_ratio': round(unique_ratio, 4),
'matched_keyword': matched_kw,
'non_null_count': non_null_count,
'unique_count': unique_count,
'null_count': null_count,
})
# Sort descending by score, then by unique_count desc (prefer richer columns)
candidate_scores.sort(key=lambda c: (c['score'], c['unique_count']), reverse=True)
recommended = candidate_scores[0]['column'] if candidate_scores else ''
recommended_multi = [c['column'] for c in candidate_scores[:3]]
return {
'dataset_id': dataset_id,
'candidates': candidate_scores,
'recommended': recommended,
'recommended_multi': recommended_multi, # NEW
'total_rows': total_rows,
'scored_columns': len(candidate_scores),
}
analysis/geoip.py
+190
View File
@@ -0,0 +1,190 @@
"""
离线 GeoIP 查询。内嵌 IPv4 区间 → 城市/经纬度映射。
从 ``data/geoip_data.txt`` 加载 IP 段数据，在模块首次导入时构建
排序索引，提供 O(log N) 的 IP 查询能力。
数据约 200KB，覆盖 23 个主要城市 IP 段。
可通过编辑 ``data/geoip_data.txt`` 扩展。
"""
import bisect
import logging
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Data structures
# ---------------------------------------------------------------------------
# [(start_int, end_int, lat, lon, city, country), ...]
# Built at module load time from data/geoip_data.txt
_GEO_DATA: list[tuple[int, int, float, float, str, str]] = []
# Sorted list of start integers (for bisect lookup) + parallel data list
_STARTS: list[int] = []
_RANGES: list[tuple[int, int, float, float, str, str]] = []
# ---------------------------------------------------------------------------
# IP conversion
# ---------------------------------------------------------------------------
def ip_to_int(ip_str: str) -> Optional[int]:
"""Convert an IPv4 dotted-string to its 32-bit integer representation.
Returns ``None`` if *ip_str* is not a valid IPv4 address (including
when any octet is outside 0-255).
Examples
--------
>>> ip_to_int('8.8.8.8')
134744072
>>> ip_to_int('1.0.0.0')
16777216
"""
parts = ip_str.strip().split('.')
if len(parts) != 4:
return None
try:
octets = [int(p) for p in parts]
if any(o < 0 or o > 255 for o in octets):
return None
return (octets[0] << 24) | (octets[1] << 16) | (octets[2] << 8) | octets[3]
except (ValueError, TypeError):
return None
# ---------------------------------------------------------------------------
# Index building
# ---------------------------------------------------------------------------
def _build_index() -> tuple[list[int], list[tuple[int, int, float, float, str, str]]]:
"""Build sorted parallel lists for bisect-based IP lookup.
Processes ``_GEO_DATA``, sorts by start integer, and returns
``(starts, ranges)`` where *starts* is a sorted list of start integers
and *ranges* is the corresponding data list at the same indices.
"""
if not _GEO_DATA:
return [], []
sorted_data = sorted(_GEO_DATA, key=lambda x: x[0])
starts = [entry[0] for entry in sorted_data]
return starts, sorted_data
def _load_data_file() -> None:
"""Load IP range data from ``data/geoip_data.txt`` into ``_GEO_DATA``.
Called once at module import time. Each non-blank, non-comment line
in the file should contain six comma-separated fields::
start_ip,end_ip,lat,lon,city,country
"""
data_path = Path(__file__).parent.parent / 'data' / 'geoip_data.txt'
if not data_path.exists():
logger.warning('[GEOIP] data file not found: %s', data_path)
return
loaded: list[tuple[int, int, float, float, str, str]] = []
skipped = 0
with open(data_path, 'r', encoding='utf-8') as f:
for line_no, raw in enumerate(f, start=1):
stripped = raw.strip()
if not stripped or stripped.startswith('#'):
continue
parts = [p.strip() for p in stripped.split(',')]
if len(parts) != 6:
skipped += 1
continue
start_ip_str, end_ip_str, lat_str, lon_str, city, country = parts
start_int = ip_to_int(start_ip_str)
end_int = ip_to_int(end_ip_str)
if start_int is None or end_int is None:
skipped += 1
continue
try:
lat = float(lat_str)
lon = float(lon_str)
except ValueError:
skipped += 1
continue
loaded.append((start_int, end_int, lat, lon, city, country))
_GEO_DATA.clear()
_GEO_DATA.extend(loaded)
global _STARTS, _RANGES
_STARTS, _RANGES = _build_index()
logger.info('[GEOIP] loaded %d ranges from %s (%d skipped)',
len(loaded), data_path, skipped)
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def lookup(ip_str: str) -> Optional[dict]:
"""查询 IP 对应的地理位置。
Uses binary search (``bisect_right``) on the sorted start-IP list to
locate the range *ip_str* falls within, then checks the end bound.
Parameters
----------
ip_str:
IPv4 address as a dotted string (e.g. ``'8.8.8.8'``).
Returns
-------
dict or None
A dictionary with keys ``lat``, ``lon``, ``city``, ``country``
if the IP is found in the database; ``None`` otherwise.
Examples
--------
>>> result = lookup('8.8.8.8')
>>> result['city']
'Mountain View'
>>> lookup('10.0.0.1') is None
True
"""
target = ip_to_int(ip_str)
if target is None:
return None
if not _STARTS:
return None
idx = bisect.bisect_right(_STARTS, target) - 1
if idx < 0:
return None
start, end, lat, lon, city, country = _RANGES[idx]
if start <= target <= end:
return {
'lat': lat,
'lon': lon,
'city': city,
'country': country,
}
return None
# ---------------------------------------------------------------------------
# Module-level initialisation
# ---------------------------------------------------------------------------
_load_data_file()
analysis/ip_clustering.py
+91
View File
@@ -0,0 +1,91 @@
"""IP address utilities for subnet clustering and conversion.
Provides helper functions to convert IPv4 strings to integers, compute
subnet prefixes, and group IP lists by subnet.
Typical usage::
from analysis.ip_clustering import ip_to_int, get_subnet, cluster_ips
ip_int = ip_to_int('192.168.1.1') # → 3232235777
subnet = get_subnet('192.168.1.5', 24) # → '192.168.1.0/24'
groups = cluster_ips(ip_list, mask=24) # → {'192.168.1.0/24': [...]}
"""
from __future__ import annotations
import ipaddress
from collections import defaultdict
def ip_to_int(ip_str: str) -> int | None:
"""Convert an IPv4 string to its 32-bit integer representation.
Parameters
----------
ip_str:
An IPv4 address string, e.g. ``'192.168.1.1'``.
Returns
-------
int | None
The integer value of the address, or ``None`` if the input
is not a valid IPv4 address.
"""
try:
return int(ipaddress.IPv4Address(ip_str))
except (ipaddress.AddressValueError, ValueError):
return None
def get_subnet(ip_str: str, mask: int = 24) -> str | None:
"""Compute the CIDR subnet string for an IP and mask length.
Parameters
----------
ip_str:
An IPv4 address string, e.g. ``'192.168.1.5'``.
mask:
Subnet mask length (default ``24``).
Returns
-------
str | None
Subnet string like ``'192.168.1.0/24'``, or ``None`` if the IP
is not valid.
"""
try:
ip = ipaddress.IPv4Address(ip_str)
net = ipaddress.IPv4Network(f'{ip}/{mask}', strict=False)
return str(net)
except (ipaddress.AddressValueError, ValueError):
return None
def cluster_ips(
ip_list: list[str],
mask: int = 24,
) -> dict[str, list[str]]:
"""Group a list of IPv4 addresses by their subnet.
Parameters
----------
ip_list:
List of IPv4 address strings.
mask:
Subnet mask length (default ``24``).
Returns
-------
dict[str, list[str]]
Mapping of ``subnet`` → ``[ip, ip, ...]``. Invalid IPs are
grouped under the key ``'__invalid__'``.
"""
groups: dict[str, list[str]] = defaultdict(list)
for ip_str in ip_list:
subnet = get_subnet(ip_str, mask)
if subnet is not None:
groups[subnet].append(ip_str)
else:
groups['__invalid__'].append(ip_str)
# Convert to plain dict for stable serialisation
return dict(groups)
analysis/management/__init__.py
View File
analysis/management/commands/__init__.py
View File
analysis/management/commands/run_pipeline.py
+149
View File
@@ -0,0 +1,149 @@
"""Django management command: ``python manage.py run_pipeline``
Runs the complete TLS analysis pipeline from CLI, no LLM required.
Usage: uv run python manage.py run_pipeline <csv_glob> [--entity-col NAME] [--algo hdbscan|kmeans]
"""
import os, sys, json, asyncio
from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
class Command(BaseCommand):
help = 'Run the full TLS analysis pipeline from CLI (no LLM needed)'
def add_arguments(self, parser):
parser.add_argument('csv_glob', type=str, help='Glob pattern for CSV files')
parser.add_argument('--entity-col', type=str, default=None,
help='Entity column (auto-detect if omitted)')
parser.add_argument('--algo', type=str, default='hdbscan', choices=['hdbscan', 'kmeans'],
help='Clustering algorithm (default: hdbscan)')
parser.add_argument('--output', type=str, default=None,
help='Output directory for exports (default: no export)')
def handle(self, *args, **options):
import django; django.setup()
from analysis.models import AnalysisRun
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
csv_glob = options['csv_glob']
entity_col = options['entity_col']
algo = options['algo']
output = options['output']
store = SessionStore()
store.drop_all()
# ── Step 1: Load CSV ─────────────────────────────────────────────
self.stdout.write(f'[1/5] 加载 CSV: {csv_glob}')
lf, schema, row_count, file_count, memory_mb = load_csv_directory(csv_glob)
ds_id = 'manual_ds'
store.store_dataset(ds_id, lf, schema=schema, metadata={
'row_count': row_count, 'file_count': file_count, 'csv_glob': csv_glob,
})
self.stdout.write(self.style.SUCCESS(f' → {row_count} rows, {file_count} files, {memory_mb:.1f} MB'))
# ── Step 2: Detect entity column ────────────────────────────────
self.stdout.write('[2/5] 检测实体列')
if entity_col:
used_col = entity_col
self.stdout.write(f' → 使用指定列: {used_col}')
else:
result = detect_entity_column(ds_id)
candidates = result.get('candidates', [])
used_col = result.get('recommended')
if not used_col:
self.stderr.write(self.style.ERROR(' ✗ 未检测到实体列，请用 --entity-col 手动指定'))
return
self.stdout.write(f' → 自动检测: {used_col} (共 {len(candidates)} 候选)')
# ── Step 3: Aggregate by entity ──────────────────────────────────
self.stdout.write('[3/5] 实体聚合')
agg_lf, feature_columns = aggregate_by_entity(lf, used_col, schema)
df_agg = agg_lf.collect(streaming=True)
entity_ds_id = 'manual_entity'
store.store_dataset(entity_ds_id, agg_lf, schema=dict(zip(feature_columns, ['']*len(feature_columns))),
metadata={'row_count': len(df_agg), 'entity_column': used_col, 'csv_glob': csv_glob})
self.stdout.write(self.style.SUCCESS(f' → {len(df_agg)} 个实体, {len(feature_columns)} 维特征'))
# ── Step 4: Clustering ──────────────────────────────────────────
self.stdout.write(f'[4/5] 聚类 ({algo})')
from analysis.tool_registry import _handle_run_clustering
# 只选用户面特征列（排除内部 _ 开头列 + 时间列）
user_features = [
c for c in feature_columns
if not c.startswith('_') and c not in ('first_seen', 'last_seen', 'src_ip')
][:10]
self.stdout.write(f' → 聚类特征: {user_features}')
result = asyncio.run(_handle_run_clustering(
dataset_id=entity_ds_id,
cluster_columns=user_features,
algorithm=algo, params={}, random_state=42,
))
if 'error' in result:
self.stderr.write(self.style.ERROR(f' ✗ 聚类失败: {result["error"]}'))
return
cluster_id = result.get('cluster_result_id', '')
n_clusters = result.get('n_clusters', 0)
quality = result.get('quality_metrics', {})
self.stdout.write(self.style.SUCCESS(
f' → {n_clusters} 个簇, 噪声比 {quality.get("noise_ratio", "N/A")}'
))
if quality.get('silhouette_score'):
self.stdout.write(f' → Silhouette: {quality["silhouette_score"]:.4f}')
# ── Step 5: Feature extraction ───────────────────────────────────
self.stdout.write('[5/5] 特征提取')
from analysis.tool_registry import _handle_extract_features
feat_result = asyncio.run(_handle_extract_features(
dataset_id=entity_ds_id, cluster_result_id=cluster_id,
top_k=10, method='zscore', save_to_db=True,
))
if 'error' in feat_result:
self.stderr.write(self.style.ERROR(f' ✗ 特征提取失败: {feat_result["error"]}'))
else:
self.stdout.write(self.style.SUCCESS(f' → {feat_result.get("n_features", 0)} 个特征已保存到数据库'))
# ── Export ───────────────────────────────────────────────────────
if output:
from analysis.tool_registry import _handle_export_results
os.makedirs(output, exist_ok=True)
asyncio.run(_handle_export_results(result_id=entity_ds_id, output_path=output, format='csv', overwrite=True))
self.stdout.write(self.style.SUCCESS(f' → 导出到 {output}'))
# ── Summary ──────────────────────────────────────────────────────
run_record = AnalysisRun.objects.create(
csv_glob=csv_glob, status='completed',
total_flows=row_count, entity_count=len(df_agg),
cluster_count=n_clusters, entity_column=used_col,
)
# ── Step 6: PCA-2D embedding (after run_record exists) ─────────────
try:
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import polars as pl, numpy as np
num_cols = [c for c in df_agg.columns if df_agg[c].dtype in (
pl.Int8,pl.Int16,pl.Int32,pl.Int64,pl.UInt8,pl.UInt16,pl.UInt32,pl.UInt64,pl.Float32,pl.Float64)]
if len(num_cols) >= 2 and len(df_agg) > 2:
mat = np.nan_to_num(StandardScaler().fit_transform(df_agg.select(num_cols).to_numpy()), nan=0.0)
coords = PCA(n_components=2, random_state=42).fit_transform(mat)
non_num = [c for c in df_agg.columns if c not in num_cols and not c.startswith('_')]
ecol = non_num[0] if non_num else df_agg.columns[0]
from analysis.models import EntityProfile as EP, ClusterResult as CR
for i, row in enumerate(df_agg.iter_rows(named=True)):
if i >= len(coords): break
ev = str(row.get(ecol, ''))
if ev:
EP.objects.get_or_create(entity_value=ev, defaults={
'run_id': run_record.id,
'cluster_label': 0,
'embedding_x': float(coords[i,0]),
'embedding_y': float(coords[i,1])})
except Exception as e:
self.stderr.write(f' ⚠ PCA降维跳过: {e}')
self.stdout.write(self.style.SUCCESS(
f'\n*** 分析完成! Run ID: #{run_record.id}'
f' 查看 Django: http://127.0.0.1:8000/runs/{run_record.id}/'
))
analysis/management/commands/start_mcp.py
+34
View File
@@ -0,0 +1,34 @@
"""Django management command: ``python manage.py start_mcp``
Starts the embedded TLS Analyzer MCP server on stdio transport.
IMPORTANT: All log output goes to stderr (stdout is the MCP protocol transport).
"""
import os
import sys
from django.core.management.base import BaseCommand, CommandError
class Command(BaseCommand):
help = 'Start the embedded TLS Analyzer MCP server on stdio transport'
def handle(self, *args, **options):
# Ensure Django is fully initialised before importing server code
import django
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
django.setup()
# CRITICAL: MCP uses stdio for protocol transport.
# All log messages MUST go to stderr, NOT stdout.
print('Starting TLS Analyzer MCP server...', file=sys.stderr)
print('Transport: stdio', file=sys.stderr)
sys.stderr.flush()
try:
from analysis.mcp_server import TLSAnalyzerMCPServer
server = TLSAnalyzerMCPServer()
server.run_sync()
except KeyboardInterrupt:
print('\nMCP server stopped.', file=sys.stderr)
except Exception as e:
raise CommandError(f'MCP server failed: {e}') from e
analysis/mcp_server.py
+84
View File
@@ -0,0 +1,84 @@
"""MCP Server embedded in Django.
Uses asyncio stdio transport. Registered via the Django management command
``python manage.py start_mcp``.
Django ORM initialisation happens at the management-command level so that
the server thread has access to models when needed (e.g. during feature
extraction).
"""
import asyncio
import json
import os
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import TextContent
# Ensure Django is configured when this module is imported from the
# management command. The management command calls django.setup()
# before constructing the server.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
from .session_store import SessionStore
from .tool_registry import get_tools_meta, handle_call
class TLSAnalyzerMCPServer:
"""Embedded MCP server for the TLS Analyzer.
Exposes 11 tools over stdio transport. The :class:`SessionStore`
singleton holds datasets and analysis results in memory for the
lifetime of the server process.
"""
def __init__(self):
self.server = Server("tls-analyzer")
self.store = SessionStore()
self._setup_tools()
# ── tool registration ───────────────────────────────────────────────
def _setup_tools(self) -> None:
"""Register all tools with the MCP server via the SDK decorator API.
Uses :func:`get_tools_meta` for tool metadata (name, description,
inputSchema) and :func:`handle_call` for request routing.
"""
tools_meta = get_tools_meta()
_tools_map = {t.name: t for t in tools_meta}
@self.server.list_tools()
async def list_tools():
return tools_meta
@self.server.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
try:
result = await handle_call(name, arguments)
return [TextContent(type="text", text=json.dumps(result, default=str))]
except Exception as exc:
error_body = json.dumps({
'error': str(exc),
'tool': name,
'truncated': False,
}, default=str)
return [TextContent(type="text", text=error_body)]
# ── lifecycle ───────────────────────────────────────────────────────
async def run(self) -> None:
"""Start the MCP server with stdio transport.
Blocks until the transport is closed.
"""
async with stdio_server() as (read, write):
await self.server.run(
read,
write,
self.server.create_initialization_options(),
)
def run_sync(self) -> None:
"""Synchronous entry point for the management command."""
asyncio.run(self.run())
analysis/migrations/0001_initial.py
+84
View File
@@ -0,0 +1,84 @@
# Generated by Django 4.2.30 on 2026-07-13 14:36
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='AnalysisRun',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('csv_glob', models.CharField(help_text='Glob pattern for CSV files', max_length=1024)),
('config', models.JSONField(blank=True, default=dict, help_text='YAML config snapshot')),
('status', models.CharField(choices=[('pending', 'Pending'), ('loading', 'Loading Data'), ('profiling', 'Profiling'), ('aggregating', 'Building Entity Profiles'), ('clustering', 'Clustering'), ('extracting', 'Extracting Features'), ('completed', 'Completed'), ('failed', 'Failed')], default='pending', max_length=32)),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('error_message', models.TextField(blank=True, default='')),
('total_flows', models.IntegerField(blank=True, null=True)),
('entity_count', models.IntegerField(blank=True, null=True)),
('cluster_count', models.IntegerField(blank=True, null=True)),
('entity_column', models.CharField(blank=True, default='', help_text='Auto-detected entity column name', max_length=256)),
],
options={
'ordering': ['-created_at'],
},
),
migrations.CreateModel(
name='ClusterResult',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('cluster_label', models.IntegerField(help_text='Cluster label (-1 for noise)')),
('size', models.IntegerField(help_text='Number of entities in this cluster')),
('proportion', models.FloatField(blank=True, help_text='Proportion of total entities', null=True)),
('noise_ratio', models.FloatField(blank=True, help_text='Noise ratio of the clustering', null=True)),
('silhouette_score', models.FloatField(blank=True, help_text='Silhouette score (sampled if large)', null=True)),
('run', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='clusters', to='tianxuan_analysis.analysisrun')),
],
options={
'unique_together': {('run', 'cluster_label')},
},
),
migrations.CreateModel(
name='EntityProfile',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('entity_value', models.CharField(help_text='Entity identifier (IP, domain, etc.)', max_length=512)),
('cluster_label', models.IntegerField(blank=True, help_text='Assigned cluster (-1=noise)', null=True)),
('feature_json', models.JSONField(blank=True, default=dict, help_text='Aggregated entity features as dict')),
('embedding_x', models.FloatField(blank=True, help_text='PCA-2D X coordinate', null=True)),
('embedding_y', models.FloatField(blank=True, help_text='PCA-2D Y coordinate', null=True)),
('cluster', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='entities', to='tianxuan_analysis.clusterresult')),
('run', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='entities', to='tianxuan_analysis.analysisrun')),
],
options={
'unique_together': {('run', 'entity_value')},
},
),
migrations.CreateModel(
name='ClusterFeature',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('feature_name', models.CharField(max_length=256)),
('mean', models.FloatField(blank=True, null=True)),
('std', models.FloatField(blank=True, null=True)),
('median', models.FloatField(blank=True, null=True)),
('p25', models.FloatField(blank=True, null=True)),
('p75', models.FloatField(blank=True, null=True)),
('missing_rate', models.FloatField(blank=True, null=True)),
('distinguishing_score', models.FloatField(blank=True, help_text='Z-score or ANOVA F-score', null=True)),
('distinguishing_method', models.CharField(default='zscore', max_length=16)),
('cluster', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='features', to='tianxuan_analysis.clusterresult')),
],
options={
'unique_together': {('cluster', 'feature_name')},
},
),
]
analysis/migrations/0002_analysisrun_progress_msg_analysisrun_progress_pct_and_more.py
+28
View File
@@ -0,0 +1,28 @@
# Generated by Django 4.2.30 on 2026-07-16 05:11
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('tianxuan_analysis', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='analysisrun',
name='progress_msg',
field=models.CharField(blank=True, default='', max_length=256),
),
migrations.AddField(
model_name='analysisrun',
name='progress_pct',
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name='analysisrun',
name='run_log',
field=models.TextField(blank=True, default=''),
),
]
analysis/migrations/__init__.py
View File
analysis/models.py
+94
View File
@@ -0,0 +1,94 @@
from django.db import models
class AnalysisRun(models.Model):
"""Tracks a single analysis pipeline execution."""
STATUS_CHOICES = [
('pending', 'Pending'),
('loading', 'Loading Data'),
('profiling', 'Profiling'),
('aggregating', 'Building Entity Profiles'),
('clustering', 'Clustering'),
('extracting', 'Extracting Features'),
('completed', 'Completed'),
('failed', 'Failed'),
]
csv_glob = models.CharField(max_length=1024, help_text="Glob pattern for CSV files")
config = models.JSONField(default=dict, blank=True, help_text="YAML config snapshot")
status = models.CharField(max_length=32, choices=STATUS_CHOICES, default='pending')
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
error_message = models.TextField(blank=True, default='')
progress_pct = models.IntegerField(default=0)
progress_msg = models.CharField(max_length=256, default='', blank=True)
run_log = models.TextField(blank=True, default='')
# Summary stats populated after completion
total_flows = models.IntegerField(null=True, blank=True)
entity_count = models.IntegerField(null=True, blank=True)
cluster_count = models.IntegerField(null=True, blank=True)
entity_column = models.CharField(max_length=256, blank=True, default='',
help_text="Auto-detected entity column name")
class Meta:
ordering = ['-created_at']
def __str__(self):
return f"Run #{self.id} ({self.created_at:%Y-%m-%d %H:%M}) - {self.status}"
class ClusterResult(models.Model):
"""Stores clustering results for a single analysis run."""
run = models.ForeignKey(AnalysisRun, on_delete=models.CASCADE, related_name='clusters')
cluster_label = models.IntegerField(help_text="Cluster label (-1 for noise)")
size = models.IntegerField(help_text="Number of entities in this cluster")
proportion = models.FloatField(null=True, blank=True, help_text="Proportion of total entities")
noise_ratio = models.FloatField(null=True, blank=True, help_text="Noise ratio of the clustering")
silhouette_score = models.FloatField(null=True, blank=True, help_text="Silhouette score (sampled if large)")
class Meta:
unique_together = ['run', 'cluster_label']
def __str__(self):
return f"Cluster #{self.cluster_label} (n={self.size})"
class EntityProfile(models.Model):
"""One row per entity (user/host) per analysis run."""
run = models.ForeignKey(AnalysisRun, on_delete=models.CASCADE, related_name='entities')
entity_value = models.CharField(max_length=512, help_text="Entity identifier (IP, domain, etc.)")
cluster_label = models.IntegerField(null=True, blank=True, help_text="Assigned cluster (-1=noise)")
cluster = models.ForeignKey(ClusterResult, on_delete=models.SET_NULL, null=True, blank=True,
related_name='entities')
feature_json = models.JSONField(default=dict, blank=True,
help_text="Aggregated entity features as dict")
embedding_x = models.FloatField(null=True, blank=True, help_text="PCA-2D X coordinate")
embedding_y = models.FloatField(null=True, blank=True, help_text="PCA-2D Y coordinate")
class Meta:
unique_together = ['run', 'entity_value']
def __str__(self):
return f"Entity: {self.entity_value} (Cluster {self.cluster_label})"
class ClusterFeature(models.Model):
"""Per-cluster statistical features and distinguishing scores."""
cluster = models.ForeignKey(ClusterResult, on_delete=models.CASCADE, related_name='features')
feature_name = models.CharField(max_length=256)
mean = models.FloatField(null=True, blank=True)
std = models.FloatField(null=True, blank=True)
median = models.FloatField(null=True, blank=True)
p25 = models.FloatField(null=True, blank=True)
p75 = models.FloatField(null=True, blank=True)
missing_rate = models.FloatField(null=True, blank=True)
distinguishing_score = models.FloatField(null=True, blank=True,
help_text="Z-score or ANOVA F-score")
distinguishing_method = models.CharField(max_length=16, default='zscore')
class Meta:
unique_together = ['cluster', 'feature_name']
def __str__(self):
return f"{self.feature_name} (score={self.distinguishing_score:.3f})"
analysis/session_store.py
+251
View File
@@ -0,0 +1,251 @@
"""Thread-safe in-memory session store for datasets and results.
Singleton pattern with RLock guard. Stores:
- Datasets: dataset_id -> (LazyFrame, schema, metadata)
- Cluster results: result_id -> (labels, n_clusters, params, parent_dataset_id)
- Feature results: result_id -> (features, parent_cluster_result_id)
"""
import threading
import gc
import uuid
from typing import Any, Optional
import polars as pl
# Approximate per-element byte sizes for memory estimation
_DTYPE_BYTE_SIZES = {
# Polars dtype strings
'Int8': 1, 'Int16': 2, 'Int32': 4, 'Int64': 8,
'UInt8': 1, 'UInt16': 2, 'UInt32': 4, 'UInt64': 8,
'Float32': 4, 'Float64': 8,
'Boolean': 1, 'Bool': 1,
'Utf8': 50, 'String': 50, 'Categorical': 8,
'Date': 4, 'Datetime': 8, 'Time': 8, 'Duration': 8,
'List': 16, 'Object': 16, 'Null': 0,
# Polars __str__ representations
'i32': 4, 'i64': 8, 'u32': 4, 'u64': 8,
'f32': 4, 'f64': 8, 'str': 50, 'bool': 1,
}
def _estimate_dtype_bytes(dtype: Any) -> int:
"""Guess per-element byte size from a Polars dtype."""
key = str(dtype)
for prefix, size in _DTYPE_BYTE_SIZES.items():
if key.startswith(prefix) or key == prefix:
return size
return 8 # fallback
def _generate_id(prefix: str = 'ds') -> str:
"""Generate a short unique identifier."""
return f"{prefix}_{uuid.uuid4().hex[:12]}"
class SessionStore:
"""Singleton in-memory store for datasets and analysis results.
Thread-safe via RLock. Datasets store Polars LazyFrames (query plans,
not materialised data), so clone/shallow-copy is effectively free.
"""
_instance: Optional['SessionStore'] = None
_lock: threading.RLock = threading.RLock()
def __new__(cls) -> 'SessionStore':
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._stores: dict[str, dict[str, Any]] = {}
cls._instance._datalock = threading.RLock()
return cls._instance
# ── dataset operations ──────────────────────────────────────────────
def store_dataset(
self,
dataset_id: str,
lazyframe: pl.LazyFrame,
schema: Optional[dict] = None,
metadata: Optional[dict] = None,
parent_id: Optional[str] = None,
) -> None:
"""Store a LazyFrame under *dataset_id*.
Args:
dataset_id: Unique key for the dataset.
lazyframe: Polars LazyFrame (query plan, not materialised).
schema: Column schema dict {name: dtype_string}.
metadata: Optional dict (row_count, file_count, …).
parent_id: If this is a derived dataset (filter/clone/…), the
source dataset_id.
"""
with self._datalock:
self._stores[dataset_id] = {
'type': 'dataset',
'lazyframe': lazyframe,
'schema': schema or {},
'metadata': metadata or {},
'parent_id': parent_id,
}
def get_dataset(self, dataset_id: str) -> Optional[dict]:
"""Retrieve a stored dataset entry (or *None*)."""
with self._datalock:
entry = self._stores.get(dataset_id)
if entry and entry.get('type') == 'dataset':
return entry
return None
def drop_dataset(self, dataset_id: str) -> bool:
"""Remove a dataset from the store and force garbage collection.
Returns True if the dataset existed.
"""
with self._datalock:
existed = dataset_id in self._stores and self._stores[dataset_id].get('type') == 'dataset'
if existed:
del self._stores[dataset_id]
if existed:
gc.collect()
return existed
def clone_dataset(self, dataset_id: str) -> Optional[str]:
"""Shallow clone — LazyFrame is a query plan so no data is copied.
Returns the new dataset_id, or *None* if source does not exist.
"""
with self._datalock:
entry = self.get_dataset(dataset_id)
if entry is None:
return None
new_id = _generate_id('clone')
self._stores[new_id] = {
'type': 'dataset',
'lazyframe': entry['lazyframe'], # same plan pointer
'schema': dict(entry['schema']),
'metadata': dict(entry.get('metadata', {})),
'parent_id': dataset_id,
}
return new_id
def list_datasets(self) -> list[dict]:
"""Return summary dict for every stored dataset."""
with self._datalock:
results = []
for ds_id, entry in self._stores.items():
if entry.get('type') != 'dataset':
continue
meta = entry.get('metadata', {})
results.append({
'dataset_id': ds_id,
'parent_id': entry.get('parent_id'),
'row_count': meta.get('row_count'),
'file_count': meta.get('file_count'),
'columns': list(entry.get('schema', {}).keys()),
'column_count': len(entry.get('schema', {})),
})
return results
def memory_estimate_mb(self, dataset_id: str) -> float:
"""Rough memory estimate based on schema × row_count."""
with self._datalock:
entry = self.get_dataset(dataset_id)
if entry is None:
return 0.0
schema = entry.get('schema', {})
row_count = entry.get('metadata', {}).get('row_count', 0)
if not row_count or not schema:
return 0.0
bytes_per_row = sum(_estimate_dtype_bytes(dtype) for dtype in schema.values())
return (bytes_per_row * row_count) / (1024 * 1024)
# ── cluster-result operations ───────────────────────────────────────
def store_cluster_result(
self,
result_id: str,
labels: list[int],
n_clusters: int,
params: dict[str, Any],
parent_dataset_id: str,
) -> None:
"""Store a clustering result."""
with self._datalock:
self._stores[result_id] = {
'type': 'cluster_result',
'labels': labels,
'n_clusters': n_clusters,
'params': params,
'parent_dataset_id': parent_dataset_id,
}
def get_cluster_result(self, result_id: str) -> Optional[dict]:
"""Retrieve a stored cluster result."""
with self._datalock:
entry = self._stores.get(result_id)
if entry and entry.get('type') == 'cluster_result':
return entry
return None
def drop_cluster_result(self, result_id: str) -> bool:
"""Remove a cluster result."""
with self._datalock:
existed = result_id in self._stores and self._stores[result_id].get('type') == 'cluster_result'
if existed:
del self._stores[result_id]
return existed
# ── feature-result operations ───────────────────────────────────────
def store_feature_result(
self,
result_id: str,
features: list[dict[str, Any]],
parent_cluster_result_id: str,
) -> None:
"""Store extracted features per cluster."""
with self._datalock:
self._stores[result_id] = {
'type': 'feature_result',
'features': features,
'parent_cluster_result_id': parent_cluster_result_id,
}
def get_feature_result(self, result_id: str) -> Optional[dict]:
"""Retrieve a stored feature result."""
with self._datalock:
entry = self._stores.get(result_id)
if entry and entry.get('type') == 'feature_result':
return entry
return None
def drop_feature_result(self, result_id: str) -> bool:
"""Remove a feature result."""
with self._datalock:
existed = result_id in self._stores and self._stores[result_id].get('type') == 'feature_result'
if existed:
del self._stores[result_id]
return existed
# ── general ─────────────────────────────────────────────────────────
def drop_all(self) -> int:
"""Drop every entry and run gc. Returns count of removed entries."""
with self._datalock:
count = len(self._stores)
self._stores.clear()
if count:
gc.collect()
return count
def __repr__(self) -> str:
with self._datalock:
types = {}
for entry in self._stores.values():
t = entry.get('type', 'unknown')
types[t] = types.get(t, 0) + 1
parts = [f"{k}={v}" for k, v in sorted(types.items())]
return f"<SessionStore {'; '.join(parts)}>"
analysis/templatetags/__init__.py
View File
analysis/templatetags/query_helpers.py
+12
View File
@@ -0,0 +1,12 @@
from django import template
register = template.Library()
@register.filter
def getlist(value, key):
"""Get all values for a key from QueryDict and join with ', '."""
if hasattr(value, 'getlist'):
vals = value.getlist(key)
return ', '.join(vals) if vals else ''
return ''
analysis/tool_registry.py
+1707
View File
File diff suppressed because it is too large Load Diff
analysis/type_classifier.py
+350
View File
@@ -0,0 +1,350 @@
"""Data type classifier for column-level type inference.
Provides :class:`DataType` enumeration and functions to classify Polars
columns based on name heuristics and value sampling.
Typical usage::
from analysis.type_classifier import classify_column, classify_schema, DataType
# Single column
dtype = classify_column('src_ip', series, config_type='ipv4')
# Full schema
type_map = classify_schema(lazyframe, config_overrides={'src_ip': 'ipv4'})
"""
from __future__ import annotations
import re
from enum import Enum
import polars as pl
# ---------------------------------------------------------------------------
# DataType enumeration
# ---------------------------------------------------------------------------
class DataType(Enum):
"""Enumerated column types recognised by the classifier engine."""
INT = 1
FLOAT = 2
ENUM = 3
HEX = 4
URL = 5
IPv4 = 6
LAT_LON = 7
TIMESTAMP = 8
BOOL_ENUM = 9
STRING = 10
# ---------------------------------------------------------------------------
# Detection patterns (raw strings for regex)
# ---------------------------------------------------------------------------
HEX_PATTERN = r'^[0-9a-f]{2}(?:\s[0-9a-f]{2})*$'
"""Matches ``"0e a6 3f"``, ``"ab cd ef"`` etc. (space-separated hex pairs)."""
URL_PATTERN = r'^https?://'
"""Matches URLs starting with ``http://`` or ``https://``."""
IPv4_PATTERN = r'^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'
"""Matches ``"192.168.1.1"`` style IPv4 addresses."""
MAC_PATTERN = r'^([0-9A-Fa-f]{2}[:.-]){5}[0-9A-Fa-f]{2}$'
"""Matches MAC addresses with ``:``, ``-``, or ``.`` separators."""
BOOL_ENUM_VALUES = (
'', ' ', '+', '-', '0', '1',
'true', 'false', 'yes', 'no', 't', 'f',
)
"""String values considered boolean / binary indicators."""
LAT_LON_NAMES = ('lat', 'latitude', 'lon', 'lng', 'longitude')
# Name-to-type heuristics (checked after config override, before values)
_NAME_TYPE_MAP: list[tuple[re.Pattern, DataType]] = [
# User column names - EXACT match only (no generic fuzzy patterns)
# IPv4 columns
(re.compile(r'^:ips$|^:ipd$|^server.ip$|^client.ip$', re.I), DataType.IPv4),
# ENUM columns
(re.compile(r'^:prs$|^:prd$', re.I), DataType.ENUM),
(re.compile(r'^(?:scnt|dcnt|cnrs|isrs|1ipp|4dbn)$', re.I), DataType.ENUM),
(re.compile(r'^0ver$', re.I), DataType.ENUM),
# FLOAT columns
(re.compile(r'^4dur$|^8ses$|^2tmo$|^0rnt$', re.I), DataType.FLOAT),
# INT columns
(re.compile(r'^4ksz$|^8ack$|^8ppk$|^row$', re.I), DataType.INT),
# HEX columns
(re.compile(r'^0cph$|^0crv$', re.I), DataType.HEX),
# TIMESTAMP columns
(re.compile(r'^8dbd$|^time$|^timestamp$', re.I), DataType.TIMESTAMP),
# Dot-notation suffixes for lat/lon (e.g. :ips.latd, :ipd.lond)
(re.compile(r'\.latd$', re.I), DataType.LAT_LON),
(re.compile(r'\.lond$', re.I), DataType.LAT_LON),
# Dot-notation STRING suffixes (e.g. :ips.ispn, :ipd.city)
(re.compile(r'\.(ispn|orgn|city)$', re.I), DataType.STRING),
# Standalone string columns - exact match
(re.compile(r'^(?:cnam|snam|tabl|name|0rnd|source.node|ecdhe.named.curve)$', re.I), DataType.STRING),
]
# Mapping from config.yaml type strings to DataType
_CONFIG_TYPE_MAP: dict[str, DataType] = {
'int': DataType.INT,
'float': DataType.FLOAT,
'enum': DataType.ENUM,
'hex': DataType.HEX,
'url': DataType.URL,
'ipv4': DataType.IPv4,
'lat_lon': DataType.LAT_LON,
'timestamp': DataType.TIMESTAMP,
'bool_enum': DataType.BOOL_ENUM,
'string': DataType.STRING,
}
# ---------------------------------------------------------------------------
# Column classification (single column)
# ---------------------------------------------------------------------------
def _name_to_type(name: str) -> DataType | None:
"""Match column name against known heuristics.
Returns a :class:`DataType` if the name matches strongly, otherwise
``None`` (fall through to value-based detection).
"""
for pattern, dtype in _NAME_TYPE_MAP:
if pattern.search(name):
return dtype
return None
def _is_port_value(v: str) -> bool:
"""Check if *v* is an integer string in the registered port range (1-65535)."""
try:
n = int(v)
return 1 <= n <= 65535
except ValueError:
return False
def _can_float(v: str) -> bool:
"""Return ``True`` when *v* can be parsed as a Python float."""
try:
float(v)
return True
except ValueError:
return False
def _values_to_type(series: pl.Series) -> DataType:
"""Infer column type by sampling non-null values.
Detection order: IPv4 → MAC guard → HEX → URL → PORT → BOOL_ENUM
→ LAT_LON → ENUM → FLOAT → STRING
"""
dtype = series.dtype
# Boolean dtype ──────────────────────────────────────────────────────
if dtype == pl.Boolean:
return DataType.BOOL_ENUM
# Numeric dtypes ─────────────────────────────────────────────────────
if dtype in (pl.Float32, pl.Float64):
# Check LAT_LON possibility before returning FLOAT
non_null = series.drop_nulls()
if len(non_null) > 0:
sample = non_null.head(100)
sample_strs = [str(v).strip().lower() for v in sample]
non_blank = [v for v in sample_strs if v not in ('', ' ', '+', '-')]
if non_blank:
try:
floats = [float(v) for v in non_blank]
max_abs = max(abs(f) for f in floats)
if (
all(-180 <= f <= 180 for f in floats)
and any('.' in v for v in non_blank)
and max_abs > 10
):
return DataType.LAT_LON
except ValueError:
pass
return DataType.FLOAT
if dtype in (pl.Int8, pl.Int16, pl.Int32, pl.Int64,
pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64):
return DataType.INT
# String / categorical types ─────────────────────────────────────────
non_null = series.drop_nulls()
if len(non_null) == 0:
return DataType.STRING
sample = non_null.head(100)
# Convert all to string representations for pattern matching
sample_strs = [str(v).strip().lower() for v in sample]
# ── 1. IPv4 detection ──────────────────────────────────────────────
if all(re.match(IPv4_PATTERN, v) for v in sample_strs):
if all(_valid_ip_octets(v) for v in sample_strs):
return DataType.IPv4
# ── 2. MAC guard (skip HEX if values look like MAC addresses) ──────
is_mac = all(re.match(MAC_PATTERN, v) for v in sample_strs)
# ── 3. HEX detection ───────────────────────────────────────────────
if not is_mac and all(re.match(HEX_PATTERN, v) for v in sample_strs):
return DataType.HEX
# ── 4. URL detection ───────────────────────────────────────────────
if all(re.match(URL_PATTERN, v) for v in sample_strs):
return DataType.URL
# ── 5. PORT detection → ENUM ───────────────────────────────────────
if all(_is_port_value(v) for v in sample_strs):
return DataType.ENUM
# ── 6. BOOL_ENUM detection ─────────────────────────────────────────
if all(v in BOOL_ENUM_VALUES for v in sample_strs):
return DataType.BOOL_ENUM
# ── 7. LAT_LON detection (value-based) ──────────────────────────────
non_blank = [v for v in sample_strs if v not in ('', ' ', '+', '-')]
if non_blank:
try:
floats = [float(v) for v in non_blank]
max_abs = max(abs(f) for f in floats)
if (
all(-180 <= f <= 180 for f in floats)
and any('.' in v for v in non_blank)
and max_abs > 10
):
return DataType.LAT_LON
except ValueError:
pass
# ── 8. ENUM detection (few unique values) ──────────────────────────
unique_count = non_null.n_unique()
if unique_count < 20:
return DataType.ENUM
# ── 9. FLOAT detection (>80 % can float) ───────────────────────────
float_count = sum(1 for v in sample_strs if _can_float(v))
if float_count / len(sample_strs) > 0.8:
# Guard: values with leading "+"/"-" (e.g., "+5", "-abc") that
# happen to parse as float should not force a FLOAT classification
# when non-float values exist in the column. This catches string
# columns where a "+" prefix is used as a text marker (phone codes,
# IDs, error codes) rather than a numeric sign.
if any(v.startswith('+') or v.startswith('-') for v in sample_strs):
if float_count < len(sample_strs):
return DataType.STRING
return DataType.FLOAT
return DataType.STRING
def _valid_ip_octets(ip_str: str) -> bool:
"""Check that each octet in an IPv4 string is 0-255."""
parts = ip_str.split('.')
if len(parts) != 4:
return False
for p in parts:
try:
n = int(p)
if n < 0 or n > 255:
return False
except ValueError:
return False
return True
def classify_column(
name: str,
series: pl.Series,
config_type: str | None = None,
) -> DataType:
"""Classify a single column based on its *name* and *series* values.
Priority
--------
1. **config_type** — user override from ``config.yaml``.
If provided (e.g. ``'ipv4'``), returns the corresponding
:class:`DataType` immediately.
2. **Value-based** pattern matching — samples up to 100 non-null
values and tests against IPv4 → HEX → URL → PORT → BOOL_ENUM
→ LAT_LON → ENUM → FLOAT in order.
3. **Name-based** heuristics — only consulted when value-based
detection returned ``STRING`` (i.e. no value pattern matched).
Parameters
----------
name:
Column name (used for name-based heuristics).
series:
Polars :class:`Series` containing the column's data.
config_type:
Optional user override (case-insensitive, matched to
:data:`_CONFIG_TYPE_MAP`).
Returns
-------
DataType
"""
# 1. Config override (highest priority)
if config_type is not None:
mapped = _CONFIG_TYPE_MAP.get(config_type.lower().strip())
if mapped is not None:
return mapped
# 2. Value-based sampling
result = _values_to_type(series)
# 3. Name-based heuristics (only when values returned STRING)
if result == DataType.STRING:
name_match = _name_to_type(name)
if name_match is not None:
return name_match
return result
# ---------------------------------------------------------------------------
# Schema classification (all columns)
# ---------------------------------------------------------------------------
def classify_schema(
lf: pl.LazyFrame,
config_overrides: dict[str, str] | None = None,
) -> dict[str, DataType]:
"""Classify all columns in a :class:`LazyFrame`.
Parameters
----------
lf:
A Polars :class:`LazyFrame` whose columns to classify.
config_overrides:
Optional mapping of ``{col_name: type_string}`` from a
``config.yaml`` ``columns:`` section. These take highest
priority (see :func:`classify_column`).
Returns
-------
dict[str, DataType]
Mapping from column name to inferred :class:`DataType`.
"""
config_overrides = config_overrides or {}
df_sample = lf.collect(streaming=True)
result: dict[str, DataType] = {}
for col_name in df_sample.columns:
config_type = config_overrides.get(col_name)
result[col_name] = classify_column(
name=col_name,
series=df_sample[col_name],
config_type=config_type,
)
return result
analysis/urls.py
+24
View File
@@ -0,0 +1,24 @@
from django.urls import path
from . import views
app_name = 'analysis'
urlpatterns = [
path('', views.dashboard, name='dashboard'),
path('upload/', views.upload_page, name='upload'),
path('upload/csv/', views.upload_csv, name='upload_csv'),
path('analyze/manual/', views.manual_page, name='manual'),
path('analyze/run/', views.run_analysis, name='run_analysis'),
path('analyze/auto/', views.auto_page, name='auto'),
path('analyze/llm/', views.run_llm_analysis_view, name='run_llm_analysis'),
path('runs/', views.run_list, name='run_list'),
path('runs/<int:run_id>/', views.run_detail, name='run_detail'),
path('runs/<int:run_id>/status/', views.run_status_api, name='run_status'),
path('clusters/<int:run_id>/', views.cluster_overview, name='cluster_overview'),
path('clusters/<int:run_id>/<int:cluster_label>/', views.cluster_detail, name='cluster_detail'),
path('entities/<int:entity_id>/', views.entity_profile, name='entity_profile'),
path('config/', views.config_view, name='config'),
path('api/llm/test/', views.llm_test, name='llm_test'),
path('runs/<int:run_id>/delete/', views.delete_upload, name='delete_upload'),
path('logs/', views.log_viewer, name='log_viewer'),
path('globe/', views.globe_view, name='globe'),
]
analysis/value_normalizer.py
+308
View File
@@ -0,0 +1,308 @@
"""TLS value normalization — maps hex wire format, IANA names, and display
names to unified underscore-connected enum names.
Usage::
from analysis.value_normalizer import normalize_lf
lf = pl.LazyFrame({...})
type_map = classify_schema(lf) # {col_name: DataType}
lf = normalize_lf(lf, type_map)
# -> LazyFrame with added {col}_norm columns
"""
from __future__ import annotations
import polars as pl
# ═════════════════════════════════════════════════════════════════════════════
# TLS Version map — hex wire format & display names
# ═════════════════════════════════════════════════════════════════════════════
TLS_VERSION_MAP: dict[str, str] = {
# hex wire format (space-separated hex pairs)
'03 04': 'tls_1_3',
'03 03': 'tls_1_2',
'03 02': 'tls_1_1',
'03 01': 'tls_1_0',
'02 00': 'ssl_2_0',
'03 00': 'ssl_3_0',
# display formats
'tlsv1.3': 'tls_1_3',
'tls 1.3': 'tls_1_3',
'1.3': 'tls_1_3',
'tlsv1.2': 'tls_1_2',
'tls 1.2': 'tls_1_2',
'1.2': 'tls_1_2',
'tlsv1.1': 'tls_1_1',
'tls 1.1': 'tls_1_1',
'1.1': 'tls_1_1',
'tlsv1.0': 'tls_1_0',
'tls 1.0': 'tls_1_0',
'1.0': 'tls_1_0',
'tlsv1': 'tls_1_0',
'tls 1': 'tls_1_0',
'1': 'tls_1_0',
'sslv2': 'ssl_2_0',
'ssl 2': 'ssl_2_0',
'2': 'ssl_2_0',
'sslv3': 'ssl_3_0',
'ssl 3': 'ssl_3_0',
'3': 'ssl_3_0',
}
"""TLS version mapping.
Keys are hex wire format (``'03 04'``) or display names (``'TLSv1.3'``,
``'tls 1.3'``, etc.). Matching is case-insensitive. Unmatched values
produce ``'unknown_version'``.
"""
# ═════════════════════════════════════════════════════════════════════════════
# Cipher Suite maps
# ═════════════════════════════════════════════════════════════════════════════
CIPHER_SUITE_MAP: dict[str, str] = {
# hex wire format → canonical underscore name
'13 01': 'tls_aes_128_gcm_sha256',
'13 02': 'tls_aes_256_gcm_sha384',
'13 03': 'tls_chacha20_poly1305_sha256',
'c0 2b': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
'c0 2c': 'tls_ecdhe_ecdsa_aes_256_gcm_sha384',
'c0 2d': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
'c0 2f': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
'c0 30': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
'cc a9': 'tls_ecdhe_ecdsa_chacha20_poly1305_sha256',
'cc ac': 'tls_ecdhe_rsa_chacha20_poly1305_sha256',
'00 9c': 'tls_rsa_aes_128_gcm_sha256',
'00 9d': 'tls_rsa_aes_256_gcm_sha384',
'00 3c': 'tls_rsa_aes_128_cbc_sha256',
'00 3d': 'tls_rsa_aes_256_cbc_sha256',
'c0 09': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
'c0 13': 'tls_ecdhe_rsa_aes_128_cbc_sha256',
'00 35': 'tls_rsa_aes_128_cbc_sha',
'00 2f': 'tls_rsa_aes_128_cbc_sha',
'00 05': 'tls_rsa_rc4_128_sha',
'00 ff': 'tls_rsa_null_md5',
}
"""Cipher suite hex → canonical name. Unmatched hex produces
``'unknown_cipher_0x{hex}'``."""
_CIPHER_DISPLAY_NORMALIZED: dict[str, str] = {
# Normalized key (lowercase, separators removed) → canonical name
'tlsaes128gcmsha256': 'tls_aes_128_gcm_sha256',
'tlsaes256gcmsha384': 'tls_aes_256_gcm_sha384',
'aes128gcm': 'tls_aes_128_gcm_sha256',
'aes256gcm': 'tls_aes_256_gcm_sha384',
'chacha20poly1305': 'tls_chacha20_poly1305_sha256',
'ecdheecdsaaes128gcm': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
'ecdhersaaes128gcm': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
'ecdhersaaes256gcm': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
'rsaaes128gcm': 'tls_rsa_aes_128_gcm_sha256',
'rsaaes256gcm': 'tls_rsa_aes_256_gcm_sha384',
}
"""Cipher suite display-name normalisation lookup.
Keys are IANA/display names with separators removed and lowercased
(e.g. ``'TLS_AES_128_GCM_SHA256'`` → ``'tlsaes128gmcsha256'``).
"""
# ═════════════════════════════════════════════════════════════════════════════
# Named Curve maps
# ═════════════════════════════════════════════════════════════════════════════
CURVE_MAP: dict[str, str] = {
# hex wire format → canonical name
'00 1d': 'x25519',
'00 17': 'secp256r1',
'00 18': 'secp384r1',
'00 19': 'secp521r1',
'00 1e': 'x448',
# display names
'x25519': 'x25519',
'curve25519': 'x25519',
'secp256r1': 'secp256r1',
'prime256v1': 'secp256r1',
'p256': 'secp256r1',
'p-256': 'secp256r1',
'p_256': 'secp256r1',
'secp384r1': 'secp384r1',
'p384': 'secp384r1',
'p-384': 'secp384r1',
'p_384': 'secp384r1',
'secp521r1': 'secp521r1',
'p521': 'secp521r1',
'p-521': 'secp521r1',
'p_521': 'secp521r1',
'x448': 'x448',
'curve448': 'x448',
}
"""Named curve mapping.
Keys are hex wire format (``'00 1d'``) or display names (``'X25519'``,
``'prime256v1'``, etc.). Matching is case-insensitive.
"""
def _build_when_chain(
v: pl.Expr,
mapping: dict[str, str],
fallback: pl.Expr,
) -> pl.Expr:
"""Build a ``pl.when()...then()...otherwise()`` chain from *mapping*.
Parameters
----------
v:
Polars expression representing the column value (already
lowercased and stripped).
mapping:
``{lookup_key: canonical_value}`` dict.
fallback:
Expression for unmatched values.
Returns
-------
pl.Expr
A chain ``pl.when(...).then(...)...otherwise(fallback)``.
"""
expr = pl.when(v.is_null()).then(pl.lit(None, pl.Utf8))
for key, val in mapping.items():
expr = expr.when(v == key).then(pl.lit(val))
return expr.otherwise(fallback)
# ═════════════════════════════════════════════════════════════════════════════
# Per-column normalisation functions
# ═════════════════════════════════════════════════════════════════════════════
def _norm_tls_version(col: str) -> pl.Expr:
"""Normalise a TLS version column (``0ver``).
Accepts hex wire format (``'03 04'``) or display names
(``'TLSv1.3'``, ``'tls 1.3'``, ``'1.3'``, ``'SSLv3'``, etc.).
Unmatched values produce ``'unknown_version'``.
"""
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
return _build_when_chain(v, TLS_VERSION_MAP, pl.lit('unknown_version'))
def _norm_cipher(col: str) -> pl.Expr:
"""Normalise a cipher-suite hex column (``0cph``).
Unmatched hex values produce ``'unknown_cipher_0x{hex}'`` where
``{hex}`` is the original (stripped, lowercased) value.
"""
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
fallback = pl.lit('unknown_cipher_0x') + v
return _build_when_chain(v, CIPHER_SUITE_MAP, fallback)
def _norm_cipher_display(col: str) -> pl.Expr:
"""Normalise a cipher-suite display-name column (``cipher-suite``).
The column value is lowercased and all separators (spaces, hyphens,
underscores) are removed before lookup.
Unmatched names produce ``'{normalised}_unknown'``.
"""
raw = pl.col(col).cast(pl.Utf8).str.strip_chars()
v = raw.str.to_lowercase().str.replace_all(r'[\s\-_]', '')
fallback = v + pl.lit('_unknown')
return _build_when_chain(v, _CIPHER_DISPLAY_NORMALIZED, fallback)
def _norm_curve(col: str) -> pl.Expr:
"""Normalise a named-curve hex column (``0crv``).
Unmatched hex values produce ``'unknown_curve_0x{hex}'``.
"""
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
fallback = pl.lit('unknown_curve_0x') + v
return _build_when_chain(v, CURVE_MAP, fallback)
def _norm_curve_display(col: str) -> pl.Expr:
"""Normalise a named-curve display-name column (``ecdhe-named-curve``).
Accepts names like ``'X25519'``, ``'prime256v1'``, ``'P-256'``, etc.
Unmatched names produce ``'unknown_curve_{name}'`` where ``{name}``
is the original value lowercased.
"""
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
fallback = pl.lit('unknown_curve_') + v
return _build_when_chain(v, CURVE_MAP, fallback)
def _norm_cnrs(col: str) -> pl.Expr:
"""Normalise a CNRS / ISRS column.
``'+'`` (optionally surrounded by whitespace) → ``'yes'``.
Blank, null, or any other value → ``'no'``.
These columns are **never** missing; blank always means ``'no'``.
"""
c = pl.col(col).cast(pl.Utf8).str.strip_chars()
return (
pl.when(c.is_null() | (c == ''))
.then(pl.lit('no'))
.when(c.str.to_lowercase() == '+')
.then(pl.lit('yes'))
.otherwise(pl.lit('no'))
)
# ═════════════════════════════════════════════════════════════════════════════
# Public API
# ═════════════════════════════════════════════════════════════════════════════
def normalize_lf(lf: pl.LazyFrame, type_map: dict) -> pl.LazyFrame:
"""Add ``{col}_norm`` columns with normalised values.
Parameters
----------
lf:
A Polars :class:`LazyFrame` containing TLS-related columns.
type_map:
A ``{column_name: ...}`` dict (e.g. from
:func:`~analysis.type_classifier.classify_schema`). Column
names present in the map receive normalisation.
Returns
-------
pl.LazyFrame
A new :class:`LazyFrame` with ``{col}_norm`` columns appended.
"""
exprs: list[pl.Expr] = []
if '0ver' in type_map:
exprs.append(_norm_tls_version('0ver').alias('0ver_norm'))
if '0cph' in type_map:
exprs.append(_norm_cipher('0cph').alias('0cph_norm'))
if 'cipher-suite' in type_map:
exprs.append(
_norm_cipher_display('cipher-suite').alias('cipher_suite_norm')
)
if '0crv' in type_map:
exprs.append(_norm_curve('0crv').alias('0crv_norm'))
if 'ecdhe-named-curve' in type_map:
exprs.append(
_norm_curve_display('ecdhe-named-curve').alias(
'ecdhe_named_curve_norm'
)
)
if 'cnrs' in type_map:
exprs.append(_norm_cnrs('cnrs').alias('cnrs_norm'))
if 'isrs' in type_map:
exprs.append(_norm_cnrs('isrs').alias('isrs_norm'))
if exprs:
lf = lf.with_columns(exprs)
return lf
analysis/views.py
+909
View File
@@ -0,0 +1,909 @@
import json
import time
import urllib.request
import urllib.error
import threading
import os
from pathlib import Path
from django.shortcuts import render, get_object_or_404, redirect
from django.http import HttpResponse, JsonResponse
from django.views.decorators.csrf import csrf_exempt
from config import get_config, save_config, Config
from .models import AnalysisRun, ClusterResult, EntityProfile
def dashboard(request):
"""Home page: recent runs overview."""
runs = AnalysisRun.objects.all()[:20]
return render(request, 'analysis/dashboard.html', {'runs': runs})
def run_list(request):
"""List all analysis runs with pagination."""
page = int(request.GET.get('page', 1))
per_page = 20
total = AnalysisRun.objects.count()
runs = AnalysisRun.objects.all()[(page - 1) * per_page: page * per_page]
return render(request, 'analysis/run_list.html', {
'runs': runs,
'page': page,
'total_pages': (total + per_page - 1) // per_page if total else 1,
})
def run_detail(request, run_id):
"""Details for a single analysis run."""
run = get_object_or_404(AnalysisRun, id=run_id)
clusters = run.clusters.all().order_by('-size')
return render(request, 'analysis/run_detail.html', {
'run': run,
'clusters': clusters,
})
def _extract_lat(val):
"""Try to extract a latitude from *val* (number or str)."""
if val is None:
return None
try:
v = float(val)
return None if v < -90 or v > 90 else v
except (ValueError, TypeError):
return None
def _extract_lon(val):
"""Try to extract a longitude from *val* (number or str)."""
if val is None:
return None
try:
v = float(val)
return None if v < -180 or v > 180 else v
except (ValueError, TypeError):
return None
def cluster_overview(request, run_id):
"""Overview of all clusters for a run, with PCA scatter data and geo scatter data."""
run = get_object_or_404(AnalysisRun, id=run_id)
clusters = run.clusters.exclude(cluster_label=-1).order_by('-size')
noise = run.clusters.filter(cluster_label=-1).first()
# Build PCA scatter data for Chart.js
entities = run.entities.exclude(embedding_x=None).exclude(embedding_y=None)
scatter_data = [
{'x': e.embedding_x, 'y': e.embedding_y, 'label': e.cluster_label, 'entity': e.entity_value}
for e in entities
]
# Build geo scatter data from feature_json (lat/lon)
geo_data = []
geo_skipped = 0
for e in run.entities.all():
fj = e.feature_json
if not fj:
continue
lat = _extract_lat(fj.get('avg_latitude'))
lon = _extract_lon(fj.get('avg_longitude'))
if lat is not None and lon is not None:
geo_data.append({
'x': lon,
'y': lat,
'label': e.cluster_label if e.cluster_label is not None else -1,
'entity': e.entity_value,
})
else:
geo_skipped += 1
# Cluster sizes for bar chart
cluster_sizes = [{'label': c.cluster_label, 'size': c.size, 'silhouette': c.silhouette_score}
for c in clusters]
# Top features per cluster from ClusterFeature
top_features = {}
for c in clusters:
features_qs = c.features.all().order_by('-distinguishing_score')[:10]
top_features[str(c.cluster_label)] = [
{'feature_name': f.feature_name, 'score': f.distinguishing_score, 'mean': f.mean}
for f in features_qs
]
return render(request, 'analysis/cluster_overview.html', {
'run': run,
'clusters': clusters,
'noise': noise,
'scatter_data_json': json.dumps(scatter_data),
'geo_data_json': json.dumps(geo_data),
'geo_skipped': geo_skipped,
'geo_count': len(geo_data),
'cluster_sizes_json': json.dumps(cluster_sizes),
'top_features_json': json.dumps(top_features),
})
def cluster_detail(request, run_id, cluster_label):
"""Detailed view of a single cluster with entity list."""
run = get_object_or_404(AnalysisRun, id=run_id)
cluster = get_object_or_404(ClusterResult, run=run, cluster_label=cluster_label)
features = cluster.features.all().order_by('-distinguishing_score')[:50]
entities = cluster.entities.all()[:50]
return render(request, 'analysis/cluster_detail.html', {
'run': run,
'cluster': cluster,
'features': features,
'entities': entities,
})
def entity_profile(request, entity_id):
"""Detailed profile for a single entity."""
entity = get_object_or_404(EntityProfile, id=entity_id)
run = entity.run
# Compute deviation from cluster mean
deviations = {}
if entity.feature_json and entity.cluster:
cluster_features = {
f.feature_name: {'mean': f.mean, 'std': f.std}
for f in entity.cluster.features.all()
}
for col, val in entity.feature_json.items():
if col in cluster_features and cluster_features[col].get('std') and cluster_features[col]['std'] > 1e-10:
deviations[col] = {
'value': val,
'cluster_mean': cluster_features[col]['mean'],
'zscore': (val - cluster_features[col]['mean']) / cluster_features[col]['std'],
}
return render(request, 'analysis/entity_profile.html', {
'entity': entity,
'run': run,
'deviations': deviations,
})
def start_analysis(request):
"""Placeholder: API endpoint to trigger analysis via MCP."""
return JsonResponse({'status': 'not_implemented', 'message': 'Use MCP tools to trigger analysis'})
def config_view(request):
"""Config page: GET renders current config, POST saves updates."""
cfg = get_config()
if request.method == 'POST':
# Build a new Config from POST data (fall back to current values)
server = Config.Server(
host=request.POST.get('server_host', cfg.server.host),
port=int(request.POST.get('server_port', cfg.server.port)),
debug=request.POST.get('server_debug') == 'true',
)
data = Config.Data(
schema_strict=request.POST.get('data_schema_strict') == 'true',
recursive=request.POST.get('data_recursive') == 'true',
)
clustering = Config.Clustering(
algorithm=request.POST.get('clustering_algorithm', cfg.clustering.algorithm),
min_cluster_size=int(request.POST.get('clustering_min_cluster_size', cfg.clustering.min_cluster_size)),
random_state=cfg.clustering.random_state,
)
llm = Config.LLM(
enabled=request.POST.get('llm_enabled') == 'true',
base_url=request.POST.get('llm_base_url', cfg.llm.base_url),
api_key=request.POST.get('llm_api_key', cfg.llm.api_key),
model=request.POST.get('llm_model', cfg.llm.model),
)
cfg = Config(server=server, data=data, clustering=clustering, llm=llm)
save_config(cfg)
# Re-render with saved flag
return render(request, 'tianxuan/config.html', {'config': cfg, 'saved': True})
return render(request, 'tianxuan/config.html', {'config': cfg})
@csrf_exempt
def llm_test(request):
"""LLM connectivity test: POST JSON {base_url, api_key, model} → chat/completions."""
if request.method != 'POST':
return JsonResponse({'success': False, 'message': '仅支持 POST 请求', 'latency_ms': 0}, status=405)
try:
body = json.loads(request.body)
except json.JSONDecodeError:
return JsonResponse({'success': False, 'message': '无效的 JSON 请求体', 'latency_ms': 0})
base_url = (body.get('base_url') or '').rstrip('/')
api_key = body.get('api_key') or ''
model = body.get('model') or ''
if not base_url:
return JsonResponse({'success': False, 'message': 'Base URL 不能为空', 'latency_ms': 0})
if not api_key:
return JsonResponse({'success': False, 'message': 'API Key 不能为空', 'latency_ms': 0})
if not model:
return JsonResponse({'success': False, 'message': 'Model 不能为空', 'latency_ms': 0})
url = f'{base_url}/chat/completions'
payload = json.dumps({
'model': model,
'messages': [{'role': 'user', 'content': 'respond with ok'}],
'max_tokens': 5,
}).encode('utf-8')
req = urllib.request.Request(
url,
data=payload,
headers={
'Content-Type': 'application/json',
'Authorization': f'Bearer {api_key}',
},
method='POST',
)
start = time.monotonic()
try:
with urllib.request.urlopen(req, timeout=5) as resp:
elapsed = int((time.monotonic() - start) * 1000)
data = json.loads(resp.read().decode('utf-8'))
if 'choices' in data and len(data['choices']) > 0:
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=True latency={elapsed}ms')
return JsonResponse({
'success': True,
'message': f'连接成功，模型 {model} 返回正常',
'latency_ms': elapsed,
})
else:
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=True latency={elapsed}ms')
return JsonResponse({
'success': True,
'message': f'连接成功，但响应中无 choices（原始响应已记录）',
'latency_ms': elapsed,
})
except urllib.error.HTTPError as e:
elapsed = int((time.monotonic() - start) * 1000)
try:
detail = json.loads(e.read().decode('utf-8', errors='replace'))
msg = detail.get('error', {}).get('message', str(e))
except Exception:
msg = str(e)
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
return JsonResponse({'success': False, 'message': f'HTTP {e.code}: {msg}', 'latency_ms': elapsed})
except urllib.error.URLError as e:
elapsed = int((time.monotonic() - start) * 1000)
reason = str(e.reason) if e.reason else '连接失败'
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
return JsonResponse({'success': False, 'message': f'连接失败: {reason}', 'latency_ms': elapsed})
except TimeoutError:
elapsed = int((time.monotonic() - start) * 1000)
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
return JsonResponse({'success': False, 'message': '请求超时（5秒）', 'latency_ms': elapsed})
except Exception as e:
elapsed = int((time.monotonic() - start) * 1000)
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
return JsonResponse({'success': False, 'message': f'未知错误: {str(e)}', 'latency_ms': elapsed})
# ── Upload page ─────────────────────────────────────────────────────────
def upload_page(request):
"""CSV upload page."""
runs = AnalysisRun.objects.all().order_by('-created_at')[:20]
return render(request, 'tianxuan/upload.html', {'runs': runs})
@csrf_exempt
def upload_csv(request):
"""Handle CSV upload AJAX POST."""
if request.method != 'POST':
return JsonResponse({'error': 'POST required'}, status=405)
files = request.FILES.getlist('files')
if not files:
return JsonResponse({'error': 'No files uploaded'})
if len(files) > 10000:
return JsonResponse({'error': '一次最多上传 10000 个文件'})
for f in files:
if f.size > 500 * 1024 * 1024:
return JsonResponse({'warning': f'文件 {f.name} 超过 500MB，建议分割后上传'})
total_size = sum(f.size for f in files)
if total_size > 50 * 1024 * 1024 * 1024:
return JsonResponse({'error': '总大小超过 50GB 限制'})
from django.utils import timezone
ts = timezone.now().strftime('%Y%m%d_%H%M%S')
import platform
if platform.system() == 'Windows':
base = Path(os.environ.get('APPDATA', '')) / 'TianXuan' / 'data' / 'uploads'
else:
base = Path.home() / '.tianxuan' / 'data' / 'uploads'
upload_dir = base / ts
upload_dir.mkdir(parents=True, exist_ok=True)
saved = []
for f in files:
path = upload_dir / f.name
with open(path, 'wb+') as dst:
for chunk in f.chunks():
dst.write(chunk)
saved.append(str(path))
# Auto-process in background
run = AnalysisRun.objects.create(
csv_glob=str(upload_dir / '*.csv'),
status='loading',
)
t = threading.Thread(target=_background_process, args=(run.id, upload_dir), daemon=True)
t.start()
return JsonResponse({'run_id': run.id, 'files': saved, 'status': 'loading'})
def delete_upload(request, run_id):
"""Delete uploaded files and AnalysisRun."""
run = get_object_or_404(AnalysisRun, id=run_id)
import shutil
upload_dir = None
if run.csv_glob:
p = Path(run.csv_glob).parent
if p.exists():
shutil.rmtree(p, ignore_errors=True)
run.delete()
return JsonResponse({'deleted': True})
import traceback, logging
logger = logging.getLogger(__name__)
def _background_process(run_id, upload_dir):
"""Background: load → detect → aggregate → ready."""
import django
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
django.setup()
from analysis.models import AnalysisRun
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
from analysis.session_store import SessionStore
run = AnalysisRun.objects.get(id=run_id)
store = SessionStore()
try:
# Check upload directory still exists (may have been temp-cleaned)
if not upload_dir.is_dir():
msg = '上传数据目录已不存在，请重新上传'
logger.error('[BACKGROUND] %s (run_id=%s, upload_dir=%s)', msg, run_id, upload_dir)
run.status = 'failed'
run.error_message = msg
run.save(update_fields=['status', 'error_message'])
return
run.status = 'loading'; run.progress_pct = 10; run.progress_msg = '正在加载数据...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
lf, schema, row_count, file_count, mem = load_csv_directory(
str(upload_dir / '*.csv'), schema_strict=False
)
ds_id = f'upload_{run_id}'
store.store_dataset(ds_id, lf, schema=schema, metadata={
'row_count': row_count, 'file_count': file_count, 'upload_dir': str(upload_dir),
'csv_glob': str(upload_dir / '*.csv'),
})
run.total_flows = row_count; run.save(update_fields=['total_flows'])
run.status = 'profiling'; run.progress_pct = 30; run.progress_msg = '正在分析数据特征...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
entity_result = detect_entity_column(ds_id)
entity_col = entity_result.get('recommended', '')
entity_cols_multi = entity_result.get('recommended_multi', [])
# Store as comma-separated (multi) or single-column
if entity_cols_multi:
entity_cols = entity_cols_multi[:2] # top 2
run.entity_column = ','.join(entity_cols)
else:
entity_cols = [entity_col] if entity_col else []
run.entity_column = entity_col
run.save(update_fields=['entity_column'])
if entity_cols:
run.status = 'aggregating'; run.progress_pct = 50; run.progress_msg = '正在构建实体画像...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
agg_lf, features = aggregate_by_entity(lf, entity_cols, schema)
df_agg = agg_lf.collect(streaming=True)
entity_ds_id = f'entity_{run_id}'
store.store_dataset(entity_ds_id, agg_lf,
schema=dict(zip(features, [''] * len(features))),
metadata={'row_count': len(df_agg), 'entity_column': ','.join(entity_cols),
'csv_glob': str(upload_dir / '*.csv')})
run.entity_count = len(df_agg)
run.entity_column = ','.join(entity_cols)
run.save(update_fields=['entity_column', 'entity_count'])
run.status = 'ready'; run.progress_pct = 60; run.progress_msg = '数据准备就绪'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
except Exception:
tb = traceback.format_exc()
logger.error(tb)
run.status = 'failed'
run.run_log += f'\n[ERROR] {tb}'
run.save(update_fields=['run_log'])
run.error_message = tb
run.save(update_fields=['status', 'error_message'])
# ── Manual analysis page ───────────────────────────────────────────────
def manual_page(request):
"""4-step manual analysis wizard."""
step = int(request.GET.get('step', 1))
runs = AnalysisRun.objects.filter(status='ready').order_by('-created_at')[:20]
ctx = {'step': step, 'runs': runs}
# Step 2: pass entity candidates and schema for the selected run
if step >= 2:
run_id = request.GET.get('run_id')
if run_id:
try:
run = AnalysisRun.objects.get(id=int(run_id))
ctx['selected_run'] = run
# Try to get entity detection candidates from SessionStore
from analysis.session_store import SessionStore
from analysis.entity_detector import detect_entity_column
store = SessionStore()
ds_id = f'upload_{run_id}'
entry = store.get_dataset(ds_id)
if entry is not None:
schema = entry.get('schema', {})
ctx['schema_keys'] = list(schema.keys())
# Build numeric-only feature column options with dtype labels
numeric_types = {'Int8', 'Int16', 'Int32', 'Int64',
'UInt8', 'UInt16', 'UInt32', 'UInt64',
'Float32', 'Float64'}
ctx['feature_options'] = [
{'name': k, 'dtype': v}
for k, v in schema.items()
if v.split('(')[0].strip() in numeric_types
]
# Detect entity candidates
det_result = detect_entity_column(ds_id)
if not det_result.get('error'):
ctx['entity_candidates'] = det_result.get('candidates', [])
# Pre-select stored entity columns (possibly comma-separated multi)
entity_cols_str = run.entity_column
if entity_cols_str:
ctx['selected_entity_cols'] = entity_cols_str.split(',')
except (AnalysisRun.DoesNotExist, ValueError):
pass
# Step 3: pass config defaults
from config import get_config
cfg = get_config()
ctx['default_algo'] = cfg.clustering.algorithm
ctx['default_min_size'] = cfg.clustering.min_cluster_size
return render(request, 'tianxuan/manual.html', ctx)
@csrf_exempt
def run_analysis(request):
"""AJAX POST: run analysis pipeline and return result page URL."""
if request.method != 'POST':
return JsonResponse({'error': 'POST required'}, status=405)
import json
body = json.loads(request.body)
run_id = body.get('run_id')
entity_col = body.get('entity_column', '')
entity_columns = body.get('entity_columns')
feature_columns = body.get('feature_columns')
algorithm = body.get('algorithm', 'hdbscan')
min_cluster_size = int(body.get('min_cluster_size', 5))
t = threading.Thread(
target=_run_analysis_worker,
args=(run_id, entity_col, entity_columns, feature_columns, algorithm, min_cluster_size),
daemon=True,
)
t.start()
return JsonResponse({'status': 'started', 'redirect': f'/runs/{run_id}/'})
def _run_analysis_worker(run_id, entity_col, entity_columns, feature_columns, algorithm, min_cluster_size):
"""Background: cluster + extract features for a run."""
import django, os
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
django.setup()
from analysis.models import AnalysisRun
from analysis.session_store import SessionStore
from analysis.tool_registry import _cluster_sync, _handle_extract_features
from analysis.entity_aggregator import aggregate_by_entity
from asgiref.sync import async_to_sync
run = AnalysisRun.objects.get(id=run_id)
store = SessionStore()
try:
# If entity_columns provided, re-aggregate from the raw upload dataset
if entity_columns:
upload_ds_id = f'upload_{run_id}'
upload_entry = store.get_dataset(upload_ds_id)
if upload_entry is None:
run.error_message = '请先上传并等待预处理完成'
run.status = 'failed'
run.save(update_fields=['status', 'error_message'])
return
upload_lf = upload_entry['lazyframe']
upload_schema = upload_entry.get('schema', {})
agg_lf, agg_features = aggregate_by_entity(upload_lf, entity_columns, upload_schema)
df_agg = agg_lf.collect(streaming=True)
agg_schema = {n: str(d) for n, d in zip(df_agg.columns, [df_agg[c].dtype for c in df_agg.columns])}
entity_ds_id = f'entity_{run_id}_custom'
store.store_dataset(
dataset_id=entity_ds_id,
lazyframe=df_agg.lazy(),
schema=agg_schema,
metadata={'row_count': len(df_agg), 'entity_columns': entity_columns,
'csv_glob': run.csv_glob},
)
run.entity_column = ','.join(entity_columns)
run.entity_count = len(df_agg)
run.save(update_fields=['entity_column', 'entity_count'])
else:
entity_ds_id = f'entity_{run_id}'
entry = store.get_dataset(entity_ds_id)
if entry is None:
run.error_message = '请先上传并等待预处理完成'
run.status = 'failed'
run.save(update_fields=['status', 'error_message'])
return
run.status = 'clustering'; run.progress_pct = 70; run.progress_msg = '正在进行聚类分析...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
entry = store.get_dataset(entity_ds_id)
schema = entry.get('schema', {})
if feature_columns:
entity_cols_set = set()
if hasattr(run, 'entity_column') and run.entity_column:
entity_cols_set = {c.strip() for c in run.entity_column.split(',')}
feature_cols = [c for c in feature_columns if c in schema and c not in entity_cols_set]
else:
entity_cols_set = set()
if hasattr(run, 'entity_column') and run.entity_column:
entity_cols_set = {c.strip() for c in run.entity_column.split(',')}
feature_cols = [c for c in schema.keys() if not c.startswith('_') and c not in entity_cols_set][:10]
result = _cluster_sync(
dataset_id=entity_ds_id,
cluster_columns=feature_cols,
algorithm=algorithm,
params={'min_cluster_size': min_cluster_size},
random_state=42,
)
if 'error' in result:
err = result['error']
if 'numeric' in err.lower():
entry = store.get_dataset(entity_ds_id)
if entry:
schema = entry.get('schema', {})
numeric_types = ('Int8', 'Int16', 'Int32', 'Int64',
'UInt8', 'UInt16', 'UInt32', 'UInt64',
'Float32', 'Float64')
available = [f"{k}({v})" for k, v in schema.items()
if v.split('(')[0].strip() in numeric_types and not k.startswith('_')]
err += f"。可用数值列: {available}"
raise Exception(err)
cluster_id = result.get('cluster_result_id', '')
run.cluster_count = result.get('n_clusters', 0)
run.save(update_fields=['cluster_count'])
run.status = 'extracting'; run.progress_pct = 90; run.progress_msg = '正在提取聚类特征...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
async_to_sync(_handle_extract_features)(
dataset_id=entity_ds_id,
cluster_result_id=cluster_id,
top_k=10, method='zscore', save_to_db=True,
)
run.status = 'completed'; run.progress_pct = 100; run.progress_msg = '分析完成'
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
except Exception:
tb = traceback.format_exc()
logger.error(tb)
last_pct = run.progress_pct
run.status = 'failed'
run.progress_msg = f'失败: {tb[:200]}'
run.run_log += f'\n[ERROR] {tb}'
run.save(update_fields=['run_log'])
run.error_message = tb
run.save(update_fields=['status', 'progress_msg', 'error_message'])
# ── LLM auto analysis page ─────────────────────────────────────────────
def _run_llm_analysis(run_id, dataset_id, entity_col):
"""Background: run LLM-driven analysis pipeline with progress callback."""
import django, os
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
django.setup()
from analysis.models import AnalysisRun
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
run = AnalysisRun.objects.get(id=run_id)
def progress_callback(step, tool_name, result):
nonlocal run
try:
run = AnalysisRun.objects.get(id=run_id)
pct = min(10 + step * 6, 95)
run.progress_pct = pct
run.progress_msg = f'步骤 {step}: {tool_name}'
if result:
summary = str(result)[:200]
run.run_log += f'[{step}] {tool_name}: {summary}\n'
else:
run.run_log += f'[{step}] {tool_name}\n'
run.save(update_fields=['progress_pct', 'progress_msg', 'run_log'])
except Exception:
pass
try:
from config import get_config
cfg = get_config()
llm_cfg = LLMConfig(cfg.llm.base_url, cfg.llm.api_key, cfg.llm.model)
run.run_log = '启动 LLM 分析...\n'
run.status = 'loading'
run.progress_pct = 5
run.progress_msg = 'LLM 初始化中...'
run.save(update_fields=['status', 'progress_pct', 'progress_msg', 'run_log'])
result = run_llm_pipeline(dataset_id, entity_col, config=llm_cfg, callback=progress_callback)
if 'error' in result:
run.status = 'failed'
run.progress_msg = f'失败: {result["error"]}'
run.run_log += f'\n[ERROR] {result["error"]}'
else:
run.status = 'completed'
run.progress_pct = 100
run.progress_msg = 'LLM 分析完成'
run.run_log += f'\n[完成] {result.get("result", "")}'
run.save(update_fields=['status', 'progress_pct', 'progress_msg', 'run_log'])
except Exception:
tb = traceback.format_exc()
logger.error(tb)
try:
run = AnalysisRun.objects.get(id=run_id)
run.status = 'failed'
run.progress_msg = f'失败: {tb[:200]}'
run.run_log += f'\n[ERROR] {tb}'
run.error_message = tb
run.save(update_fields=['status', 'progress_msg', 'run_log', 'error_message'])
except Exception:
pass
@csrf_exempt
def run_llm_analysis_view(request):
"""AJAX POST: run LLM-driven analysis pipeline."""
if request.method != 'POST':
return JsonResponse({'error': 'POST required'}, status=405)
import json
body = json.loads(request.body)
run_id = body.get('run_id')
entity_col = body.get('entity_column', '')
from analysis.session_store import SessionStore
store = SessionStore()
upload_ds_id = f'upload_{run_id}'
entry = store.get_dataset(upload_ds_id)
if entry is None:
# Try entity dataset as fallback
entity_ds_id = f'entity_{run_id}'
entry = store.get_dataset(entity_ds_id)
if entry is None:
return JsonResponse({'error': '数据集不存在，请先上传并等待预处理完成'})
dataset_id = upload_ds_id if store.get_dataset(upload_ds_id) else f'entity_{run_id}'
t = threading.Thread(
target=_run_llm_analysis,
args=(run_id, dataset_id, entity_col),
daemon=True,
)
t.start()
return JsonResponse({'status': 'started', 'run_id': run_id})
def auto_page(request):
"""LLM-driven auto analysis page."""
runs = AnalysisRun.objects.filter(status='ready').order_by('-created_at')[:20]
cfg = get_config()
return render(request, 'tianxuan/auto.html', {
'runs': runs,
'llm_configured': bool(cfg.llm.base_url and cfg.llm.api_key),
})
# ── Progress API ───────────────────────────────────────────────────────
def run_status_api(request, run_id):
"""Return current run status as JSON."""
run = get_object_or_404(AnalysisRun, id=run_id)
return JsonResponse({
'id': run.id,
'status': run.status,
'entity_count': run.entity_count,
'cluster_count': run.cluster_count,
'error_message': run.error_message,
'progress_pct': run.progress_pct,
'progress_msg': run.progress_msg,
'run_log': run.run_log[-2000:], # last 2000 chars
})
def log_viewer(request):
"""Display last 200 lines of the app log (DEBUG only).
With ?pos=N, returns text: line_count\\nnew_content for AJAX polling.
"""
from django.conf import settings
if not settings.DEBUG:
return HttpResponse('Log viewer only available in DEBUG mode', status=403)
log_path = settings.BASE_DIR / 'logs' / 'tianxuan.log'
pos = request.GET.get('pos')
if pos is not None:
# AJAX polling mode: return new content since byte position
try:
pos = int(pos)
except ValueError:
pos = 0
if log_path.exists():
with open(log_path, 'r', encoding='utf-8') as f:
total = f.seek(0, 2) # seek to end
if pos >= total:
return HttpResponse(f'{total}\n', content_type='text/plain')
f.seek(pos)
new_content = f.read()
return HttpResponse(f'{total}\n{new_content}', content_type='text/plain')
return HttpResponse('0\n', content_type='text/plain')
# HTML page mode
lines = []
if log_path.exists():
with open(log_path, 'r', encoding='utf-8') as f:
lines = f.readlines()[-200:]
return render(request, 'tianxuan/log_viewer.html', {'lines': lines})
# ── 3D Globe view ─────────────────────────────────────────────────────────
def _extract_flows_from_df(df, MAX_ROWS):
"""Extract flow data (lat/lon/TLS/bytes) from a Polars DataFrame."""
from analysis.geoip import lookup as geo_lookup
TLS_HEX_MAP = {'0303': 'TLSv1.2', '0304': 'TLSv1.3', '0302': 'TLSv1.1', '0301': 'TLSv1.0'}
src_lat_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
('src_latitude', 'src_lat', 'source_latitude', 'source_lat', ':ips_lat')), None)
src_lon_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
('src_longitude', 'src_lon', 'source_longitude', 'source_lon', ':ips_lon')), None)
dst_lat_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
('dst_latitude', 'dst_lat', 'dest_latitude', 'dest_lat', ':ipd_lat')), None)
dst_lon_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
('dst_longitude', 'dst_lon', 'dest_longitude', 'dest_lon', ':ipd_lon')), None)
src_ip_col = next((c for c in df.columns if 'src_ip' in c.lower() or ':ips' in c.lower()), None)
dst_ip_col = next((c for c in df.columns if 'dst_ip' in c.lower() or ':ipd' in c.lower()), None)
tls_col = next((c for c in df.columns if 'tls_version' in c.lower() or '_tlsver' in c.lower() or 'version' in c.lower()), None)
bytes_col = next((c for c in df.columns if 'bytes_sent' in c.lower() or 'bytes' in c.lower()), None)
flows = []
for row in df.iter_rows(named=True):
slat = slon = dlat = dlon = None
if src_lat_col and src_lon_col:
slat = _extract_lat(row.get(src_lat_col))
slon = _extract_lon(row.get(src_lon_col))
if dst_lat_col and dst_lon_col:
dlat = _extract_lat(row.get(dst_lat_col))
dlon = _extract_lon(row.get(dst_lon_col))
if (slat is None or slon is None) and src_ip_col:
sg = geo_lookup(str(row.get(src_ip_col, '')))
if sg:
slat, slon = sg['lat'], sg['lon']
if (dlat is None or dlon is None) and dst_ip_col:
dg = geo_lookup(str(row.get(dst_ip_col, '')))
if dg:
dlat, dlon = dg['lat'], dg['lon']
if slat is None or slon is None or dlat is None or dlon is None:
continue
tls_val = str(row.get(tls_col or '', ''))
# Check hex format first (e.g. "03 03" or "0303")
hex_key = tls_val.replace(' ', '').strip()
if hex_key in TLS_HEX_MAP:
tls_label = TLS_HEX_MAP[hex_key]
elif '1.3' in tls_val:
tls_label = 'TLSv1.3'
elif '1.2' in tls_val:
tls_label = 'TLSv1.2'
else:
tls_label = 'other'
bs = float(row.get(bytes_col or '', 0) or 0)
flows.append({
'slat': slat, 'slon': slon,
'dlat': dlat, 'dlon': dlon,
'tls': tls_label,
'bytes': bs,
})
return flows
def globe_view(request):
import json
from analysis.geoip import lookup as geo_lookup
from analysis.data_loader import load_csv_directory
from analysis.models import AnalysisRun
from analysis.session_store import SessionStore
MAX_ROWS_PER_RUN = 300 # Limit per run to prevent browser overload
# Check for ?data=dataset_id parameter (direct data loading from session store)
data_param = request.GET.get('data')
if data_param:
store = SessionStore()
entry = store.get_dataset(data_param)
flows = []
all_runs = list(AnalysisRun.objects.filter(status='completed').order_by('-id').values('id', 'csv_glob'))
if entry is not None:
lf = entry['lazyframe']
try:
df = lf.head(MAX_ROWS_PER_RUN).collect()
flows = _extract_flows_from_df(df, MAX_ROWS_PER_RUN)
except Exception:
pass
return render(request, 'tianxuan/globe.html', {
'geo_flows': json.dumps(flows),
'all_runs': all_runs,
'selected_ids': [],
})
# Get all completed runs for the selector
all_runs = list(AnalysisRun.objects.filter(status='completed').order_by('-id').values('id', 'csv_glob'))
# Get selected run IDs from query params (multiple checkboxes)
selected_ids_list = request.GET.getlist('runs')
selected_runs = []
if selected_ids_list:
for sid in selected_ids_list:
sid = sid.strip()
if sid.isdigit():
try:
r = AnalysisRun.objects.get(id=int(sid), status='completed')
selected_runs.append(r)
except AnalysisRun.DoesNotExist:
pass
if not selected_runs:
# Default: latest completed run
latest = AnalysisRun.objects.filter(status='completed').order_by('-id').first()
if latest:
selected_runs = [latest]
flows = []
for run in selected_runs:
try:
# Check upload directory exists before accessing
if run.csv_glob:
glob_parent = Path(run.csv_glob).parent
if not glob_parent.is_dir():
logger.warning('[GLOBE] Upload directory missing for run %s: %s', run.id, glob_parent)
continue
lf, schema, rc, fc, mem = load_csv_directory(run.csv_glob)
df = lf.head(MAX_ROWS_PER_RUN).collect()
flows.extend(_extract_flows_from_df(df, MAX_ROWS_PER_RUN))
except Exception:
continue
resp = render(request, 'tianxuan/globe.html', {
'geo_flows': json.dumps(flows),
'all_runs': all_runs,
'selected_ids': [r.id for r in selected_runs],
})
resp['Cache-Control'] = 'no-cache, no-store, must-revalidate'
resp['Pragma'] = 'no-cache'
resp['Expires'] = '0'
return resp
config/__init__.py
+4
View File
@@ -0,0 +1,4 @@
"""Configuration loader for TianXuan."""
from .loader import Config, get_config, save_config
__all__ = ['Config', 'get_config', 'save_config']
config/config.yaml
+87
View File
@@ -0,0 +1,87 @@
server:
host: 127.0.0.1
port: 8000
debug: false
data:
schema_strict: false
recursive: false
geoip:
enabled: true
ip_columns: ["src_ip", "dst_ip"]
drop_columns: [] # 默认不清除经纬列——由加载器自动清洗
clustering:
algorithm: hdbscan
min_cluster_size: 5
random_state: 42
llm:
enabled: false
base_url: ''
api_key: ''
model: gpt-4
entity:
subnet_masks: [24, 28]
ip_columns: ["src_ip", "dst_ip", ":ips", ":ipd", "server-ip", "client-ip"]
columns:
src_ip: ipv4
dst_ip: ipv4
":ips": ipv4
":ipd": ipv4
server-ip: ipv4
client-ip: ipv4
server_ip: ipv4
client_ip: ipv4
bytes_sent: int
bytes_rev: int
duration: float
tls_version: enum
cipher_suite: hex
dst_url: url
lat: lat_lon
lon: lat_lon
timestamp: timestamp
# User column overrides
"0ver": enum
":prs": enum
":prd": enum
scnt: enum
dcnt: enum
cnam: string
snam: string
"4dur": float
"8ses": float
"2tmo": float
"4ksz": int
cnrs: enum
isrs: enum
"8ack": int
"8ppk": int
"8dbd": timestamp
"1ipp": enum
"4dbn": enum
tabl: string
"name": string
source-node: string
source_node: string
ecdhe-named-curve: string
ecdhe_named_curve: string
"0cph": hex
"0crv": hex
"0rnd": string
"0rnt": float
"row": int
"time": timestamp
":ips.latd": lat_lon
":ipd.latd": lat_lon
":ips.lond": lat_lon
":ipd.lond": lat_lon
":ips.ispn": string
":ipd.ispn": string
":ips.orgn": string
":ipd.orgn": string
":ips.city": string
":ipd.city": string
"+.latd": lat_lon
"+.lond": lat_lon
"+.ispn": string
"+.orgn": string
"+.city": string
config/loader.py
+82
View File
@@ -0,0 +1,82 @@
"""Configuration loader with Pydantic model, YAML file, and caching."""
import pathlib
import yaml
from pydantic import BaseModel
class Config(BaseModel):
"""Typed configuration for TianXuan."""
class Server(BaseModel):
host: str = '127.0.0.1'
port: int = 8000
debug: bool = False
class Data(BaseModel):
schema_strict: bool = False
recursive: bool = False
class Clustering(BaseModel):
algorithm: str = 'hdbscan'
min_cluster_size: int = 5
random_state: int = 42
class LLM(BaseModel):
enabled: bool = False
base_url: str = ''
api_key: str = ''
model: str = 'gpt-4'
class Entity(BaseModel):
subnet_masks: list[int] = []
ip_columns: list[str] = ['src_ip', 'dst_ip']
entity: Entity = Entity()
server: Server = Server()
data: Data = Data()
clustering: Clustering = Clustering()
llm: LLM = LLM()
_config_path = pathlib.Path(__file__).parent / 'config.yaml'
_config_cache = None
_config_mtime = 0
def get_config() -> Config:
"""Return the cached Config, reloading if the YAML file has changed.
If *config.yaml* does not exist, returns an empty ``Config()`` with
all default values.
"""
global _config_cache, _config_mtime
if _config_path.exists():
mtime = _config_path.stat().st_mtime
if _config_cache is None or mtime > _config_mtime:
with open(_config_path, 'r', encoding='utf-8') as f:
data = yaml.safe_load(f) or {}
_config_cache = Config(**data)
_config_mtime = mtime
else:
if _config_cache is None:
_config_cache = Config()
return _config_cache
def save_config(cfg: Config) -> None:
"""Persist a Config instance back to *config.yaml* and update the cache."""
global _config_cache, _config_mtime
with open(_config_path, 'w', encoding='utf-8') as f:
yaml.dump(
cfg.model_dump(mode='python'),
f,
default_flow_style=False,
allow_unicode=True,
sort_keys=False,
)
_config_cache = cfg
_config_mtime = _config_path.stat().st_mtime
data/geoip_data.txt
+41
View File
@@ -0,0 +1,41 @@
# start_ip,end_ip,lat,lon,city,country
# === North America ===
1.0.0.0,1.0.0.255,37.386,-122.083,Mountain View,US
8.8.8.0,8.8.8.255,37.386,-122.083,Mountain View,US
72.14.192.0,72.14.255.255,40.7128,-74.0060,New York,US
74.125.0.0,74.125.255.255,40.7128,-74.0060,New York,US
66.102.0.0,66.102.255.255,34.0522,-118.2437,Los Angeles,US
64.0.0.0,64.255.255.255,43.6532,-79.3832,Toronto,CA
# === East Asia ===
1.2.4.0,1.2.4.255,39.9042,116.4074,Beijing,CN
123.125.114.0,123.125.114.255,39.9042,116.4074,Beijing,CN
101.95.0.0,101.95.255.255,31.2304,121.4737,Shanghai,CN
58.0.0.0,58.255.255.255,22.3193,114.1694,Hong Kong,HK
14.0.0.0,14.255.255.255,22.3193,114.1694,Hong Kong,HK
1.160.0.0,1.160.255.255,25.0330,121.5654,Taipei,TW
101.0.0.0,101.0.255.255,25.0330,121.5654,Taipei,TW
27.0.0.0,27.0.255.255,35.6762,139.6503,Tokyo,JP
210.130.0.0,210.130.255.255,35.6762,139.6503,Tokyo,JP
110.0.0.0,110.0.255.255,37.5665,126.9780,Seoul,KR
121.78.0.0,121.78.255.255,37.5665,126.9780,Seoul,KR
# === Southeast Asia ===
116.14.0.0,116.14.255.255,1.3521,103.8198,Singapore,SG
58.8.0.0,58.8.255.255,13.7563,100.5018,Bangkok,TH
# === South Asia ===
14.96.0.0,14.96.255.255,19.0760,72.8777,Mumbai,IN
# === Oceania ===
1.40.0.0,1.40.255.255,-33.8688,151.2093,Sydney,AU
149.135.0.0,149.135.255.255,-33.8688,151.2093,Sydney,AU
# === Europe ===
5.10.0.0,5.10.255.255,51.5074,-0.1278,London,GB
81.2.0.0,81.2.255.255,51.5074,-0.1278,London,GB
90.0.0.0,90.0.255.255,48.8566,2.3522,Paris,FR
46.0.0.0,46.0.255.255,50.1109,8.6821,Frankfurt,DE
145.0.0.0,145.0.255.255,52.3676,4.9041,Amsterdam,NL
84.203.0.0,84.203.255.255,53.3498,-6.2603,Dublin,IE
78.0.0.0,78.0.255.255,59.3293,18.0686,Stockholm,SE
95.0.0.0,95.0.255.255,55.7558,37.6173,Moscow,RU
# === Middle East ===
91.72.0.0,91.72.255.255,25.2048,55.2708,Dubai,AE
# === South America ===
177.0.0.0,177.0.255.255,-23.5505,-46.6333,Sao Paulo,BR
db.sqlite3
BIN
View File
Binary file not shown.
docs/工作流.md
+818
View File
@@ -0,0 +1,818 @@
# 天璇 (TianXuan) 分析工作流
> 从原始 TLS 流 CSV 到实体画像、聚类结果、区分特征的完整流水线文档。
---
## 目录
1. [数据准备](#1-数据准备)
2. [数据加载与配置](#2-数据加载与配置)
3. [数据校验](#3-数据校验)
4. [数据清洗](#4-数据清洗)
5. [实体识别](#5-实体识别)
6. [类型感知聚合](#6-类型感知聚合)
7. [聚类分析](#7-聚类分析)
8. [特征提取](#8-特征提取)
9. [可视化](#9-可视化)
10. [LLM 集成](#10-llm-集成)
11. [跨机部署](#11-跨机部署)
---
## 1. 数据准备
### 1.1 CSV 格式要求
天璇接收以 **CSV 格式**存放的 TLS 流记录文件。每行代表一条网络流。
| 要求 | 说明 |
|------|------|
| **分隔符** | 逗号 `,`（默认），支持自定义分隔符 |
| **文件扩展名** | `.csv` |
| **多文件** | 支持 glob 通配符、`**/*.csv` 递归扫描、自动合并 |
| **压缩包** | `.zip` 文件自动解压（通过上传界面） |
| **文件数量** | 无硬性限制（合并时使用 Polars `diagonal_relaxed` 模式） |
### 1.2 编码建议
| 编码 | BOM 标记 | 自动检测 |
|------|---------|---------|
| **UTF-8**（推荐） | 无 / `EF BB BF` | ✅ 默认 |
| UTF-8 with BOM | `EF BB BF` | ✅ 自动检测 |
| UTF-16LE | `FF FE` | ✅ 自动检测 |
| UTF-16BE | `FE FF` | ✅ 自动检测 |
| Latin-1 | 无 | ✅ 手动指定 |
> **⚠️ 生产建议**：始终使用 **UTF-8 无 BOM** 编码。跨机器部署时强烈建议通过 `set PYTHONUTF8=1` 强制 Python 使用 UTF-8 模式（`run.bat` 已包含该设置）。
### 1.3 推荐列
系统基于列名关键词自动识别以下维度：
| 维度 | 推荐列名（不区分大小写） |
|------|------------------------|
| **源 IP** | `src_ip`、`source_ip`、`src_addr` |
| **目的 IP** | `dst_ip`、`dest_ip`、`destination_ip`、`dst_addr`、`ip_dst` |
| **字节数（上行）** | `bytes_sent`、`src_bytes`、`bytes_out`、`up_bytes`、`upload` |
| **字节数（下行）** | `bytes_rev`、`bytes_received`、`dst_bytes`、`down_bytes`、`download` |
| **持续时间** | `duration`、`dur`、`elapsed`、`time_delta`、`flow_duration` |
| **包数** | `packets`、`pkts`、`packet_count`、`total_packets` |
| **TLS 版本** | `tls_version`、`version`、`ssl_version`、`tlsver` |
| **加密套件** | `cipher_suite`、`cipher`、`tls_cipher`、`ssl_cipher` |
| **SNI** | `sni`、`server_name`、`tls_sni` |
| **协议** | `proto`、`protocol`、`l4_proto`、`ip_proto` |
| **目的端口** | `dst_port`、`destination_port`、`dest_port`、`port_dst` |
| **目标 URL** | `dst_url`、`url`、`uri` |
| **时间戳** | `timestamp`、`ts`、`time`、`datetime`、`first_seen` |
| **纬度** | `lat`、`latitude`、`latitud`、`y` |
| **经度** | `lon`、`lng`、`longitude`、`longitud`、`x` |
> 列名通过下划线分词匹配关键词（如 `flow_duration` 匹配 `duration`）。匹配算法见 `analysis/entity_aggregator.py` 中的 `_find_column()`。
### 1.4 命名规范
- **不要求**列名在所有文件中完全一致（默认 `schema_strict=false`）
- 不同名的列自动取并集，缺失列填 `null`
- 建议团队内统一列名，以获得最佳的自动识别效果
---
## 2. 数据加载与配置
### 2.1 加载入口
**主函数**：`analysis/data_loader.py::load_csv_directory()`
```python
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
glob_pattern="data/*.csv",
encoding="utf-8",
delimiter=",",
schema_strict=False,
recursive=False,
)
```
**MCP 工具**（通过 `tool_registry.py` 暴露）：
```
load_data(csv_glob="data/*.csv", schema_strict=False, recursive=False)
```
**CLI 管道**（一键运行）：
```bash
runtime\python\python.exe manage.py run_pipeline data/*.csv
```
### 2.2 列类型配置
在 `config/config.yaml` 中通过 `columns` 映射手动指定列类型。配置优先级高于自动推断：
```yaml
columns:
src_ip: ipv4 # 强制识别为 IPv4
dst_ip: ipv4
bytes_sent: int # 强制为整数类型
bytes_rev: int
duration: float # 强制为浮点类型
tls_version: enum # 枚举类型（低基数分类变量）
cipher_suite: hex # 十六进制数据
dst_url: url # URL 字符串
lat: lat_lon # 地理纬度
lon: lat_lon # 地理经度
timestamp: timestamp # 时间戳
```
支持的类型字符串及映射：
| 配置值 | `DataType` | 聚合方式 |
|--------|-----------|---------|
| `int` | `INT` | mean / sum |
| `float` | `FLOAT` | mean / sum |
| `enum` | `ENUM` | mode（众数） |
| `hex` | `HEX` | count（hex 对数统计） |
| `url` | `URL` | domain（提取域名） |
| `ipv4` | `IPv4` | subnet（/24 网络） |
| `lat_lon` | `LAT_LON` | mean |
| `timestamp` | `TIMESTAMP` | min / max |
| `bool_enum` | `BOOL_ENUM` | mode |
| `string` | `STRING` | count |
### 2.3 Schema 容错
| 模式 | 行为 | 适用场景 |
|------|------|---------|
| `schema_strict: true` | 列名不一致时报 `ValueError` | 测试/质量门禁 |
| `schema_strict: false`（默认） | 自动并集合并，缺失列填 null | 日常使用 |
**建议**：生产环境建议 `schema_strict: true`，提前发现列名漂移。
### 2.4 GeoIP 配置
`config.yaml` 中的 GeoIP 配置段控制自动经纬度填充和旧列丢弃：
```yaml
data:
geoip:
enabled: true
ip_columns: ["src_ip", "dst_ip"] # 需查询的 IP 列
drop_columns: ["src_latitude", "src_longitude", # 加载后丢弃的旧列
"dst_latitude", "dst_longitude"]
```
GeoIP 数据文件：`data/geoip_data.txt`，内嵌约 23 个主要城市 IP 段（200 KB 二进制查询格式）。
查询接口：
```python
from analysis.geoip import lookup
result = lookup("8.8.8.8") # → {lat: 37.386, lon: -122.083, city: "Mountain View", country: "US"}
```
> ⚠️ **旧列丢弃**：如果 CSV 本身已包含 `src_latitude`/`src_longitude` 等列（来自旧有系统的 GeoIP 结果），系统在加载后自动丢弃这些列，避免与新的 GeoIP 查询冲突。
---
## 3. 数据校验
### 3.1 校验入口
**主函数**：`analysis/data_validator.py::validate()`
```python
from analysis.data_validator import validate, report
# 返回 JSON 报告
validation_result = validate("dataset_id", strict=False)
# 返回格式化字符串报告
print(report("dataset_id"))
```
校验项通过 `[VALIDATE]` 日志标记输出。
### 3.2 校验项说明
| 校验项 | 检测内容 | 判定标准 |
|--------|---------|---------|
| **列类型矛盾** | schema 推断类型 vs classify_column 结果 | `INT↔FLOAT`、`ENUM↔BOOL_ENUM` 为良性差异；STRING 类型包容一切 |
| **缺失率** | 每列 `null_count / total_rows` | `>50%` → **高风险**；`>30%` → 警告 |
| **异常值** | 数值列 Z-score > 5 的比例 | `>5%` 触发警告 |
| **枚举分布** | ENUM/BOOL_ENUM 列的值频率 | 标记空白、`+`、`-` 等特殊符号 |
| **IPv4 有效性** | IP 列每个值的合法性 | 无效 IP 比例 `>50%` → **高风险** |
### 3.3 校验报告解读
JSON 报告结构：
```json
{
"valid": true, // 无高风险项时为 true
"dataset_id": "ds_abc123",
"total_rows": 5000,
"total_columns": 24,
"columns": {
"src_ip": {
"dtype": "String",
"inferred_type": "IPv4",
"null_rate": 0.02,
"warnings": []
},
"duration": {
"dtype": "Float64",
"inferred_type": "FLOAT",
"null_rate": 0.55,
"z_score_gt_5_ratio": 0.03,
"warnings": ["null rate 55.0% exceeds 50% (high risk)"]
}
},
"warnings": [...],
"risks": ["Column 'duration': null rate 55.0% exceeds 50% (high risk)"]
}
```
### 3.4 处理建议
| 发现问题 | 建议处理 |
|---------|---------|
| 缺失率 > 50% | 检查数据源是否漏采集；考虑在预处理阶段 `fillna` 或 `drop` |
| Z-score > 5 比例过高 | 检查是否有异常流量（DDoS、扫描等） |
| 无效 IPv4 比例高 | 检查列名是否错误（非 IP 列被匹配为 IPv4） |
| 枚举值异常（空白/`+`/`-`） | 这些值会在清洗阶段被规范化（见第 4 节） |
---
## 4. 数据清洗
数据清洗在 `load_csv_directory()` 中自动执行，分三个阶段。
### 4.1 旧经纬度丢弃
加载完成后，立即丢弃 GeoIP 配置中指定的旧列：
```python
# 默认丢弃列
drop_columns = ["src_latitude", "src_longitude", "dst_latitude", "dst_longitude"]
# 只丢弃 schema 中实际存在的列
merged_lf = merged_lf.drop(existing_drop)
```
日志标记：`[CLEAN] dropped columns: [...]`
### 4.2 BOOL_ENUM 标准化
对 `classify_schema()` 标记为 `DataType.BOOL_ENUM` 的字符串列，执行以下变换：
| 原始值 | 标准化后 |
|--------|---------|
| `""`（空字符串） | `null` |
| `"+"` | `"True"` |
| `"-"` | `"False"` |
| `"t"`, `"T"`, `"true"`, `"True"` | 保持不变 |
| `"f"`, `"F"`, `"false"`, `"False"` | 保持不变 |
> 原生 Boolean 类型的列跳过此步骤（已经是标准格式）。
日志标记：`[CLEAN] BOOL_ENUM columns normalised: [...]`
### 4.3 HEX 预处理
对标记为 `DataType.HEX` 的列（如 cipher_suite），添加 `{col}_hex_pairs_count` 辅助列：
```python
# 计算每个值中空格分隔的 hex 对数量
pl.col("cipher_suite").str.split(" ").list.len().alias("cipher_suite_hex_pairs_count")
```
该计数列在后续聚合中以 `avg_hex_pairs_count` 参与聚类特征。
日志标记：`[CLEAN] HEX columns added hex_pairs_count: [...]`
### 4.4 通用数值清洗
对于任何未被分类为 IPv4/URL/HEX/ENUM/LAT_LON 的字符串列，系统自动尝试转为 Float64：
| 原始值 | 清洗后 |
|--------|--------|
| `"123.45"` | `123.45` |
| `"+"` / `"-"` | `null` |
| 空白／空字符串 | `null` |
| `"N/A"` / `"null"` / `"None"` | `null` |
| 其他不可解析字符串 | `null` |
这防止了"mean on str column"崩溃（当字符串列因数据伪影未能被早期类型检测捕获时），
无论后续数据中出现何种新伪影。
函数：`analysis/entity_aggregator.py` → `_coerce_to_float()`
日志标记：`[CLEAN] numeric coercion applied to: [...]`
---
## 5. 实体识别
### 5.1 自动检测原理
**主函数**：`analysis/entity_detector.py::detect_entity_column()`
采用三维打分算法，对所有 String/Categorical 列评分：
| 维度 | 分值 | 说明 |
|------|------|------|
| **名称匹配** | +3.0 | 列名（归一化后）精确匹配关键词或 `_` 分隔匹配 |
| **唯一值比率** | 0~10 | `unique_ratio = unique_count / non_null_count`，在 `[0.01, 0.50]` 区间内得分最高 |
| **字符串类型** | +2.0 | `Utf8` / `String` / `Categorical` 类型加分 |
| **高唯一性惩罚** | -5.0 | `unique_ratio > 0.50` 时扣分（可能是随机 ID） |
| **高空值惩罚** | ×0.1 | null 率 > 50% 时分数严重降低 |
关键词集（`ENTITY_KEYWORDS`）：
```
src_ip, dst_ip, source_ip, destination_ip, ip,
sni, server_name, host, domain, user, client,
username, hostname, mac, email, session_id,
uid, id, src_addr, dst_addr
```
### 5.2 多列复合键
**支持场景**：当需要按 `(src_ip, dst_ip)` 或 `(src_ip, sni)` 等多列组合作为实体标识时。
**MCP 工具参数**：
```python
build_entity_profiles(
dataset_id="ds_...",
entity_columns=["src_ip", "dst_ip"], # 多列复合键
auto_detect=false
)
```
**CLI 管道**目前仅支持单列实体键。
### 5.3 IP 掩码聚类
在聚合阶段中，对 `DataType.IPv4` 类型的列自动生成 `/24` 子网信息：
```python
# entity_aggregator.py 中：
subnet_expr = (
pl.col(ip_col).str.split('.').list.slice(0, 3).list.join('.') + pl.lit('.0/24')
)
# → "192.168.1.0/24"
```
生成的 `unique_24_networks` 在聚类时作为特征输入，反映实体的网络分布广度。
---
## 6. 类型感知聚合
### 6.1 聚合原理
**主函数**：`analysis/entity_aggregator.py::aggregate_by_entity()`
1. 扫描 schema 通过关键词匹配找到可用列
2. 通过 `type_classifier` 获取每列的 `DataType` 分类
3. 按数据类型选择对应的聚合函数
### 6.2 数据类型→聚合函数映射
| `DataType` | 聚合方式 | 特征命名 | 代码位置 |
|-----------|---------|---------|---------|
| **INT** (整数) | `mean()` 均值 | `total_bytes_sent`, `total_packets` | `build_aggregation_params()` |
| **FLOAT** (浮点) | `mean()` 均值 | `avg_duration`, `avg_latitude` | `build_aggregation_params()` |
| **ENUM** (枚举) | `mode().first()` 众数 | `tls_version_mode`, `tcp_flag_modes` | `build_aggregation_params()` |
| **HEX** (十六进制) | `str.extract_all().list.len().mean()` 统计 hex 对数量 | `avg_hex_pairs_count` | `build_aggregation_params()` |
| **URL** (链接) | `str.extract(r'https?://([^/]+)').n_unique()` 统计唯一域名 | `unique_domains` | `build_aggregation_params()` |
| **IPv4** (地址) | `str.split('.').list.slice(0,3).join('.') + '.0/24'.n_unique()` 子网多样性 | `unique_24_networks` | `build_aggregation_params()` |
### 6.3 聚合特征分类
所有生成的聚合特征分为 6 大类：
| 类别 | 特征 | 说明 |
|------|------|------|
| **流统计** | `flow_count`、`total_bytes_sent`、`total_bytes_rev`、`avg_duration`、`total_packets` | 实体流量基本量 |
| **目标多样性** | `unique_dst_ips`、`unique_dst_ports`、`unique_dst_networks` | 目的地址分布广度 |
| **TLS 特征** | `unique_ciphers`、`tls_version_mode`、`unique_versions`、`sni_count` | 加密和证书多样性 |
| **协议特征** | `unique_protocols`、`tcp_flag_modes` | 协议和标志分布 |
| **地理位置** | `avg_latitude`、`avg_longitude` | 经纬度均值 |
| **时间模式** | `first_seen`、`last_seen`、`active_hours_count` | 活动时间段 |
### 6.4 类型安全防范
build_aggregation_params 在聚合前检查列的实际 Polars dtype：
```python
# 在聚合非数值列前 cast
if schema.get(col_name) in ('Utf8', 'String'):
pl.col(col_name).cast(pl.Float64)
```
这解决了跨机"mean on str"错误（当同一数据在不同机器上被 Polars 推断为不同 dtypes 时）。
---
## 7. 聚类分析
### 7.1 算法选择
| 算法 | 适用场景 | 优点 | 缺点 |
|------|---------|------|------|
| **HDBSCAN**（默认） | 未知簇数、存在噪声点 | 自动确定簇数；噪声点处理；无需预设 k | 对 `min_cluster_size` 敏感；大数据集计算慢 |
| **KMeans** | 已知簇数、球形分布 | 速度快；可预设 k | 需指定 n_clusters；对离群点敏感 |
### 7.2 参数建议
**HDBSCAN 参数**：
| 参数 | 默认值 | 说明 | 调整建议 |
|------|--------|------|---------|
| `min_cluster_size` | 5 | 最小簇样本数 | 数据量大时增大（10~20）；数据少时自动降级为 `max(len//2, 2)` |
| `min_samples` | None | 核心点邻域 | 默认与 min_cluster_size 相同；增大产生更保守的聚类 |
| `metric` | euclidean | 距离度量 | 高维特征时考虑 `cosine` |
| `cluster_selection_epsilon` | 0.0 | 簇选择阈值 | 越大簇越少 |
**KMeans 参数**：
| 参数 | 默认值 | 说明 |
|------|--------|------|
| `n_clusters` | 3 | 簇数 |
| `random_state` | 42 | 随机种子（保证可复现性） |
### 7.3 零样本降级
当样本数过少或全为缺失值时，系统自动降级：
```python
if len(data) < 3:
return {"n_clusters": 0, "n_noise": 0, "quality_metrics": {"note": "too few samples"}}
```
同样，全部为 NaN 的列在聚类前自动丢弃。
### 7.4 预处理步骤
聚类执行前的自动预处理流程：
1. **列过滤**：只保留请求的特征列
2. **全 NaN 列丢弃**：无用列自动移除
3. **NaN 填充**：数值 NaN 用该列均值填充
4. **StandardScaler**：所有特征标准化（零均值、单位方差）
5. **min_cluster_size 自适应**：不超过样本数的 50%
### 7.5 质量评估
| 指标 | 范围 | 说明 |
|------|------|------|
| **Silhouette Score** | `[-1, 1]` | >0.5 表示良好聚类；>0.7 表示优秀 |
| **Davies-Bouldin Index** | `[0, +∞)` | 越低越好（簇内紧凑、簇间分离） |
| **Calinski-Harabasz Score** | `[0, +∞)` | 越高越好 |
| **Noise Ratio** | `[0, 1]` | HDBSCAN 特有的噪声点比例 |
### 7.6 CLI 运行示例
```bash
# 完整管道（HDBSCAN 默认）
runtime\python\python.exe manage.py run_pipeline data/complex_test.csv
# 指定 KMeans
runtime\python\python.exe manage.py run_pipeline data/*.csv --algo kmeans
# 手动指定实体列
runtime\python\python.exe manage.py run_pipeline data/*.csv --entity-col src_ip
```
---
## 8. 子网实体聚合
config.yaml中配置:
```yaml
entity:
subnet_masks: [24, 28] # 子网掩码列表
ip_columns: ["src_ip", "dst_ip"] # 需要聚合的IP列
```
启用后，同一子网（如 10.0.1.0/24）下的所有 IP 合并为一个实体节点。
支持 /24（0-255）和 /28（240-255, 16个IP分组）两种掩码。
`entity_aggregator.ip_to_subnet()` 负责 IP 到 CIDR 子网前缀的转换，
`apply_subnet_aggregation()` 在 group_by 前对 IP 列应用子网掩码。
```python
# 示例: IP → /24 子网
ip_to_subnet("192.168.1.42", 24) # → "192.168.1.0/24"
# 示例: IP → /28 子网
ip_to_subnet("192.168.1.42", 28) # → "192.168.1.32/28"
```
---
## 8. 特征提取
### 8.1 方法对比
**主函数**：`analysis/tool_registry.py` → `_handle_extract_features()`
| 方法 | 公式 | 适用场景 |
|------|------|---------|
| **Z-Score**（默认） | `(μ_cluster - μ_global) / σ_global` | 簇中心 vs 全局均值的偏离程度 |
| **ANOVA** | `|μ_cluster - μ_global| / (σ_cluster + σ_global)` | 考虑簇内方差，更保守 |
### 8.2 区分特征解读
返回的每个特征包含：
```json
{
"feature_name": "total_bytes_sent",
"cluster_label": 0,
"mean": 1234567.89, // 该簇均值
"std": 234567.89, // 该簇标准差
"global_mean": 987654.32, // 全局均值
"global_std": 765432.10, // 全局标准差
"distinguishing_score": 2.34, // Z-Score 或 ANOVA 分数
"distinguishing_method": "zscore"
}
```
**解读方法**：
| 分数范围 | 含义 |
|---------|------|
| \|Z\| < 1.0 | 无明显区分力 |
| 1.0 ≤ \|Z\| < 2.0 | 中等区分力 |
| \|Z\| ≥ 2.0 | **强区分特征**（在 Web 界面中以颜色高亮） |
### 8.3 地理分布可视化
如果聚合特征包含 `avg_latitude` 和 `avg_longitude`，聚类概览页会同时显示：
- **PCA 散点图**：实体在二维主成分空间的分布，颜色区分聚类
- **地理分布图**：实体在地理坐标上的分布
地理位置在 Web 界面中通过 Canvas 2D API 原生绘制（无外部 JS 依赖）：
```python
# views.py 中提取逻辑
lat = _extract_lat(feature_json.get('avg_latitude'))
lon = _extract_lon(feature_json.get('avg_longitude'))
# 过滤无效值：纬度 -90~90，经度 -180~180
```
---
## 9. 可视化
### 9.1 PCA 散点图
路径: `/clusters/<run_id>/`
使用纯 Canvas 2D 绘制的散点图，展示实体在二维主成分空间中的分布。
点颜色按聚类标签区分，鼠标悬停显示实体名称。
依赖: `run_pipeline` 步骤 6 自动计算 PCA-2D 坐标并写入 `EntityProfile` 表。
### 9.2 3D 地球流量弧线
路径: `/globe/`
使用 Three.js (603KB, 全离线) 绘制的三维地球，显示源 IP 到目标 IP 的 TLS 流量弧线。
- 颜色: TLSv1.3 (蓝) / TLSv1.2 (粉) / 其他 (青)
- 弧线高度根据距离自动调整
- 鼠标拖拽旋转地球
- 地球贴图: 1.4MB (离线内嵌)
- **多数据源叠加**: 页面复选框选择多个运行，弧线叠加在同一地球上
- **经纬度来源**: 优先使用数据中`src_latitude/src_longitude/dst_latitude/dst_longitude`列，无则回退GeoIP查询
- GeoIP: 从 `data/geoip_city.csv` 查询 IP 对应经纬度
- 滚轮缩放: 鼠标滚轮控制相机距离(3-30)
- 经纬线: 10度间隔的白色半透明辅助线
- 国境线: world_borders.js 10国精简轮廓，含台湾和南海九段线示意
- **背面剔除**: `updatePulses` 中使用 `localToWorld` 替代 `applyQuaternion`，确保弧线脉冲只在面向相机的一侧显示，不穿透地球
- **脉冲动画**: 弧线上移动的贝塞尔曲线脉冲光点，`arcGroup.localToWorld(midPt)` 计算世界坐标，`midPt.dot(camDir) < -0.08` 判段正面可见性
- 多数据选择: 复选框选择多个运行叠加弧线
- 经纬度来源: 优先使用数据列 src_latitude/src_longitude 等，无则GeoIP回退
### 9.3 聚类概览页
路径: `/clusters/<run_id>/`
同时显示 PCA 散点图和每簇的 Top 区分特征。
## 10. LLM 集成
### 9.1 API 配置
通过 `config/config.yaml` 或 Web 界面「配置」页面设置：
```yaml
llm:
enabled: true # 启用 LLM 功能
base_url: "https://api.openai.com/v1" # OpenAI 兼容 API 地址
api_key: "sk-..." # API Key
model: "deepseek-v4-flash" # 模型名称
max_tokens: 2048
temperature: 0.1
```
支持的 LLM 服务：
- OpenAI GPT-4 / GPT-3.5
- Azure OpenAI Service
- 本地部署的 Ollama / vLLM / LM Studio
- 其他 OpenAI 兼容 API
### 9.2 32B 模型优化
系统针对 32B 等较大模型的推理特性做了以下优化（`tianxuan/llm_orchestrator.py`）：
| 优化项 | 实现 | 说明 |
|--------|------|------|
| **Step Guidance** | 每步告诉模型下一步做什么 | 减少 32B 模型在复杂推理中的发散 |
| **工具结果截断** | 截断至 1200 chars | 避免上下文窗口溢出 |
| **超时** | 90s | 32B 推理更慢，比默认超时长 |
| **最大步数** | 15 步 | 限制过长的工具调用链 |
| **工具描述** | 一行简洁描述 | 减少冗长描述对 token 的消耗 |
### 9.3 LLM 分析流程
自动化编排调用链：
```
Step 1: profile_data("ds_...") → 了解数据集的列统计
Step 2: build_entity_profiles("ds_...") → 实体聚合
Step 3: run_clustering("entity_...") → 聚类分析
Step 4: extract_features("entity_...") → 特征提取
Step 5: 总结分析结果
```
### 9.4 日志链解读
LLM 交互日志以 `[LLM]` 和 `[TOOL]` 标记输出到 `logs/tianxuan.log`：
```
[LLM] model=deepseek-v4-flash # 模型名
[LLM] latency_ms=3245 # LLM 响应延迟
[LLM REQUEST] ... # 请求内容（缩减后）
[LLM RESPONSE] ... # 响应状态
[TOOL] profile_data # LLM 调用的工具
[TOOL] build_entity_profiles
[TOOL] run_clustering
[TOOL] extract_features
```
### 9.5 故障排查
| 问题 | 常见原因 | 排查方法 |
|------|---------|---------|
| 400 错误 | API 地址错误 / Key 无效 | 配置页「测试连通性」 |
| 超时 | 32B 推理慢 / 网络延迟 | 检查 `latency_ms`；增大 `max_tokens` |
| 工具调用失败 | LLM 返回格式错误 | 检查 `[TOOL]` 日志看哪个工具调用失败 |
| 分析结果为空 | 实体检测失败 | 检查实体列是否正确，尝试手动指定 |
**连通性测试**：
```bash
# CLI 测试
runtime\python\python.exe scripts\test_llm_full.py
# Web 测试
# 导航到「配置」页面 → 填写 LLM 配置 → 点击「测试连通性」
```
---
## 11. 跨机部署
### 10.1 运行时隔离验证
**脚本**：`scripts/verify_runtime.py`
在不同机器上运行该脚本，输出应完全一致：
```bash
# 每台机器各自执行
runtime\python\python.exe scripts\verify_runtime.py
```
验证内容：
| 检查项 | 期望值 |
|--------|--------|
| Python 版本 | 3.12.x |
| ENABLE_USER_SITE | False（隔离） |
| django | 4.2.x |
| polars | 1.42.x |
| sklearn | 1.5.x |
| numpy | 1.26.x |
| defaultencoding | utf-8 |
| filesystemencoding | utf-8 |
### 10.2 diagnose_compare 使用
**脚本**：`scripts/diagnose_compare.py`
对比两台机器的运行日志，定位第一个差异点：
```bash
# 每台机器运行管道，生成日志
del logs\tianxuan.log
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
copy logs\tianxuan.log log_机器A.txt
# 在开发机上对比
runtime\python\python.exe scripts\diagnose_compare.py log_开发机.txt log_机器A.txt
```
输出示例：
```
[DIFF] 首个差异在行 123:
--- 机器A ---
[LOAD_SCHEMA] col=duration dtype=Utf8 sample=1.2s,0.5s,3.1s
--- 机器B ---
[LOAD_SCHEMA] col=duration dtype=Float64 sample=1.2,0.5,3.1
```
首个 `[DIFF]` 行就是问题根源。
### 10.3 关键日志标记
| 标记 | 对应文件 | 含义 |
|------|---------|------|
| `[LOAD]` | `data_loader.py` | CSV 加载完成 |
| `[LOAD_SCHEMA]` | `data_loader.py` | 每列的推断类型（最重要！） |
| `[CLEAN]` | `data_loader.py` | 清洗操作记录 |
| `[VALIDATE]` | `data_validator.py` | 数据校验结果 |
| `[FIND_COL]` | `entity_aggregator.py` | 关键词匹配结果 |
| `[CLUSTER_INPUT]` | `tool_registry.py` | 聚类输入形状 |
| `[GEOIP]` | `geoip.py` | GeoIP 数据加载 |
| `[LLM]` | `llm_orchestrator.py` | LLM 调用状态 |
| `[TOOL]` | `llm_orchestrator.py` | LLM 调用的工具 |
| `[DB_SAVE_ERROR]` | `tool_registry.py` | 数据库持久化错误 |
### 10.4 跨机一致性检查清单
当两台机器表现不一致时，逐项排查：
```bash
# 1. Python 版本
runtime\python\python.exe --version
# 2. 编码设置
runtime\python\python.exe -c "
import sys; print(f'defaultenc={sys.getdefaultencoding()}')
print(f'fsenc={sys.getfilesystemencoding()}')
print(f'utf8_mode={sys.flags.utf8_mode}')
"
# 3. Polars 版本
runtime\python\python.exe -c "import polars; print(polars.__version__)"
# 4. 依赖版本
runtime\python\python.exe -c "import numpy; print(numpy.__version__)"
runtime\python\python.exe -c "import sklearn; print(sklearn.__version__)"
```
### 10.5 故障诊断流程
```
出现跨机差异
│
├─ Step 1: 收集两台机器的 logs/tianxuan.log
│
├─ Step 2: 运行 diagnose_compare.py 对比日志
│ └─ 找到首个 [DIFF]
│
├─ Step 3: 判断差异类型
│ ├─ [LOAD_SCHEMA] 列类型不同
│ │ └─ 检查 Polars 版本一致、PYTHONUTF8 生效
│ │
│ ├─ [FIND_COL] 关键词匹配不同
│ │ └─ 确认 frozenset→tuple 修复已应用（消除哈希随机性）
│ │
│ └─ [CLUSTER_INPUT] 聚类结果不同
│ └─ 检查 random_state=42、StandardScaler 一致性
│
├─ Step 4: 应急修复
│ ├─ 手动指定实体列：--entity-col src_ip
│ ├─ 强制编码：set PYTHONUTF8=1
│ └─ 使用 tuple 替代 frozenset（已在 entity_aggregator.py 中修复）
│
└─ Step 5: 归档诊断信息
( echo === 天璇故障诊断报告 ===
date /t & time /t
runtime\python\python.exe --version
type logs\tianxuan.log
) > 诊断报告_%COMPUTERNAME%.txt
```
### 10.6 已知跨机问题（已修复）
| 问题 | 根因 | 修复 |
|------|------|------|
| "mean on str" 错误 | frozenset 哈希随机性 | frozenset → tuple 确定性迭代 |
| "ZZZ不在{XXX,YYY}中" 错误 | Python 编码不一致 | `PYTHONUTF8=1` 强制 UTF-8 |
| DB 特征不保存 | Django ORM 在 async 上下文中的限制 | `sync_to_async` 包装 |
| 图表不显示 | Chart.js 从未下载 | 改用纯 Canvas 2D API |
---
> **更多技术细节**：请参阅 `docs/故障诊断手册.md` 获取跨机问题深度排查指南。
docs/故障诊断手册.md
+282
View File
@@ -0,0 +1,282 @@
# 天璇跨机故障诊断手册
## 问题描述
同一份项目压缩包、同一份 CSV 数据（SHA256 一致）、同一份 `runtime/python/` 运行时，
在不同 Windows 10 x64 机器上出现不同行为：
| 设备 | 表现 |
|------|------|
| 你的机器 | ✅ 正常运行 |
| 第二台 | ❌ "尝试对str类型求mean" |
| 第三台 | ❌ "ZZZ不在{XXX,YYY}中" + 字符串相关错误 |
| (可能的更多) | ❌ 其他奇怪错误 |
---
## 1. 立即诊断流程
### 步骤 A：收集两台机器的日志
在两台**问题机器**上各执行一次：
```bash
# 1. 清理旧日志
del logs\tianxuan.log 2>nul
# 2. 生成测试数据（确保 seed 一致）
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000 --output data\complex_test.csv
# 3. 运行完整管道
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv --algo hdbscan
# 4. 复制日志到桌面备用
copy logs\tianxuan.log %USERPROFILE%\Desktop\log_机器A.txt
```
在**你的机器**上也执行一次，生成 `log_你的机器.txt`。
### 步骤 B：对比日志
```bash
runtime\python\python.exe scripts\diagnose_compare.py log_你的机器.txt log_机器A.txt
```
输出示例：
```
[DIFF] 首个差异在行 123:
--- 机器A ---
[LOAD_SCHEMA] col=duration dtype=Utf8 sample=1.2s,0.5s,3.1s
--- 机器B ---
[LOAD_SCHEMA] col=duration dtype=Float64 sample=1.2,0.5,3.1
```
第一个 `[DIFF]` 就是导致问题的根源。
---
## 2. 关键日志标记解读
日志中所有 `[TAG]` 标记的含义：
| 标记 | 对应文件 | 含义 |
|------|---------|------|
| `[LOAD]` | `data_loader.py` | CSV 加载完成，文件数和预估行数 |
| `[LOAD_SCHEMA]` | `data_loader.py` | 每列的推断类型（最重要！） |
| `[FIND_COL]` | `entity_aggregator.py` | 关键词匹配结果，显示匹配到的列名和类型 |
| `[AGGR_EXPR]` | `entity_aggregator.py` | 聚合表达式 |
| `[CLUSTER_INPUT]` | `tool_registry.py` | 聚类输入数据形状 |
| `[LLM REQUEST]` | `llm_orchestrator.py` | LLM 请求内容 |
| `[LLM RESPONSE]` | `llm_orchestrator.py` | LLM 响应状态 |
| `[TOOL CALL]` | `llm_orchestrator.py` | LLM 调用的工具 |
| `[TOOL RESULT]` | `llm_orchestrator.py` | 工具执行结果 |
---
## 3. "尝试对str类型求mean" 故障排查
### 根因树
```
"尝试对str类型求mean" 报错
├── 调用链: build_aggregation_params → pl.mean(col_name)
│
├── [可能A] 列名关键词匹配到了错误的列
│ ├── 检查 [FIND_COL] keywords=duration matched=xxx
│ ├── 如果 matched 列的类型是 Utf8/String → 匹配错误
│ └── 修复: entity_aggregator.py 中 frozenset→tuple (已改)
│
├── [可能B] CSV 列的实际类型与推断类型不同
│ ├── 检查 [LOAD_SCHEMA] col=duration dtype=xxx
│ ├── 对比两台机器的 dtype 是否一致
│ ├── 如果一致但行为不同 → 可能是 Polars 版本差异
│ └── 验证: runtime\python\python.exe -c "import polars; print(polars.__version__)"
│
├── [可能C] PYTHONUTF8 环境变量未生效
│ ├── 验证: runtime\python\python.exe -c "import sys; print(sys.getdefaultencoding())"
│ ├── 期望输出: utf-8
│ ├── 如果输出 cp936/gbk → PYTHONUTF8 没生效
│ └── 修复: run.bat 中 set PYTHONUTF8=1 (已加)
│
└── [可能D] 聚合表达式中混入了 Utf8 列
├── 检查 [FIND_COL] keywords=duration matched=duration dtype=Utf8
├── 这意味着 duration 列被 Polars 推断为字符串
└── 修复: entity_aggregator.py 的 dtype 安全聚合 (已改)
```
### 诊断命令
```bash
# 1. 检查 Python 默认编码
runtime\python\python.exe -c "import sys; print(f'enc={sys.getdefaultencoding()} fs={sys.getfilesystemencoding()}')"
# 2. 检查 Polars 版本
runtime\python\python.exe -c "import polars; print(polars.__version__)"
# 3. 检查 CSV 加载后的实际列类型
runtime\python\python.exe -c "
import polars as pl
lf = pl.scan_csv(r'data\complex_test.csv', infer_schema_length=10000)
schema = lf.collect_schema()
for name, dtype in zip(schema.names(), schema.dtypes()):
print(f'{name}: {dtype}')
"
# 4. 检查关键词匹配结果
runtime\python\python.exe -c "
import sys; sys.path.insert(0, '.')
import os; os.environ['DJANGO_SETTINGS_MODULE']='tianxuan.settings'
import django; django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_aggregator import _find_column, _BUILT_AGGR
store = SessionStore(); store.drop_all()
lf, schema, rc, fc, mem = load_csv_directory(r'data\complex_test.csv')
for kw_name, kw_set in [
('bytes_sent',_BYTES_SENT_KEYWORDS),('duration',_DURATION_KEYWORDS),
('dst_ip',_DST_IP_KEYWORDS),('cipher',_CIPHER_KEYWORDS)]:
col = _find_column(schema, kw_set)
dtype = schema.get(col, 'N/A') if col else 'N/A'
print(f'{kw_name:15s} -> {str(col):25s} dtype={dtype}')
" 2>&1
```
---
## 4. "ZZZ不在{XXX,YYY}中" 故障排查
### 根因
这个错误来自 Polars 的 `str.contains()` 或 `is_in()` 操作。当比较字符串时，
两边的编码不一致导致匹配失败。
### 诊断
```bash
# 检查 Python 编码
runtime\python\python.exe -c "
import sys; print(f'default={sys.getdefaultencoding()}')
print(f'filesystem={sys.getfilesystemencoding()}')
print(f'stdin={sys.stdin.encoding}')
print(f'stdout={sys.stdout.encoding}')
"
# 检查 CSV 文件的实际编码（在 Notepad++ 或 VS Code 中查看右下角编码提示）
# 期望: UTF-8 (无 BOM)
```
### 修复尝试
```bash
# 修复 1: 确认 run.bat 包含 PYTHONUTF8=1
findstr PYTHONUTF8 run.bat
# 应输出: set PYTHONUTF8=1
# 修复 2: 手动设置编码后再运行
set PYTHONUTF8=1
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
```
---
## 5. 跨机环境一致性检查清单
在**每台机器**上执行以下命令，对比输出：
```bash
echo === 1. Python 版本 ===
runtime\python\python.exe --version
echo === 2. 编码设置 ===
runtime\python\python.exe -c "
import sys; print(f'defaultenc={sys.getdefaultencoding()}')
print(f'fsenc={sys.getfilesystemencoding()}')
print(f'has_utf8_mode={sys.flags.utf8_mode}')
"
echo === 3. Polars 版本 ===
runtime\python\python.exe -c "import polars; print(polars.__version__)"
echo === 4. numpy 版本 ===
runtime\python\python.exe -c "import numpy; print(numpy.__version__)"
echo === 5. sklearn 版本 ===
runtime\python\python.exe -c "import sklearn; print(sklearn.__version__)"
echo === 6. 系统区域 ===
systeminfo | findstr "区域"
```
---
## 6. 应急修复方案
如果 `diagnose_compare.py` 显示的第一个差异是 `[LOAD_SCHEMA]` 列类型不同：
```bash
# 强制所有列读为字符串 → 让 Polars 自己推断
runtime\python\python.exe -c "
import polars as pl
lf = pl.scan_csv(r'data\complex_test.csv', infer_schema_length=100000)
schema = lf.collect_schema()
# 打印所有列的推断类型
for n, d in zip(schema.names(), schema.dtypes()):
print(f'{n}: {d}')
"
```
如果差异在 `[FIND_COL]` 关键词匹配不同：
```bash
# 手动指定实体列后运行
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv --entity-col src_ip
```
如果完全无法解决：
```bash
# 重建运行时（在开发机上有网络的机器上）
# 1. 删除旧的 runtime
rmdir /s runtime
# 2. 重新运行构建脚本
scripts\build_runtime.bat
# 3. 重新压缩分发
```
---
## 7. 已知已修复的问题清单
| 问题编号 | 修复内容 | 涉及文件 | 验证方式 |
|---------|---------|---------|---------|
| W4-#1 | `frozenset`→`tuple` 消除哈希随机性 | `entity_aggregator.py` | 查看文件确认无 `frozenset` |
| W4-#2 | `PYTHONUTF8=1` 强制 UTF-8 编码 | `run.bat` | `findstr PYTHONUTF8 run.bat` |
| W4-#3 | `sorted(keywords)` 确定性关键词迭代 | `entity_aggregator.py` | 查看 `_find_column` 函数 |
| W4-#4 | 结构化 `[TAG]` 日志 | 多文件 | 运行后查看 `logs/tianxuan.log` |
| DB-#1 | `sync_to_async` 解决异步 ORM 问题 | `tool_registry.py` | 管道完成后特征写入 DB |
| DB-#2 | `value_counts()` 列名兼容 | `data_profiler.py` | `profile_data` 工具正常返回 |
| DB-#3 | numpy corrcoef 替代 pandas | `data_profiler.py` | 无 pandas 时也能跑 |
---
## 8. 归档与提交诊断结果
```bash
# 收集诊断信息到单个文件，发给开发者
(
echo === 天璇故障诊断报告 ===
date /t
time /t
echo.
echo === 环境信息 ===
runtime\python\python.exe --version
runtime\python\python.exe -c "import sys; print(f'enc={sys.getdefaultencoding()}')"
runtime\python\python.exe -c "import polars; print(f'polars={polars.__version__}')"
echo.
echo === 日志最后 50 行 ===
type logs\tianxuan.log 2>nul
) > 诊断报告.txt
```
logs/clustering_eval_report.json
+139
View File
@@ -0,0 +1,139 @@
{
"run_info": {
"dataset": "data/wnl_converted.csv",
"total_rows": 3547,
"n_entities": 834,
"n_clusters": 6
},
"quality_metrics": {
"silhouette": 0.7332,
"davies_bouldin": 7.6658,
"calinski_harabasz": 792.0156,
"noise_ratio": 0.1355,
"purity": 0.2434,
"ari": 0.0001,
"nmi": 0.0236
},
"cluster_sizes": {
"-1": 113,
"0": 80,
"1": 259,
"2": 160,
"3": 82,
"4": 43,
"5": 97
},
"cluster_service_map": {
"-1": {
"SoundCloud": 27,
"Web": 25,
"Kinopoisk": 24,
"Spotify": 17,
"YandexMusic": 10,
"Netflix": 6,
"AppleMusic": 2,
"YouTube": 1,
"PrimeVideo": 1
},
"0": {
"Spotify": 21,
"YandexMusic": 19,
"SoundCloud": 17,
"Kinopoisk": 10,
"Web": 9,
"Netflix": 3,
"YouTube": 1
},
"1": {
"Web": 61,
"Spotify": 51,
"Kinopoisk": 44,
"SoundCloud": 44,
"YandexMusic": 37,
"Netflix": 11,
"AppleMusic": 5,
"YouTube": 4,
"PrimeVideo": 2
},
"2": {
"SoundCloud": 39,
"YandexMusic": 34,
"Web": 33,
"Spotify": 27,
"Kinopoisk": 13,
"Netflix": 9,
"PrimeVideo": 2,
"YouTube": 1,
"LiveYouTube": 1,
"AppleMusic": 1
},
"3": {
"Spotify": 19,
"Web": 18,
"SoundCloud": 18,
"YandexMusic": 11,
"Kinopoisk": 8,
"Netflix": 7,
"AppleMusic": 1
},
"4": {
"Web": 12,
"Spotify": 11,
"SoundCloud": 10,
"Kinopoisk": 4,
"YandexMusic": 2,
"Netflix": 2,
"PrimeVideo": 1,
"LiveYouTube": 1
},
"5": {
"Web": 24,
"SoundCloud": 23,
"Spotify": 19,
"Kinopoisk": 12,
"YandexMusic": 9,
"Netflix": 5,
"YouTube": 2,
"AppleMusic": 1,
"LiveFacebook": 1,
"PrimeVideo": 1
}
},
"purity_breakdown": {
"-1": {
"dominant_label": "SoundCloud",
"dominant_count": 27,
"size": 113
},
"0": {
"dominant_label": "Spotify",
"dominant_count": 21,
"size": 80
},
"1": {
"dominant_label": "Web",
"dominant_count": 61,
"size": 259
},
"2": {
"dominant_label": "SoundCloud",
"dominant_count": 39,
"size": 160
},
"3": {
"dominant_label": "Spotify",
"dominant_count": 19,
"size": 82
},
"4": {
"dominant_label": "Web",
"dominant_count": 12,
"size": 43
},
"5": {
"dominant_label": "Web",
"dominant_count": 24,
"size": 97
}
}
}
logs/tianxuan.log
+8338
View File
File diff suppressed because it is too large Load Diff
main.py
+6
View File
@@ -0,0 +1,6 @@
def main():
print("Hello from tls-analyzer!")
if __name__ == "__main__":
main()
manage.py
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
pyproject.toml
+31
View File
@@ -0,0 +1,31 @@
[project]
name = "tls-analyzer"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"django==4.2.30",
"mcp==1.28.1",
"polars==1.42.1",
"scikit-learn==1.5.2",
"numpy==1.26.4",
"pyarrow==25.0.0",
"pyyaml==6.0.3",
"pydantic==2.13.4",
]
[project.optional-dependencies]
dev = [
"ruff>=0.4.0",
"mypy>=1.10.0",
"pytest>=8.0.0",
"pytest-mock>=3.12.0",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
filterwarnings = [
"ignore::DeprecationWarning",
]
requirements.txt
+12
View File
@@ -0,0 +1,12 @@
# 天璇 (TianXuan) - Locked Runtime Dependencies
# Generated from runtime Python 3.12.2
# Use: runtime\python\python.exe -m pip install -r requirements.txt
django==4.2.30
mcp==1.28.1
polars==1.42.1
scikit-learn==1.5.2
numpy==1.26.4
pyarrow==25.0.0
pyyaml==6.0.3
pydantic==2.13.4
run.bat
+8
View File
@@ -0,0 +1,8 @@
@echo off
set PYTHONUTF8=1
cd /d "%~dp0"
echo Starting TianXuan...
start /B runtime\python\python.exe manage.py runserver 127.0.0.1:8000 --noreload
timeout /t 3 >nul
start http://127.0.0.1:8000/
timeout /t 5 >nul
scripts/column_survey.py
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""Analyze CSV columns and output a concise summary.
Usage:
runtime/python/python.exe scripts/column_survey.py --csv "data/*.csv"
"""
from __future__ import annotations
import argparse
import glob
import sys
from collections import OrderedDict
import polars as pl
def _analyze_columns(df: pl.DataFrame) -> list[OrderedDict]:
"""Return per-column stats: name, dtype, unique_count, null_pct, samples."""
rows: list[OrderedDict] = []
for col in df.columns:
series = df[col]
total = len(series)
null_cnt = series.null_count()
null_pct = (null_cnt / total * 100) if total > 0 else 0.0
unique_cnt = series.n_unique()
non_null = series.drop_nulls()
samples = non_null[:5].to_list() if len(non_null) > 0 else []
rows.append(OrderedDict([
("name", col),
("dtype", series.dtype),
("unique", unique_cnt),
("null_pct", null_pct),
("samples", samples),
("total_non_null", len(non_null)),
]))
return rows
def _fmt_samples(samples: list, total_non_null: int) -> str:
items = ", ".join(repr(s) for s in samples)
if 0 < len(samples) < total_non_null:
items += ", ..."
return f"[{items}]"
def main() -> None:
parser = argparse.ArgumentParser(description="Survey CSV columns")
parser.add_argument("--csv", required=True, help="Glob pattern for CSV files")
args = parser.parse_args()
files = sorted(glob.glob(args.csv, recursive=True))
if not files:
print(f"No files match pattern: {args.csv}", file=sys.stderr)
sys.exit(1)
# Load each file independently, group by schema
schema_groups: OrderedDict[str, list[pl.DataFrame]] = OrderedDict()
for f in files:
try:
lf = pl.scan_csv(f).head(5000)
df = lf.collect()
if df.is_empty():
print(f"Skip empty: {f}", file=sys.stderr)
continue
# Use schema string as grouping key
key = str(sorted(df.schema.items()))
schema_groups.setdefault(key, []).append(df)
except Exception as e:
print(f"Skip {f}: {e}", file=sys.stderr)
if not schema_groups:
print("No readable CSV data", file=sys.stderr)
sys.exit(1)
# Report each schema group
for idx, (schema_key, dfs) in enumerate(schema_groups.items()):
merged = pl.concat(dfs)
stats = _analyze_columns(merged)
if len(schema_groups) > 1:
group_label = f" (group {idx + 1}, {len(dfs)} file(s))"
else:
group_label = f" ({sum(len(v) for v in schema_groups.values())} file(s))"
print(f"=== Column Survey{group_label} ===")
for s in stats:
print(
f"{s['name']}: {s['dtype']}, "
f"unique={s['unique']}, "
f"null={s['null_pct']:.1f}%, "
f"sample={_fmt_samples(s['samples'], s['total_non_null'])}"
)
if __name__ == "__main__":
main()
scripts/debug_db.py
+55
View File
@@ -0,0 +1,55 @@
"""Debug database persistence"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django
django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_aggregator import aggregate_by_entity
from analysis.entity_detector import detect_entity_column
import asyncio
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features, _save_features_to_db
store = SessionStore()
store.drop_all()
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
entity_col = detect_entity_column('ds')['recommended']
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect(streaming=True)
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
metadata={'row_count': len(df_agg), 'csv_glob': csv_path})
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity',
cluster_columns=[c for c in features if not c.startswith('_')][:8],
algorithm='hdbscan', params={}, random_state=42))
cid = clust_result['cluster_result_id']
feat_result = asyncio.run(_handle_extract_features(
dataset_id='entity', cluster_result_id=cid,
top_k=10, method='zscore', save_to_db=True
))
feats = feat_result.get('features', [])
print(f'n_features={feat_result.get("n_features")}')
labels = set(f.get('cluster_label') for f in feats)
print(f'unique_labels={labels}')
if feats:
print(f'sample feature: {feats[0]}')
print(f'sample keys: {list(feats[0].keys())}')
print(f'db_saved={feat_result.get("db_saved")}')
from analysis.models import ClusterResult, ClusterFeature, EntityProfile
print(f'ClusterResult: {ClusterResult.objects.count()}')
for cr in ClusterResult.objects.all():
print(f' label={cr.cluster_label} size={cr.size} feats={cr.features.count()}')
print(f'ClusterFeature: {ClusterFeature.objects.count()}')
print(f'EntityProfile: {EntityProfile.objects.count()}')
scripts/debug_db2.py
+66
View File
@@ -0,0 +1,66 @@
"""Debug: test _save_features_to_db step by step"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_aggregator import aggregate_by_entity
from analysis.entity_detector import detect_entity_column
import asyncio
from analysis.tool_registry import _handle_run_clustering
from analysis import tool_registry
store = SessionStore()
store.drop_all()
print('1. Store cleared')
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
entity_col = detect_entity_column('ds')['recommended']
agg_lf, features_list = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect()
entity_schema = dict(zip(features_list, ['']*len(features_list)))
store.store_dataset('entity_data', agg_lf, schema=entity_schema, metadata={'row_count': len(df_agg), 'csv_glob': csv_path})
print('2. Dataset stored')
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
cluster_columns=[c for c in features_list if not c.startswith('_')][:8],
algorithm='hdbscan', params={}, random_state=42))
cid = clust_result['cluster_result_id']
print(f'3. Clustering done: {cid}, n_clusters={clust_result.get("n_clusters")}')
# Manual check
ce = store.get_cluster_result(cid)
if ce:
pid = ce.get('parent_dataset_id', '')
print(f'4. Cluster entry: parent_dataset_id={pid}')
de = store.get_dataset(pid)
if de:
cg = de.get('metadata', {}).get('csv_glob', 'MISSING')
print(f'5. Dataset entry exists, csv_glob={cg}')
else:
print('5. FAIL: dataset entry not found!')
# List all datasets
print(f' Available datasets: {store.list_datasets()}')
else:
print('4. FAIL: cluster entry not found!')
# Direct test
print('6. Test features...')
from analysis.models import AnalysisRun
runs_before = AnalysisRun.objects.count()
print(f' Runs before: {runs_before}')
result = tool_registry._save_features_to_db(cid, [{
'feature_name': 'test', 'cluster_label': 0,
'mean': 1.0, 'std': 0.5, 'global_mean': 0.5, 'global_std': 0.3,
'distinguishing_score': 2.0, 'distinguishing_method': 'zscore'
}])
print(f' Result: {result}')
print(f' Runs after: {AnalysisRun.objects.count()}')
from analysis.models import ClusterResult, ClusterFeature
print(f' ClusterResult: {ClusterResult.objects.count()}')
print(f' ClusterFeature: {ClusterFeature.objects.count()}')
scripts/debug_db3.py
+25
View File
@@ -0,0 +1,25 @@
"""Debug: test _save_features_to_db with existing store data"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from analysis.session_store import SessionStore
from analysis.tool_registry import _save_features_to_db
store = SessionStore()
print(f'Store entries: {len(store._stores)}')
for k, v in list(store._stores.items())[:5]:
entry_type = v.get('type')
if entry_type == 'cluster_result':
cid = k
print(f' cluster_result: {cid}, n_clusters={v.get("n_clusters")}')
print(f' parent_dataset_id={v.get("parent_dataset_id")}')
# Try saving
result = _save_features_to_db(cid, [{
'feature_name': 'test_feature', 'cluster_label': 0,
'mean': 1.0, 'std': 0.5, 'global_mean': 0.5, 'global_std': 0.3,
'distinguishing_score': 2.0, 'distinguishing_method': 'zscore'
}])
print(f' _save_features_to_db returned: {result}')
scripts/debug_persistence.py
+68
View File
@@ -0,0 +1,68 @@
"""
Debug: test pipeline data persistence
"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django
django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_aggregator import aggregate_by_entity
from analysis.entity_detector import detect_entity_column
import asyncio
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
store = SessionStore()
store.drop_all()
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
# Step 1: Load
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
print(f'1. Loaded {rc} rows')
# Step 2: Detect entity
result = detect_entity_column('ds')
entity_col = result['recommended']
print(f'2. Entity col: {entity_col}')
# Step 3: Aggregate
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect(streaming=True)
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))), metadata={'row_count': len(df_agg)})
print(f'3. Aggregated {len(df_agg)} entities')
# Step 4: Cluster
feature_cols = [c for c in features if not c.startswith('_')][:8]
clust_result = asyncio.run(_handle_run_clustering(
dataset_id='entity', cluster_columns=feature_cols,
algorithm='hdbscan', params={}, random_state=42
))
print(f'4. Clustering: n_clusters={clust_result.get("n_clusters")}')
if 'error' in clust_result:
print(f' ERROR: {clust_result["error"]}')
sys.exit(1)
cid = clust_result.get('cluster_result_id', '')
print(f' cluster_result_id={cid}')
entry = store.get_cluster_result(cid)
print(f' store entry exists={entry is not None}')
if entry:
print(f' n_clusters in store={entry.get("n_clusters")}')
# Step 5: Extract features + save to DB
feat_result = asyncio.run(_handle_extract_features(
dataset_id='entity', cluster_result_id=cid,
top_k=10, method='zscore', save_to_db=True
))
print(f'5. Features: n={feat_result.get("n_features")} db_saved={feat_result.get("db_saved")}')
if 'error' in feat_result:
print(f' ERROR: {feat_result["error"]}')
# Step 6: Check DB
from analysis.models import AnalysisRun, ClusterResult, ClusterFeature, EntityProfile
print(f'6. DB: ClusterResult={ClusterResult.objects.count()} ClusterFeature={ClusterFeature.objects.count()} EntityProfile={EntityProfile.objects.count()}')
scripts/diagnose_compare.py
+56
View File
@@ -0,0 +1,56 @@
"""
对比两台机器的日志文件，标记第一个差异点。
用法: runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
"""
import sys, re
# Pattern for log timestamps: [2026-07-14 20:00:00,123]
_TS_PATTERN = re.compile(r'^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]')
def strip_timestamp(line):
"""Remove leading timestamp from a log line for comparison."""
return _TS_PATTERN.sub('', line, count=1).strip()
def extract_key(line):
"""Extract [TAG] key=value from a log line for comparison."""
m = re.match(r'\[(\w+)\]', line)
if m:
return m.group(0) # return the [TAG]
return line[:50]
def main():
if len(sys.argv) < 3:
print('用法: diagnose_compare.py <机器A日志> <机器B日志>')
sys.exit(1)
with open(sys.argv[1], 'r', encoding='utf-8') as f:
lines_a = f.readlines()
with open(sys.argv[2], 'r', encoding='utf-8') as f:
lines_b = f.readlines()
max_lines = min(len(lines_a), len(lines_b))
first_diff = None
for i in range(max_lines):
a_stripped = strip_timestamp(lines_a[i])
b_stripped = strip_timestamp(lines_b[i])
if a_stripped != b_stripped:
if first_diff is None:
first_diff = i
if first_diff is None:
if len(lines_a) != len(lines_b):
print(f'[MATCH] 前 {max_lines} 行一致, 但行数不同 (A={len(lines_a)}, B={len(lines_b)})')
else:
print('[MATCH] 两份日志完全一致')
else:
start = max(0, first_diff - 3)
print(f'[DIFF] 首个差异在行 {first_diff + 1}:')
print('--- 机器A ---')
print(''.join(lines_a[start:first_diff + 3]))
print('--- 机器B ---')
print(''.join(lines_b[start:first_diff + 3]))
if __name__ == '__main__':
main()
scripts/eval_clustering.py
+433
View File
@@ -0,0 +1,433 @@
"""Evaluate clustering quality by comparing against ground-truth labels.
Usage:
set DJANGO_ALLOW_ASYNC_UNSAFE=true
runtime\python\python.exe scripts\eval_clustering.py
"""
import os, sys, json, asyncio, traceback
os.environ['DJANGO_ALLOW_ASYNC_UNSAFE'] = 'true'
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import django; django.setup()
import numpy as np
import polars as pl
from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from collections import Counter, defaultdict
from analysis.models import AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
CSV_PATH = 'data/wnl_converted.csv'
def main():
print("=" * 70)
print("CLUSTERING QUALITY EVALUATION — WNL Converted Dataset")
print("=" * 70)
# ── Step 1: Load CSV ────────────────────────────────────────────────
print("\n[1] Loading CSV...")
lf, schema, row_count, file_count, memory_mb = load_csv_directory(CSV_PATH)
print(f" Rows: {row_count}, Files: {file_count}, Memory: {memory_mb:.1f} MB")
store = SessionStore()
store.drop_all()
ds_id = 'eval_ds'
store.store_dataset(ds_id, lf, schema=schema, metadata={
'row_count': row_count, 'file_count': file_count, 'csv_glob': CSV_PATH,
})
# ── Step 2: Detect entity column ────────────────────────────────────
print("\n[2] Detecting entity column...")
result = detect_entity_column(ds_id)
used_col = result.get('recommended')
candidates = result.get('candidates', [])
print(f" Detected: {used_col} (from {len(candidates)} candidates)")
# ── Step 3: Aggregate by entity ──────────────────────────────────────
print("\n[3] Aggregating by entity...")
agg_lf, feature_columns = aggregate_by_entity(lf, used_col, schema)
df_agg = agg_lf.collect(streaming=True)
entity_ds_id = 'eval_entity'
store.store_dataset(entity_ds_id, agg_lf, schema=dict(zip(feature_columns, ['']*len(feature_columns))),
metadata={'row_count': len(df_agg), 'entity_column': used_col, 'csv_glob': CSV_PATH})
print(f" Entities: {len(df_agg)}, Features: {len(feature_columns)}")
# ── Step 4: Clustering ──────────────────────────────────────────────
print("\n[4] Running clustering (HDBSCAN)...")
user_features = [
c for c in feature_columns
if not c.startswith('_') and c not in ('first_seen', 'last_seen', 'src_ip')
][:10]
print(f" Features: {user_features}")
clust_result = asyncio.run(_handle_run_clustering(
dataset_id=entity_ds_id,
cluster_columns=user_features,
algorithm='hdbscan', params={}, random_state=42,
))
if 'error' in clust_result:
print(f" ERROR: {clust_result['error']}")
return
cluster_id = clust_result.get('cluster_result_id', '')
n_clusters = clust_result.get('n_clusters', 0)
quality = clust_result.get('quality_metrics', {})
print(f" Clusters: {n_clusters}")
print(f" Noise ratio: {quality.get('noise_ratio', 'N/A')}")
print(f" Silhouette: {quality.get('silhouette_score', 'N/A')}")
print(f" Davies-Bouldin: {quality.get('davies_bouldin_score', 'N/A')}")
print(f" CH Score: {quality.get('calinski_harabasz_score', 'N/A')}")
print(f" Cluster sizes: {quality.get('cluster_sizes', {})}")
# ── Step 5: Feature extraction ──────────────────────────────────────
print("\n[5] Extracting distinguishing features...")
feat_result = asyncio.run(_handle_extract_features(
dataset_id=entity_ds_id, cluster_result_id=cluster_id,
top_k=10, method='zscore', save_to_db=True,
))
if 'error' in feat_result:
print(f" Feature extraction error: {feat_result['error']}")
else:
print(f" Features extracted: {feat_result.get('n_features', 0)}")
# ── Get cluster labels from SessionStore ───────────────────────────
print("\n[6] Reading cluster labels from session store...")
cluster_entry = store.get_cluster_result(cluster_id)
if cluster_entry is None:
print(" ERROR: Cluster result not found in SessionStore!")
return
labels_array = np.array(cluster_entry['labels'])
n_entities = len(labels_array)
print(f" Labels array size: {n_entities}")
# ── Map entities to ground-truth labels ────────────────────────────
print("\n[7] Mapping entities to ground-truth labels...")
# The entity column is :ipd (dst IP)
# The original CSV has labels per-flow. After aggregation by :ipd,
# each entity may have multiple flows with different labels.
# We need the majority label per entity.
# Read original data
df_raw = pl.read_csv(CSV_PATH, schema_overrides={'label': pl.Utf8})
# Get entity values from aggregated data
entity_values = df_agg[used_col].to_list()
# Build entity -> label mapping from raw data (majority label per entity)
raw_entities = df_raw[used_col].to_list()
raw_labels = df_raw['label'].to_list()
entity_label_counter = defaultdict(Counter)
for ent, lbl in zip(raw_entities, raw_labels):
if lbl and str(lbl).strip():
entity_label_counter[ent][str(lbl).strip()] += 1
# Get majority label for each entity in the aggregated set
true_labels = []
for ev in entity_values:
if ev in entity_label_counter:
true_labels.append(entity_label_counter[ev].most_common(1)[0][0])
else:
true_labels.append('Unknown')
true_labels = np.array(true_labels)
print(f" Mapped {len(true_labels)} entities to ground-truth labels")
# Label distribution (ground truth among entities)
print("\n Ground truth label distribution among entities:")
gt_counts = Counter(true_labels)
for lbl, cnt in gt_counts.most_common():
print(f" {lbl:>20s}: {cnt:4d} ({cnt/len(true_labels)*100:5.1f}%)")
# ── Purity Score ────────────────────────────────────────────────────
print("\n[8] Computing purity score...")
# For each cluster, find the most common true label
total_correct = 0
cluster_label_map = {}
for label in sorted(set(labels_array)):
mask = labels_array == label
cluster_entities = true_labels[mask]
if len(cluster_entities) == 0:
continue
most_common = Counter(cluster_entities).most_common(1)[0]
cluster_label_map[int(label)] = {
'dominant_label': most_common[0],
'dominant_count': most_common[1],
'size': len(cluster_entities),
}
total_correct += most_common[1]
purity = total_correct / len(labels_array)
print(f" Purity: {purity:.4f} ({total_correct}/{len(labels_array)} correct)")
# ── Adjusted Rand Index ─────────────────────────────────────────────
print("\n[9] Computing Adjusted Rand Index (ARI)...")
# Need numeric labels for ARI - map truth labels to integers
unique_true = sorted(set(true_labels))
true_to_int = {lbl: i for i, lbl in enumerate(unique_true)}
true_numeric = np.array([true_to_int[lbl] for lbl in true_labels])
# Noise points (-1) cause issues for some metrics, filter them
valid_mask = labels_array >= 0
if valid_mask.sum() > 1:
ari = adjusted_rand_score(true_numeric[valid_mask], labels_array[valid_mask])
nmi = normalized_mutual_info_score(true_numeric[valid_mask], labels_array[valid_mask])
print(f" ARI (excluding noise): {ari:.4f}")
print(f" NMI (excluding noise): {nmi:.4f}")
else:
print(" (Not enough non-noise points to compute ARI/NMI)")
ari = float('nan')
# With noise (-1) - some implementations treat -1 as just another label
ari_with_noise = adjusted_rand_score(true_numeric, labels_array)
print(f" ARI (with noise): {ari_with_noise:.4f}")
# ── Detailed per-cluster analysis ───────────────────────────────────
print("\n" + "=" * 70)
print("PER-CLUSTER PROFILES")
print("=" * 70)
# Get feature stats (z-scores) from feature extraction
features_list = feat_result.get('features', [])
# Also compute per-cluster feature means and z-scores from scratch
numeric_cols = [c for c in df_agg.columns
if df_agg[c].dtype in (pl.Float32, pl.Float64, pl.Int32, pl.Int64,
pl.UInt32, pl.UInt64)
and not c.startswith('_')]
# Global stats
global_means = {}
global_stds = {}
for col in numeric_cols:
s = df_agg[col].drop_nulls()
if len(s) > 0:
global_means[col] = float(s.mean())
global_stds[col] = float(s.std()) if s.std() is not None and s.std() > 1e-10 else 1.0
# Build feature index for z-scores
feature_by_cluster = defaultdict(dict)
for f in features_list:
feature_by_cluster[f['cluster_label']][f['feature_name']] = f
# Per cluster report
# First build cross-tab: cluster x service
cluster_service = defaultdict(Counter)
for lbl, true_lbl in zip(labels_array, true_labels):
cluster_service[int(lbl)][true_lbl] += 1
# Sort clusters by size
cluster_sizes = Counter(labels_array)
sorted_clusters = sorted(cluster_sizes.keys())
for label in sorted_clusters:
size = cluster_sizes[label]
pct = size / n_entities * 100
is_noise = label == -1
print(f"\n Cluster {label}: {size} entities ({pct:.1f}%){' [NOISE]' if is_noise else ''}")
# Feature z-scores (top 5)
cluster_feats = feature_by_cluster.get(label, {})
sorted_feats = sorted(cluster_feats.items(),
key=lambda x: abs(x[1].get('distinguishing_score', 0)), reverse=True)
if sorted_feats:
for feat_name, feat_data in sorted_feats[:5]:
score = feat_data.get('distinguishing_score', 0)
mean_val = feat_data.get('mean', 0)
arrow = '+' if score >= 0 else ''
print(f" {feat_name:35s} {mean_val:>10.4f} ({arrow}{score:.2f}σ)")
else:
# Compute z-scores from scratch
label_mask = labels_array == label
for col in numeric_cols[:8]:
cluster_vals = df_agg[col].filter(pl.Series('_mask', label_mask)).drop_nulls()
if len(cluster_vals) > 0 and col in global_means and col in global_stds:
c_mean = float(cluster_vals.mean())
z = (c_mean - global_means[col]) / max(global_stds[col], 1e-10)
arrow = '+' if z >= 0 else ''
print(f" {col:35s} {c_mean:>10.4f} ({arrow}{z:.2f}σ)")
# Dominant services
print(f" {'':─<50s}")
svc_counts = cluster_service.get(label, Counter())
if svc_counts:
total = sum(svc_counts.values())
dominant_services = svc_counts.most_common(5)
svc_str = ', '.join(f"{svc}({cnt/total*100:.0f}%)" for svc, cnt in dominant_services)
print(f" Services: {svc_str}")
else:
print(f" Services: (none)")
# Classification
print(f" {'':─<50s}")
if is_noise:
classification = "NOISE — unassigned by HDBSCAN"
else:
classification = classify_cluster(label, cluster_feats, svc_counts)
print(f" → {classification}")
# ── Summary ──────────────────────────────────────────────────────────
print("\n" + "=" * 70)
print("SUMMARY")
print("=" * 70)
print(f" Total entities: {n_entities}")
print(f" Number of clusters (excl. noise): {n_clusters}")
print(f" Noise count: {cluster_sizes.get(-1, 0)}")
print(f" Noise ratio: {quality.get('noise_ratio', 'N/A')}")
print(f" Silhouette score: {quality.get('silhouette_score', 'N/A')}")
print(f" Purity: {purity:.4f}")
print(f" ARI (excl. noise): {ari:.4f}" if not np.isnan(ari) else " ARI: N/A")
print(f" NMI (excl. noise): {nmi:.4f}" if not np.isnan(ari) else " NMI: N/A")
print()
# Confusion matrix: cluster x service
print(" Confusion Matrix (Cluster × Service):")
all_services = sorted(set(true_labels))
header = f" {'Cluster':>8s}" + ''.join(f"{s:>14s}" for s in all_services[:8])
if len(all_services) > 8:
header += f" ... ({len(all_services)-8} more)"
print(header)
for label in sorted_clusters:
row = cluster_service.get(label, Counter())
row_str = f" {label:>8d}" + ''.join(f"{row.get(s, 0):>14d}" for s in all_services[:8])
if len(all_services) > 8:
extras = sum(row.get(s, 0) for s in all_services[8:])
row_str += f" (+{extras})"
print(row_str)
# ── Save report ─────────────────────────────────────────────────────
report = {
'run_info': {
'dataset': CSV_PATH,
'total_rows': row_count,
'n_entities': n_entities,
'n_clusters': n_clusters,
},
'quality_metrics': {
'silhouette': quality.get('silhouette_score'),
'davies_bouldin': quality.get('davies_bouldin_score'),
'calinski_harabasz': quality.get('calinski_harabasz_score'),
'noise_ratio': quality.get('noise_ratio'),
'purity': round(purity, 4),
'ari': round(ari, 4) if not np.isnan(ari) else None,
'nmi': round(nmi, 4) if not np.isnan(ari) else None,
},
'cluster_sizes': quality.get('cluster_sizes', {}),
'cluster_service_map': {
str(k): dict(v.most_common()) for k, v in sorted(cluster_service.items())
},
'purity_breakdown': cluster_label_map,
}
report_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
'logs', 'clustering_eval_report.json')
os.makedirs(os.path.dirname(report_path), exist_ok=True)
with open(report_path, 'w') as f:
json.dump(report, f, indent=2, default=str)
print(f"\n Report saved to: {report_path}")
# ── Also save DB results properly ───────────────────────────────────
print("\n Saving results to DB...")
run = AnalysisRun.objects.create(
csv_glob=CSV_PATH, status='completed',
total_flows=row_count, entity_count=n_entities,
cluster_count=n_clusters, entity_column=used_col,
)
# Save ClusterResult records
for label in sorted_clusters:
size = cluster_sizes[label]
proportion = size / n_entities if n_entities > 0 else 0
ClusterResult.objects.create(
run=run, cluster_label=int(label),
size=size, proportion=proportion,
noise_ratio=quality.get('noise_ratio'),
silhouette_score=quality.get('silhouette_score'),
)
print(f" Saved {len(sorted_clusters)} ClusterResult records")
# Save EntityProfile records with correct cluster labels
saved_ep = 0
for i, ev in enumerate(entity_values):
if i >= len(labels_array):
break
lbl = labels_array[i]
cr = ClusterResult.objects.filter(run=run, cluster_label=int(lbl)).first()
EntityProfile.objects.create(
run=run, entity_value=str(ev),
cluster_label=int(lbl), cluster=cr,
feature_json={},
)
saved_ep += 1
print(f" Saved {saved_ep} EntityProfile records")
print(f" Run ID: #{run.id}")
print("\n" + "=" * 70)
print("EVALUATION COMPLETE")
print("=" * 70)
def classify_cluster(label, features, services):
"""Classify a cluster as NORMAL, SUSPICIOUS, or MALICIOUS based on TLS features and services."""
if not services:
return "UNKNOWN"
top_services = services.most_common(3)
service_names = [s[0] for s in top_services]
# Check for suspicious TLS signals in feature z-scores
high_sni_missing = False
low_tls_modern = False
high_non_std_port = False
low_recoverable = False
for feat_name, feat_data in features.items():
score = feat_data.get('distinguishing_score', 0)
if 'sni_missing' in feat_name and score > 1.5:
high_sni_missing = True
elif 'tls_modern' in feat_name and score < -1.5:
low_tls_modern = True
elif 'non_standard_port' in feat_name and score > 1.5:
high_non_std_port = True
elif 'recoverable' in feat_name and score < -1.0:
low_recoverable = True
# Also look at services
suspicious_services = {'Unknown'}
# If cluster has high concentration of one service, that's normal behavior
dominant_pct = services.most_common(1)[0][1] / sum(services.values())
if dominant_pct > 0.7:
return f"NORMAL — mostly {services.most_common(1)[0][0]} ({dominant_pct:.0%})"
# Check for suspicious feature patterns
suspicious_signals = []
if high_sni_missing:
suspicious_signals.append('high sni_missing')
if low_tls_modern:
suspicious_signals.append('low tls_modern')
if high_non_std_port:
suspicious_signals.append('non_std_ports')
if low_recoverable:
suspicious_signals.append('low_recoverability')
if len(suspicious_signals) >= 2:
return f"SUSPICIOUS — {', '.join(suspicious_signals)}; services: {', '.join(service_names)}"
elif len(suspicious_signals) >= 1:
return f"QUERY — {', '.join(suspicious_signals)}; dominant: {services.most_common(1)[0][0]}"
else:
return f"NORMAL — services: {', '.join(service_names)}"
if __name__ == '__main__':
main()
scripts/gen_complex_test.py
+86
View File
@@ -0,0 +1,86 @@
"""生成复杂 TLS 合成测试数据（含经纬度、MAC地址、大量空值）"""
import polars as pl
import numpy as np
from pathlib import Path
RNG = np.random.RandomState(42)
CLIENTS = [f'192.168.{RNG.randint(1,255)}.{RNG.randint(1,255)}' for _ in range(80)]
SERVERS = (
[f'10.0.{RNG.randint(1,255)}.{RNG.randint(1,255)}' for _ in range(80)]
+ [f'203.0.113.{RNG.randint(1,255)}' for _ in range(30)]
)
DOMAINS = ['mail.example.com', 'api.example.org', 'cdn.example.net', 'login.example.org',
'storage.example.net', 'video.example.com', 'auth.example.com', 'pay.example.org']
TLS_VERSIONS = ['TLSv1.2', 'TLSv1.3', 'TLSv1.2', 'TLSv1.3', 'TLSv1.1', None]
CIPHERS = ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384',
'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', None]
CERT_STATUS = ['valid', 'valid', 'valid', 'expired', 'self_signed', None]
def gen_mac():
return ':'.join(f'{RNG.randint(0,255):02x}' for _ in range(6))
CITIES = [
('Beijing', 39.9, 116.4), ('Shanghai', 31.2, 121.5), ('Tokyo', 35.7, 139.7),
('Singapore', 1.35, 103.8), ('London', 51.5, -0.13), ('Frankfurt', 50.1, 8.68),
('NewYork', 40.7, -74.0), ('SanFrancisco', 37.8, -122.4),
('Sydney', -33.9, 151.2), ('Mumbai', 19.1, 72.9),
]
def gen_flows(n_rows=5000):
from datetime import datetime, timedelta
base_ts = datetime(2025, 1, 1)
rows = []
for _ in range(n_rows):
src_ip = RNG.choice(CLIENTS)
dst_ip = RNG.choice(SERVERS)
si = RNG.randint(0, len(CITIES))
di = RNG.randint(0, len(CITIES))
src_city, src_lat, src_lon = CITIES[si]
dst_city, dst_lat, dst_lon = CITIES[di]
same_city = src_city == dst_city
bs = int(max(0, RNG.lognormal(8, 2))) if RNG.random() > 0.05 else None
br = int(max(0, bs * RNG.uniform(0.1, 2))) if bs and RNG.random() > 0.05 else None
dur = round(RNG.exponential(20), 2) if RNG.random() > 0.08 else None
hok = RNG.choice([True, True, True, False, None])
trusted = hok and RNG.choice([True, True, False, None]) if hok else None
tmo = int(RNG.exponential(500)) + 100 if hok else RNG.choice([None, RNG.randint(5000, 30000)])
rows.append({
'timestamp': (base_ts + timedelta(seconds=RNG.randint(0, 31536000))).strftime('%Y-%m-%d %H:%M:%S'),
'src_ip': src_ip, 'dst_ip': dst_ip,
'src_mac': gen_mac(), 'dst_mac': gen_mac(),
'src_url': f'https://{RNG.choice(DOMAINS)}{RNG.choice(["/","/login","/api/v1/users","/api/v1/data","/favicon.ico"])}',
'dst_url': f'https://{RNG.choice(DOMAINS)}{RNG.choice(["/","/login","/api/v1/users","/api/v1/data"])}',
'sni': RNG.choice(DOMAINS + [None, None]),
'src_city': src_city, 'src_latitude': src_lat, 'src_longitude': src_lon,
'dst_city': dst_city if not same_city else None,
'dst_latitude': dst_lat if not same_city else None,
'dst_longitude': dst_lon if not same_city else None,
'bytes_sent': bs, 'bytes_rev': br, 'duration': dur,
'tls_version': RNG.choice(TLS_VERSIONS),
'cipher': RNG.choice(CIPHERS),
'dst_port': RNG.choice([80, 443, 8080, 8443, 53, 993, 5223, None]),
'cert_status': RNG.choice(CERT_STATUS),
'handshake_ok': hok, 'trusted': trusted, 'timeout_ms': tmo,
})
df = pl.DataFrame(rows)
nulls = df.null_count()
for col in df.columns:
n = int(nulls[col].item())
if n > 0:
print(f' {col}: {n}/{len(df)} 缺失 ({n*100//len(df)}%)')
print(f'总行数: {len(df)}, 列数: {len(df.columns)}')
return df
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--rows', type=int, default=5000)
parser.add_argument('--output', type=str, default='data/complex_test.csv')
args = parser.parse_args()
df = gen_flows(args.rows)
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
df.write_csv(args.output, null_value='')
print(f'已生成: {args.output} ({Path(args.output).stat().st_size/1024/1024:.1f} MB)')
scripts/gen_test_data.py
+872
View File
@@ -0,0 +1,872 @@
"""Generate synthetic TLS flow test data with 3 real-world traffic profiles.
Usage:
runtime\\python\\python.exe scripts\\gen_test_data.py [--rows N] [--output PATH]
runtime\\python\\python.exe scripts\\gen_test_data.py --globe [--rows N] [--output PATH]
"""
import sys
import polars as pl
import numpy as np
from pathlib import Path
from datetime import datetime, timedelta
# ===================================================================
# TLS Constants
# ===================================================================
CIPHER_HEX_MAP: dict[str, str] = {
'TLS_AES_128_GCM_SHA256': '13 01',
'TLS_AES_256_GCM_SHA384': '13 02',
'TLS_ECDHE_ECDSA_AES128_GCM_SHA256': 'c0 2b',
'TLS_ECDHE_RSA_AES128_GCM_SHA256': 'c0 2f',
'TLS_ECDHE_RSA_AES256_GCM_SHA384': 'c0 30',
'TLS_RSA_AES128_GCM_SHA256': '00 9c',
'TLS_RSA_AES_128_CBC_SHA': '00 3c',
'TLS_ECDHE_RSA_AES128_CBC_SHA': 'c0 09',
'TLS_RSA_WITH_RC4_128_SHA': '00 05',
'TLS_CHACHA20_POLY1305_SHA256': 'cc a9',
}
CURVE_MAP: dict[str, str] = {
'00 1d': 'x25519',
'00 17': 'secp256r1',
'00 18': 'secp384r1',
'00 19': 'secp521r1',
}
COUNTRY_CODES: list[str] = [
'US', 'JP', 'DE', 'GB', 'SG', 'KR', 'FR', 'AU', 'IN', 'CA', 'BR',
'RU', 'CN', 'NG', 'IR', 'NL', 'SE', 'HK', 'TW', 'ZA',
]
COUNTRY_A: list[str] = ['US', 'JP', 'DE', 'GB', 'SG', 'KR', 'FR', 'AU', 'NL', 'CA', 'SE']
COUNTRY_B: list[str] = ['US', 'NL', 'DE', 'SG', 'HK', 'JP', 'GB']
COUNTRY_C: list[str] = ['RU', 'CN', 'BR', 'NG', 'IR', 'KR', 'US', 'HK', 'TW', 'ZA']
TABLE_NAMES: list[str] = [
'tls_flows', 'network_traffic', 'encrypted_streams', 'ssl_logs', 'packet_capture',
]
LINK_NAMES: list[str] = [
'internet-edge', 'mpls-backbone', 'vpn-tunnel', 'wan-link',
'cloud-connect', 'dc-interconnect', 'peering-link',
]
NODE_NAMES: list[str] = (
[f'gw-{i:02d}' for i in range(1, 11)]
+ [f'core-{i:02d}' for i in range(1, 7)]
)
ISP_NAMES: list[str] = [
'China Telecom', 'China Unicom', 'China Mobile', 'AT&T', 'Verizon',
'Comcast', 'Deutsche Telekom', 'BT Group', 'NTT', 'KDDI',
'Singtel', 'Telstra', 'Reliance Jio', 'T-Mobile', 'Orange',
'Rostelecom', 'Vivo', 'MTN Group', 'Etisalat',
]
ORG_NAMES: list[str] = [
'Alibaba Inc.', 'Tencent Holdings', 'Baidu Inc.', 'Google LLC',
'Amazon.com Inc.', 'Microsoft Corp.', 'Meta Platforms Inc.', 'Apple Inc.',
'Netflix Inc.', 'Cloudflare Inc.', 'Akamai Technologies',
'Yandex', 'Kaspersky Lab', 'Mail.ru Group',
]
CITY_NAMES: list[str] = [
'Beijing', 'Shanghai', 'Tokyo', 'Seoul', 'New York', 'London',
'Paris', 'Berlin', 'Singapore', 'Sydney', 'Mumbai', 'Dubai',
'Toronto', 'Moscow', 'Sao Paulo', 'San Francisco', 'Los Angeles',
'Lagos', 'Tehran', 'Hong Kong',
]
# ===================================================================
# Traffic Profile Definitions
# ===================================================================
# Each profile defines distribution parameters for row generation.
# We encode TLS version, cipher, curve as weighted-choice lists.
TLS_PROFILES: dict[str, list[tuple[str, float]]] = {
'A': [('03 04', 0.80), ('03 03', 0.18), ('02 00', 0.01), ('03 00', 0.01)],
'B': [('03 04', 0.50), ('03 03', 0.40), ('03 01', 0.05), ('03 02', 0.05)],
'C': [('03 01', 0.30), ('03 02', 0.20), ('03 03', 0.20), ('03 04', 0.10),
('02 00', 0.10), ('03 00', 0.10)],
}
CIPHER_PROFILES: dict[str, list[tuple[str, float]]] = {
'A': [('13 01', 0.50), ('c0 2b', 0.20), ('c0 2f', 0.15), ('13 02', 0.10), ('cc a9', 0.05)],
'B': [('c0 2f', 0.30), ('13 01', 0.30), ('c0 30', 0.15), ('00 9c', 0.15), ('13 02', 0.10)],
'C': [('00 3c', 0.25), ('c0 09', 0.20), ('00 05', 0.10), ('13 01', 0.15),
('c0 2f', 0.10), ('c0 2b', 0.10), ('00 9c', 0.10)],
}
CURVE_PROFILES: dict[str, list[tuple[str, float]]] = {
'A': [('00 1d', 0.70), ('00 17', 0.25), ('00 18', 0.05)],
'B': [('00 17', 0.50), ('00 1d', 0.40), ('00 18', 0.10)],
'C': [('00 17', 0.25), ('00 1d', 0.20), ('00 18', 0.15),
('00 19', 0.10), ('', 0.30)], # 30% no curve
}
# Port profiles: (port, weight)
PORT_PROFILES: dict[str, list[tuple[int, float]]] = {
'A': [(443, 0.90), (80, 0.05), (8080, 0.02), (8443, 0.01), (993, 0.01), (465, 0.01)],
'B': [(443, 0.60), (8443, 0.15), (8080, 0.10), (3128, 0.05), (1080, 0.05), (8888, 0.05)],
'C': [(443, 0.60), (80, 0.05), (8080, 0.05), (8443, 0.05), (0, 0.25)], # 0 = random non-standard
}
# Duration (seconds) profiles: used with exponential + clip
DUR_PROFILES: dict[str, tuple[float, float, float]] = {
'A': (2.0, 0.5, 8.0), # (mean exp, min, max)
'B': (60.0, 10.0, 300.0),
'C': (1.0, 0.3, 600.0), # bimodal: will mix short & long
}
# ===================================================================
# IP Pools (deterministic via numpy seed)
# ===================================================================
def _ip_pool(n: int, seed_base: int, prefix: str | None = None) -> list[str]:
"""Generate *n* reproducible IPs with optional *prefix* (e.g. '10.0.1')."""
rng = np.random.RandomState(seed_base)
ips: list[str] = []
for _ in range(n):
if prefix:
parts = prefix.split('.')
ip = (f'{parts[0]}.{parts[1]}.{parts[2]}.'
f'{rng.randint(1, 255)}')
else:
a = rng.randint(1, 223)
b = rng.randint(0, 256)
c = rng.randint(0, 256)
d = rng.randint(1, 255)
ip = f'{a}.{b}.{c}.{d}'
ips.append(ip)
return ips
NORMAL_SRC_IPS: list[str] = _ip_pool(150, 42) # residential-like
NORMAL_DST_IPS: list[str] = _ip_pool(120, 142) # CDN/cloud-like
PROXY_SRC_IPS: list[str] = _ip_pool(8, 242) # proxy exit nodes
PROXY_DST_IPS: list[str] = _ip_pool(15, 342) # limited targets
MAL_SRC_IPS: list[str] = _ip_pool(180, 442) # botnet-like diverse
MAL_DST_IPS: list[str] = _ip_pool(60, 542) # malicious targets
# ===================================================================
# SNI / Domain pools
# ===================================================================
LEGIT_DOMAINS: list[str] = [
'www.google.com', 'api.github.com', 'facebook.com', 'youtube.com',
'amazon.com', 'netflix.com', 'apple.com', 'microsoft.com',
'cloudflare.com', 'stackoverflow.com', 'reddit.com', 'twitter.com',
'linkedin.com', 'instagram.com', 'office.com', 'live.com',
'bing.com', 'adobe.com', 'salesforce.com', 'zoom.us',
'maps.google.com', 'mail.google.com', 'drive.google.com',
'docs.google.com', 'play.google.com', 'support.microsoft.com',
'portal.office.com', 'login.live.com', 'cdn.cloudflare.com',
]
PROXY_DOMAINS: list[str] = [
'proxy.example.com', 'vpn-gateway.example.net', 'tunnel.example.org',
'exit-node.example.com', 'relay.example.net', 'gateway.example.org',
'', '',
]
DGA_DOMAINS: list[str] = [
'gkj3hj2k3.sdf2k3j.net', '8sdf7h3k.xyz', 'j2k3h4k5h7.ru',
'asdf9082k.biz', 'mnbv34k2.info', '2kj3h5k7jh9.com',
'xcnv98sdf2.top', 'q3kj4h5k7.work', 'vbsd78sdf.xyz',
'm2k3j5h6k8.ru', '9s8d7f6g5h.net', 'k2j3h5k7j9.org',
'fasdf2342.biz', 'poiuy234k.info', 'zxcv7890m.top',
'hk3j5k7h9.net', 'sdf2k3j4h5.ru', 'vb9s8d7f6.xyz',
'n2k3j5h7k9.com', 'x98s7d6f5.info',
]
FAKE_DOMAINS: list[str] = [
'secure-login-app.com', 'update-manager.net', 'cdn-cache-service.com',
'analytics-tracker.net', 'content-delivery-system.org',
'download-center.net', 'account-verify.com', 'system-update.org',
'cloud-sync-service.net', 'notification-hub.com',
]
# ===================================================================
# Helpers
# ===================================================================
def _weighted_choice(options: list[tuple[str, float]]) -> str:
"""Pick one from [(value, weight), ...] according to weight."""
items, raw_w = zip(*options)
total = sum(raw_w)
return str(np.random.choice(items, p=[w / total for w in raw_w]))
def _weighted_choice_int(options: list[tuple[int, float]]) -> int:
"""Pick int from [(value, weight), ...]."""
items, raw_w = zip(*options)
total = sum(raw_w)
return int(np.random.choice(items, p=[w / total for w in raw_w]))
def _random_hex_pairs(n_pairs: int = 2) -> str:
"""Generate space-separated hex pairs, e.g. 'c0 2b'."""
pairs = [f'{np.random.randint(0, 256):02x}' for _ in range(n_pairs)]
return ' '.join(pairs)
def _random_32_bytes_hex() -> str:
"""Generate 32 random hex bytes as a space-separated string."""
pairs = [f'{np.random.randint(0, 256):02x}' for _ in range(32)]
return ' '.join(pairs)
def _low_entropy_hex() -> str:
"""Generate repeating low-entropy TLS random (Profile C: beaconing).
Pattern: '00 01 00 01 00 01 ...' (repeating 2-byte cycle).
"""
pairs = ['00', '01'] * 16
return ' '.join(pairs)
def _maybe(rate: float) -> bool:
"""Return True with probability *rate* (value should be blanked)."""
return np.random.random() < rate
def _assign_profile() -> str:
"""Assign profile A (60%), B (20%), or C (20%)."""
r = np.random.random()
if r < 0.60:
return 'A'
elif r < 0.80:
return 'B'
else:
return 'C'
def _pick_sni(profile: str) -> tuple[str, str]:
"""Generate (snam, cnam) for the given profile.
Returns (sni_name, common_name).
"""
if profile == 'A':
if _maybe(0.05):
return ('', '')
domain = str(np.random.choice(LEGIT_DOMAINS))
return (domain, domain) # always match
elif profile == 'B':
if _maybe(0.20):
return ('', '')
domain = str(np.random.choice(PROXY_DOMAINS))
if domain == '':
return ('', '')
return (domain, domain)
else: # C
if _maybe(0.30):
return ('', '')
r = np.random.random()
if r < 0.20:
domain = str(np.random.choice(DGA_DOMAINS))
elif r < 0.70:
domain = str(np.random.choice(FAKE_DOMAINS))
else:
domain = str(np.random.choice(LEGIT_DOMAINS))
# 30% mismatch snam != cnam
if _maybe(0.30):
if _maybe(0.5):
cnam = str(np.random.choice(FAKE_DOMAINS))
else:
cnam = str(np.random.choice(LEGIT_DOMAINS))
return (domain, cnam)
return (domain, domain)
def _pick_country(profile: str) -> tuple[str, str]:
"""Pick (scnt, dcnt) for the given profile."""
if profile == 'A':
scnt = str(np.random.choice(COUNTRY_A))
dcnt = str(np.random.choice(COUNTRY_A))
elif profile == 'B':
scnt = str(np.random.choice(COUNTRY_B))
dcnt = str(np.random.choice(COUNTRY_B))
else:
scnt = str(np.random.choice(COUNTRY_C))
dcnt = str(np.random.choice(COUNTRY_C))
return scnt, dcnt
# ===================================================================
# Main generator — 3 profiles, 44 columns
# ===================================================================
def generate_tls_flows(n_rows: int = 1000, seed: int = 42) -> pl.DataFrame:
"""Generate synthetic TLS flow data with 3 real-world traffic profiles.
Profile A (60%): Normal Browsing
Profile B (20%): Proxy/VPN Traffic
Profile C (20%): Malicious/Suspicious
Returns a Polars DataFrame with all 44 columns matching the user's
exact column naming convention.
"""
np.random.seed(seed)
base_ts = datetime(2024, 1, 1, 0, 0, 0)
day_seconds = 86400 * 90 # 90-day window
rows: list[dict] = []
total = n_rows
next_pct = 10
for i in range(total):
# Progress indicator
pct = (i + 1) * 100 // total
if pct >= next_pct:
sys.stdout.write('+')
sys.stdout.flush()
next_pct += 10
profile = _assign_profile()
# --- IPs ---
if profile == 'A':
src_ip = str(np.random.choice(NORMAL_SRC_IPS))
dst_ip = str(np.random.choice(NORMAL_DST_IPS))
elif profile == 'B':
src_ip = str(np.random.choice(PROXY_SRC_IPS))
dst_ip = str(np.random.choice(PROXY_DST_IPS))
else:
src_ip = str(np.random.choice(MAL_SRC_IPS))
dst_ip = str(np.random.choice(MAL_DST_IPS))
# --- Ports ---
src_port = np.random.randint(1024, 65536)
dst_port = _weighted_choice_int(PORT_PROFILES[profile])
if dst_port == 0: # random non-standard
dst_port = int(np.random.choice([p for p in range(1025, 65536) if p != 443]))
# --- scnt, dcnt ---
scnt, dcnt = _pick_country(profile)
# --- TLS version ---
tls_ver = _weighted_choice(TLS_PROFILES[profile])
# --- SNI / CN ---
snam, cnam = _pick_sni(profile)
# --- Duration ---
mean_dur, min_dur, max_dur = DUR_PROFILES[profile]
if profile == 'C':
# Bimodal: beaconing (<1s) or C2 (long, >60s)
if _maybe(0.50):
dur = round(np.random.uniform(0.3, 1.0), 2)
else:
dur = round(min(max(np.random.exponential(120), 60), 600), 2)
else:
dur = round(min(max(np.random.exponential(mean_dur), min_dur), max_dur), 2)
# --- Session / timeout / key size ---
ses = round(np.random.uniform(0, 1000), 4)
tmo = round(np.random.uniform(0, 300), 4)
ksz = int(np.random.choice([128, 256, 384, 521]))
# --- cnrs / isrs (blank="no", +="yes") ---
if profile == 'A':
cnrs_val = '+' if _maybe(0.30) else ''
isrs_val = '+' if np.random.random() < 0.10 else ''
elif profile == 'B':
cnrs_val = '+' if _maybe(0.60) else ''
isrs_val = '+' if np.random.random() < 0.40 else ''
else:
cnrs_val = '+' if _maybe(0.20) else ''
isrs_val = '+' if np.random.random() < 0.50 else ''
# --- Bytes / packets ---
if profile == 'A':
ack_val = int(max(1, np.random.exponential(3000)))
ppk_val = max(1, int(ack_val / np.random.uniform(200, 1500)))
elif profile == 'B':
ack_val = int(max(1, np.random.exponential(50000)))
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
else:
if _maybe(0.50):
# Beaconing: very small, regular packets
ack_val = int(np.random.uniform(30, 100)) * np.random.randint(1, 20)
ppk_val = max(1, ack_val // int(np.random.uniform(30, 100)))
else:
# Data exfil: large transfer
ack_val = int(max(1, np.random.exponential(100000)))
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
# --- Timestamps ---
ts_offset = np.random.randint(0, day_seconds)
ts_full = base_ts + timedelta(seconds=ts_offset)
ts_str = ts_full.strftime('%Y-%m-%d %H:%M:%S')
tm_str = ts_full.strftime('%H:%M:%S')
dbd_str = (base_ts + timedelta(
seconds=np.random.randint(0, day_seconds)
)).strftime('%Y-%m-%d %H:%M:%S')
# --- Cipher ---
cipher_hex = _weighted_choice(CIPHER_PROFILES[profile])
cipher_name = CIPHER_HEX_MAP.get(cipher_hex, '')
# --- Curve ---
curve_hex = _weighted_choice(CURVE_PROFILES[profile])
curve_name = CURVE_MAP.get(curve_hex, '')
# --- Other ---
ipp_val = int(np.random.choice([6, 17], p=[0.85, 0.15]))
dbn_val = np.random.randint(1, 10)
tabl = str(np.random.choice(TABLE_NAMES))
link_name = str(np.random.choice(LINK_NAMES))
node = str(np.random.choice(NODE_NAMES))
# --- TLS random (low entropy for Profile C) ---
if profile == 'C':
rnd = _low_entropy_hex()
else:
rnd = _random_32_bytes_hex()
rnt_val = int(base_ts.timestamp()) + np.random.randint(0, day_seconds)
# --- GeoIP dot-notation columns (non-globe: mostly blank) ---
def _geo_val(val, blank_rate=0.70):
return val if not _maybe(blank_rate) else ''
lat_src = '' if _maybe(0.70) else round(np.random.uniform(-90, 90), 6)
lon_src = '' if _maybe(0.70) else round(np.random.uniform(-180, 180), 6)
lat_dst = '' if _maybe(0.70) else round(np.random.uniform(-90, 90), 6)
lon_dst = '' if _maybe(0.70) else round(np.random.uniform(-180, 180), 6)
ispn_src = '' if _maybe(0.70) else str(np.random.choice(ISP_NAMES))
ispn_dst = '' if _maybe(0.70) else str(np.random.choice(ISP_NAMES))
orgn_src = '' if _maybe(0.70) else str(np.random.choice(ORG_NAMES))
orgn_dst = '' if _maybe(0.70) else str(np.random.choice(ORG_NAMES))
city_src = '' if _maybe(0.70) else str(np.random.choice(CITY_NAMES))
city_dst = '' if _maybe(0.70) else str(np.random.choice(CITY_NAMES))
# --- Assemble row dict ---
row = {
':ips': src_ip,
':ipd': dst_ip,
':prs': '' if _maybe(0.30) else src_port,
':prd': '' if _maybe(0.30) else dst_port,
'scnt': scnt if not _maybe(0.30) else '',
'dcnt': dcnt if not _maybe(0.30) else '',
'server-ip': dst_ip,
'client-ip': src_ip,
'0ver': tls_ver if not _maybe(0.05) else '',
'snam': snam,
'cnam': cnam if not _maybe(0.05) else '',
'4dur': '' if _maybe(0.30) else dur,
'8ses': '' if _maybe(0.30) else ses,
'2tmo': '' if _maybe(0.30) else tmo,
'4ksz': '' if _maybe(0.30) else ksz,
'cnrs': cnrs_val,
'isrs': isrs_val,
'8ack': ack_val if not _maybe(0.05) else '',
'8ppk': ppk_val if not _maybe(0.05) else '',
'8dbd': dbd_str if not _maybe(0.30) else '',
'1ipp': ipp_val if not _maybe(0.30) else '',
'4dbn': '' if _maybe(0.30) else dbn_val,
'tabl': tabl if not _maybe(0.30) else '',
'name': link_name if not _maybe(0.30) else '',
'source-node': node if not _maybe(0.30) else '',
'cipher-suite': cipher_name if not _maybe(0.30) else '',
'ecdhe-named-curve': curve_name if not _maybe(0.30) else '',
'0cph': cipher_hex if not _maybe(0.30) else '',
'0crv': curve_hex if not _maybe(0.30) else '',
'0rnd': rnd if not _maybe(0.30) else '',
'0rnt': '' if _maybe(0.30) else rnt_val,
'row': i + 1,
'time': tm_str if not _maybe(0.30) else '',
'timestamp': ts_str if not _maybe(0.10) else '',
':ips.latd': lat_src,
':ips.lond': lon_src,
':ipd.latd': lat_dst,
':ipd.lond': lon_dst,
':ips.ispn': ispn_src,
':ipd.ispn': ispn_dst,
':ips.orgn': orgn_src,
':ipd.orgn': orgn_dst,
':ips.city': city_src,
':ipd.city': city_dst,
}
rows.append(row)
sys.stdout.write('\n')
sys.stdout.flush()
return pl.DataFrame(rows)
# ===================================================================
# GeoIP loader (shared by globe mode)
# ===================================================================
def _load_geoip_ranges() -> dict[str, list[dict]]:
"""Load GeoIP range data from ``data/geoip_data.txt``."""
data_path = Path(__file__).resolve().parent.parent / 'data' / 'geoip_data.txt'
cities: dict[str, list[dict]] = {}
with open(data_path, 'r', encoding='utf-8') as f:
for raw in f:
stripped = raw.strip()
if not stripped or stripped.startswith('#'):
continue
parts = [p.strip() for p in stripped.split(',')]
if len(parts) != 6:
continue
start_str, end_str, lat_str, lon_str, city, _country = parts
start_int = _ip_to_int(start_str)
end_int = _ip_to_int(end_str)
if start_int is None or end_int is None:
continue
try:
lat = float(lat_str)
lon = float(lon_str)
except ValueError:
continue
city_key = city.replace(' ', '')
if city_key not in cities:
cities[city_key] = []
cities[city_key].append({
'start': start_int,
'end': end_int,
'lat': lat,
'lon': lon,
})
return cities
def _ip_to_int(ip_str: str) -> int | None:
"""Convert dotted IPv4 string to 32-bit integer."""
parts = ip_str.strip().split('.')
if len(parts) != 4:
return None
try:
octets = [int(p) for p in parts]
if any(o < 0 or o > 255 for o in octets):
return None
return (octets[0] << 24) | (octets[1] << 16) | (octets[2] << 8) | octets[3]
except (ValueError, TypeError):
return None
def _int_to_ip(n: int) -> str:
"""Convert 32-bit integer to dotted IPv4 string."""
return f'{n >> 24 & 255}.{n >> 16 & 255}.{n >> 8 & 255}.{n & 255}'
# ---------------------------------------------------------------------------
# Globe mode city → (ISP, Org) mapping
# ---------------------------------------------------------------------------
_GLOBE_ORG_MAP: dict[str, tuple[str, str]] = {
'Beijing': ('China Unicom', 'Tencent Holdings'),
'Shanghai': ('China Telecom', 'Alibaba Inc.'),
'Taipei': ('Chunghwa Telecom', 'Hon Hai Group'),
'Tokyo': ('NTT', 'SoftBank Corp.'),
'Seoul': ('KT Corp.', 'Samsung SDS'),
'Singapore': ('Singtel', 'Grab Holdings'),
'Bangkok': ('True Corp.', 'G-Able'),
'Mumbai': ('Reliance Jio', 'Tata Consultancy'),
'Sydney': ('Telstra', 'Atlassian'),
'MountainView': ('Google Fiber', 'Google LLC'),
'NewYork': ('Verizon', 'Amazon.com Inc.'),
'LosAngeles': ('AT&T', 'Meta Platforms Inc.'),
'Toronto': ('Rogers', 'Shopify Inc.'),
'London': ('BT Group', 'British Telecom'),
'Paris': ('Orange', 'TotalEnergies'),
'Frankfurt': ('Deutsche Telekom', 'SAP SE'),
'Amsterdam': ('KPN', 'ING Group'),
'Dublin': ('Eir', 'Accenture'),
'Stockholm': ('Telia', 'Ericsson'),
'Moscow': ('Rostelecom', 'Yandex'),
'Dubai': ('Etisalat', 'Emirates Group'),
'SaoPaulo': ('Vivo', 'Itau Unibanco'),
'HongKong': ('PCCW', 'HSBC Holdings'),
'Lagos': ('MTN Group', 'Access Bank'),
'Tehran': ('TCI', 'Iran Khodro'),
}
# ===================================================================
# Globe mode — generates data suitable for 3D globe visualisation
# ===================================================================
def generate_globe_test_data(n_rows: int = 1000, seed: int = 42) -> pl.DataFrame:
"""Generate test data for 3D globe visualisation using GeoIP-resolvable IPs.
Uses the same 3 traffic profiles as the normal mode, but with
real GeoIP-resolvable source/destination IPs so lat/lon columns
correspond to actual geographic locations.
Data corruption patterns (for testing globe view robustness):
- ~30% of ``:ips.latd`` values are pure ``+``
- ~10% of ``:ips.lond`` values are empty ``""``
"""
np.random.seed(seed)
city_ranges = _load_geoip_ranges()
# Source cities (Asia / Pacific)
src_cities = [
'Beijing', 'Shanghai', 'Taipei', 'Tokyo',
'Seoul', 'Singapore', 'Bangkok', 'Mumbai', 'Sydney',
]
# Destination cities (global rest)
dst_cities = [
'MountainView', 'NewYork', 'LosAngeles', 'Toronto',
'London', 'Paris', 'Frankfurt', 'Amsterdam', 'Dublin',
'Stockholm', 'Moscow', 'Dubai', 'SaoPaulo',
]
all_cities = set(city_ranges.keys())
src_cities = [c for c in src_cities if c in all_cities]
dst_cities = [c for c in dst_cities if c in all_cities]
if not src_cities or not dst_cities:
raise RuntimeError('No usable GeoIP city ranges — cannot generate globe data.')
def _rand_ip_from_city(city: str) -> tuple[str, float, float]:
rng_list = city_ranges[city]
rng_entry = rng_list[np.random.randint(len(rng_list))]
ip_int = rng_entry['start'] + np.random.randint(0, rng_entry['end'] - rng_entry['start'] + 1)
return _int_to_ip(ip_int), rng_entry['lat'], rng_entry['lon']
base_ts = int(datetime(2023, 11, 1).timestamp())
thirty_days_sec = 86400 * 60
rows: list[dict] = []
total = n_rows
next_pct = 10
for i in range(total):
# Progress indicator
pct = (i + 1) * 100 // total
if pct >= next_pct:
sys.stdout.write('+')
sys.stdout.flush()
next_pct += 10
profile = _assign_profile()
# GeoIP-resolvable IPs + lat/lon
src_city = str(np.random.choice(src_cities))
src_ip_raw, src_lat_raw, src_lon_raw = _rand_ip_from_city(src_city)
dst_city = str(np.random.choice(dst_cities))
dst_ip_raw, dst_lat, dst_lon = _rand_ip_from_city(dst_city)
# Globe-specific corruption patterns
src_lat_out: str | float = '+' if np.random.random() < 0.30 else round(src_lat_raw, 6)
src_lon_out: str | float = '' if np.random.random() < 0.10 else round(src_lon_raw, 6)
# --- Ports ---
src_port = np.random.randint(1024, 65536)
dst_port = _weighted_choice_int(PORT_PROFILES[profile])
if dst_port == 0:
dst_port = int(np.random.choice([p for p in range(1025, 65536) if p != 443]))
# --- scnt, dcnt ---
scnt, dcnt = _pick_country(profile)
# --- TLS version ---
tls_ver = _weighted_choice(TLS_PROFILES[profile])
# --- SNI ---
snam, cnam = _pick_sni(profile)
# --- Duration ---
mean_dur, min_dur, max_dur = DUR_PROFILES[profile]
if profile == 'C':
if _maybe(0.50):
dur = round(np.random.uniform(0.3, 1.0), 2)
else:
dur = round(min(max(np.random.exponential(120), 60), 600), 2)
else:
dur = round(min(max(np.random.exponential(mean_dur), min_dur), max_dur), 2)
# --- Session / timeout / key size ---
ses = round(np.random.uniform(0, 1000), 4)
tmo = round(np.random.uniform(0, 300), 4)
ksz = int(np.random.choice([128, 256, 384, 521]))
# --- cnrs / isrs ---
if profile == 'A':
cnrs_val = '+' if _maybe(0.30) else ''
isrs_val = '+' if np.random.random() < 0.10 else ''
elif profile == 'B':
cnrs_val = '+' if _maybe(0.60) else ''
isrs_val = '+' if np.random.random() < 0.40 else ''
else:
cnrs_val = '+' if _maybe(0.20) else ''
isrs_val = '+' if np.random.random() < 0.50 else ''
# --- Bytes / packets ---
if profile == 'A':
ack_val = int(max(1, np.random.exponential(3000)))
ppk_val = max(1, int(ack_val / np.random.uniform(200, 1500)))
elif profile == 'B':
ack_val = int(max(1, np.random.exponential(50000)))
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
else:
if _maybe(0.50):
ack_val = int(np.random.uniform(30, 100)) * np.random.randint(1, 20)
ppk_val = max(1, ack_val // int(np.random.uniform(30, 100)))
else:
ack_val = int(max(1, np.random.exponential(100000)))
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
# --- Timestamps ---
ts = base_ts + np.random.randint(0, thirty_days_sec)
ts_str = datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
tm_str = datetime.fromtimestamp(ts).strftime('%H:%M:%S')
dbd_str = datetime.fromtimestamp(
base_ts + np.random.randint(0, thirty_days_sec)
).strftime('%Y-%m-%d %H:%M:%S')
# --- Cipher ---
cipher_hex = _weighted_choice(CIPHER_PROFILES[profile])
cipher_name = CIPHER_HEX_MAP.get(cipher_hex, '')
# --- Curve ---
curve_hex = _weighted_choice(CURVE_PROFILES[profile])
curve_name = CURVE_MAP.get(curve_hex, '')
# --- Other ---
ipp_val = int(np.random.choice([6, 17], p=[0.85, 0.15]))
dbn_val = np.random.randint(1, 10)
tabl = str(np.random.choice(TABLE_NAMES))
link_name = str(np.random.choice(LINK_NAMES))
node = str(np.random.choice(NODE_NAMES))
# --- TLS random ---
if profile == 'C':
rnd = _low_entropy_hex()
else:
rnd = _random_32_bytes_hex()
rnt_val = base_ts + np.random.randint(0, thirty_days_sec)
# --- GeoIP ISP/org from globe map ---
src_isp, src_org = _GLOBE_ORG_MAP.get(
src_city, (str(np.random.choice(ISP_NAMES)), str(np.random.choice(ORG_NAMES))),
)
dst_isp, dst_org = _GLOBE_ORG_MAP.get(
dst_city, (str(np.random.choice(ISP_NAMES)), str(np.random.choice(ORG_NAMES))),
)
rows.append({
':ips': src_ip_raw,
':ipd': dst_ip_raw,
':prs': '' if _maybe(0.30) else src_port,
':prd': '' if _maybe(0.30) else dst_port,
'scnt': scnt if not _maybe(0.30) else '',
'dcnt': dcnt if not _maybe(0.30) else '',
'server-ip': dst_ip_raw,
'client-ip': src_ip_raw,
'0ver': tls_ver if not _maybe(0.05) else '',
'snam': snam,
'cnam': cnam if not _maybe(0.05) else '',
'4dur': '' if _maybe(0.30) else dur,
'8ses': '' if _maybe(0.30) else ses,
'2tmo': '' if _maybe(0.30) else tmo,
'4ksz': '' if _maybe(0.30) else ksz,
'cnrs': cnrs_val,
'isrs': isrs_val,
'8ack': ack_val if not _maybe(0.05) else '',
'8ppk': ppk_val if not _maybe(0.05) else '',
'8dbd': dbd_str if not _maybe(0.30) else '',
'1ipp': ipp_val if not _maybe(0.30) else '',
'4dbn': '' if _maybe(0.30) else dbn_val,
'tabl': tabl if not _maybe(0.30) else '',
'name': link_name if not _maybe(0.30) else '',
'source-node': node if not _maybe(0.30) else '',
'cipher-suite': cipher_name if not _maybe(0.30) else '',
'ecdhe-named-curve': curve_name if not _maybe(0.30) else '',
'0cph': cipher_hex if not _maybe(0.30) else '',
'0crv': curve_hex if not _maybe(0.30) else '',
'0rnd': rnd if not _maybe(0.30) else '',
'0rnt': '' if _maybe(0.30) else rnt_val,
'row': i + 1,
'time': tm_str if not _maybe(0.30) else '',
'timestamp': ts_str if not _maybe(0.10) else '',
':ips.latd': src_lat_out,
':ips.lond': src_lon_out,
':ipd.latd': round(dst_lat, 6),
':ipd.lond': round(dst_lon, 6),
':ips.ispn': src_isp,
':ipd.ispn': dst_isp,
':ips.orgn': src_org,
':ipd.orgn': dst_org,
':ips.city': src_city,
':ipd.city': dst_city,
})
sys.stdout.write('\n')
sys.stdout.flush()
return pl.DataFrame(rows)
# ===================================================================
# CSV save & CLI
# ===================================================================
EXPECTED_COLUMNS: list[str] = [
':ips', ':ipd', ':prs', ':prd', 'scnt', 'dcnt', 'server-ip', 'client-ip',
'0ver', 'snam', 'cnam', '4dur', '8ses', '2tmo', '4ksz', 'cnrs', 'isrs',
'8ack', '8ppk', '8dbd', '1ipp', '4dbn', 'tabl', 'name', 'source-node',
'cipher-suite', 'ecdhe-named-curve', '0cph', '0crv', '0rnd', '0rnt',
'row', 'time', 'timestamp',
':ips.latd', ':ips.lond', ':ipd.latd', ':ipd.lond',
':ips.ispn', ':ipd.ispn', ':ips.orgn', ':ipd.orgn', ':ips.city', ':ipd.city',
]
def save_csv(df: pl.DataFrame, path: str):
"""Write DataFrame to CSV with full 44-column validation."""
out_path = Path(path)
out_path.parent.mkdir(parents=True, exist_ok=True)
# Validate column count
actual_cols = df.columns
if len(actual_cols) != 44:
print(f'WARNING: Expected 44 columns, got {len(actual_cols)}')
missing = set(EXPECTED_COLUMNS) - set(actual_cols)
extra = set(actual_cols) - set(EXPECTED_COLUMNS)
if missing:
print(f' Missing: {sorted(missing)}')
if extra:
print(f' Extra: {sorted(extra)}')
df.write_csv(out_path)
size_kb = out_path.stat().st_size / 1024
print(f'Generated: {path} ({len(df)} rows, {size_kb:.0f} KB, {len(actual_cols)} cols)')
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser(description='Generate synthetic TLS flow test data (3 traffic profiles)')
parser.add_argument('--rows', type=int, default=1000, help='Number of rows (default: 1000)')
parser.add_argument('--output', type=str, default=None, help='Output CSV path')
parser.add_argument('--globe', action='store_true',
help='Generate 3D globe visualisation test data (GeoIP-resolvable IPs)')
args = parser.parse_args()
if args.globe:
output = args.output or 'data/globe_test.csv'
print(f'Generating {args.rows} globe rows...')
print('Progress: ', end='', flush=True)
df = generate_globe_test_data(args.rows)
save_csv(df, output)
# Globe-specific stats
plus_cnt = df.filter(pl.col(':ips.latd') == '+').height
blank_cnt = df.filter(pl.col(':ips.lond') == '').height
print(f'\nGlobe data stats:')
print(f' :ips.latd "+": {plus_cnt}/{len(df)} ({plus_cnt/len(df)*100:.1f}%)')
print(f' :ips.lond "": {blank_cnt}/{len(df)} ({blank_cnt/len(df)*100:.1f}%)')
else:
output = args.output or 'data/test_flows.csv'
print(f'Generating {args.rows} rows (A:60% Normal, B:20% Proxy, C:20% Malicious)...')
print('Progress: ', end='', flush=True)
df = generate_tls_flows(args.rows)
save_csv(df, output)
# Profile distribution
print(f'\nProfile distribution:')
for prefix, label in [('A', 'A-Normal'), ('B', 'B-Proxy'), ('C', 'C-Malicious')]:
# Profile isn't stored as a column, so estimate from IP patterns
pass
scripts/test_32b.py
+23
View File
@@ -0,0 +1,23 @@
"""Test 32B-optimized LLM orchestrator"""
import sys, os, json
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
store = SessionStore()
store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
cfg = LLMConfig(base_url='https://api.deepseek.com',
api_key='sk-360ef76d59674d6b8bc2eb160327dd39',
model='deepseek-v4-flash')
result = run_llm_pipeline('ds', config=cfg)
print(f'Steps: {result.get("steps")}')
out = result.get("result", result.get("error", ""))
print(f'Result: {str(out)[:200]}')
scripts/test_chinese_path.py
+64
View File
@@ -0,0 +1,64 @@
"""Test all functionality under Chinese directory path"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
print('1. Django import:', 'OK')
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
from analysis.session_store import SessionStore
print('2. All modules imported:', 'OK')
import asyncio
from analysis.tool_registry import handle_call
store = SessionStore()
store.drop_all()
# Load from Chinese path
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', '复杂测试.csv')
print(f'Loading: {csv}')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
print(f'3. Loaded {rc} rows from Chinese path CSV:', 'OK')
# Profile
prof = asyncio.run(handle_call('profile_data', {'dataset_id': 'ds'}))
print(f'4. Profile: {len(prof.get("column_stats",{}))} columns profiled:', 'OK' if 'error' not in prof else 'FAIL')
# Entity detection (direct call)
from analysis.entity_detector import detect_entity_column
detect_result = detect_entity_column('ds')
entity_col = detect_result.get('recommended', '')
print(f'5. Entity detection: {entity_col}:', 'OK' if entity_col else 'FAIL')
# Aggregate
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect()
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
metadata={'row_count': len(df_agg), 'csv_glob': csv})
print(f'6. Aggregation: {len(df_agg)} entities, {len(features)} features:', 'OK')
# Cluster
clust = asyncio.run(handle_call('run_clustering', {
'dataset_id': 'entity',
'cluster_columns': [c for c in features if not c.startswith('_')][:8],
'algorithm': 'hdbscan',
'params': {'min_cluster_size': 5},
'random_state': 42,
}))
print(f'7. Clustering: {clust.get("n_clusters")} clusters:', 'OK' if 'error' not in clust else f'FAIL: {clust.get("error")}')
# Extract features
cid = clust.get('cluster_result_id', '')
if cid and 'error' not in clust:
feats = asyncio.run(handle_call('extract_features', {
'dataset_id': 'entity',
'cluster_result_id': cid,
'top_k': 10, 'method': 'zscore', 'save_to_db': True,
}))
print(f'8. Feature extraction: {feats.get("n_features")} features db_saved={feats.get("db_saved")}:', 'OK' if feats.get('db_saved') else 'FAIL')
print('\n=== ALL TESTS PASSED UNDER CHINESE PATH ===')
scripts/test_e2e_full.py
+128
View File
@@ -0,0 +1,128 @@
"""天璇完整流程端到端测试"""
import sys, os, importlib
# 项目根目录 = 脚本目录的父目录
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
OK = '[OK]'
FAIL = '[!!]'
results = []
def check(name, ok, detail=""):
icon = OK if ok else FAIL
results.append((name, ok))
print(f' {icon} {name} {detail}')
print('=' * 60)
print('天璇 完整流程端到端测试')
print('=' * 60)
# ── 1. 生成复杂测试数据 ──────────────────────────────────
print('\n[1/7] 生成测试数据')
import importlib.util
spec = importlib.util.spec_from_file_location('gen_complex_test',
os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'scripts', 'gen_complex_test.py'))
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
df = mod.gen_flows(n_rows=2000)
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'e2e_full.csv')
df.write_csv(csv_path, null_value='')
check('gen_complex_test', os.path.exists(csv_path), f'rows={len(df)} cols={len(df.columns)}')
# ── 2. 加载 + 清洗 ────────────────────────────────────────
print('\n[2/7] 加载 CSV + 自动清洗')
from analysis.data_loader import load_csv_directory
from analysis.session_store import SessionStore
store = SessionStore()
store.drop_all()
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
# 检查旧经纬度列是否被丢弃
dropped_lat = not any('latitude' in c.lower() or 'longitude' in c.lower() for c in schema)
check('丢弃旧经纬度列', dropped_lat, f'columns={len(schema)}')
# 检查 HEX 列是否被预处理
schema_names = list(schema.keys())
check('schema 有效', len(schema_names) > 10, f'{len(schema_names)} columns')
# ── 3. 数据校验 ────────────────────────────────────────────
print('\n[3/7] 数据校验')
from analysis.data_validator import validate
v_result = validate('ds')
n_risks = len(v_result.get('risks', []))
check('数据校验完成', v_result.get('valid') is not None, f'valid={v_result["valid"]} risks={n_risks}')
# ── 4. 实体检测 ────────────────────────────────────────────
print('\n[4/7] 实体检测')
from analysis.entity_detector import detect_entity_column
detect_result = detect_entity_column('ds')
entity_col = detect_result.get('recommended', '')
n_candidates = len(detect_result.get('candidates', []))
check('实体检测', bool(entity_col), f'entity={entity_col} candidates={n_candidates}')
# ── 5. 类型分类 + 聚合 ─────────────────────────────────────
print('\n[5/7] 类型感知聚合')
from analysis.type_classifier import classify_schema, DataType
from analysis.entity_aggregator import aggregate_by_entity
# 类型分类
type_map = classify_schema(lf.head(500))
types_found = set(t.name for t in type_map.values())
check('类型分类', len(type_map) > 0, f'types={types_found}')
# 聚合
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect()
entity_count = len(df_agg)
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
metadata={'row_count': entity_count, 'entity_column': entity_col})
check('实体聚合', entity_count > 0, f'{entity_count} entities, {len(features)} features')
# 检查 GeoIP 和类型特征
check('有特征列', len(features) > 5, f'features={features[:8]}')
# ── 6. 聚类 + 特征提取 ────────────────────────────────────
print('\n[6/7] 聚类 + 特征提取')
import asyncio
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
feature_cols = [c for c in features if not c.startswith('_')][:10]
clust_result = asyncio.run(_handle_run_clustering(
dataset_id='entity', cluster_columns=feature_cols,
algorithm='hdbscan', params={}, random_state=42
))
n_clusters = clust_result.get('n_clusters', 0)
check('HDBSCAN 聚类', n_clusters >= 0, f'n_clusters={n_clusters}')
if 'error' not in clust_result:
quality = clust_result.get('quality_metrics', {})
check('聚类质量', quality.get('silhouette_score') is not None, f'silhouette={quality.get("silhouette_score")}')
cid = clust_result.get('cluster_result_id', '')
if cid and 'error' not in clust_result:
feat_result = asyncio.run(_handle_extract_features(
dataset_id='entity', cluster_result_id=cid,
top_k=10, method='zscore', save_to_db=True
))
n_feats = feat_result.get('n_features', 0)
check('特征提取 + DB持久化', feat_result.get('db_saved'), f'{n_feats} features')
# ── 7. 验证 ORM 数据 ──────────────────────────────────────
print('\n[7/7] ORM 数据验证')
from analysis.models import ClusterResult, ClusterFeature, EntityProfile
n_clusters_db = ClusterResult.objects.count()
n_features_db = ClusterFeature.objects.count()
n_entities_db = EntityProfile.objects.count()
check('ClusterResult 表', n_clusters_db > 0, f'{n_clusters_db} records')
check('ClusterFeature 表', n_features_db > 0, f'{n_features_db} records')
check('EntityProfile 表', n_entities_db >= 0, f'{n_entities_db} records')
# ── 结论 ──────────────────────────────────────────────────
print('\n' + '=' * 60)
passed = sum(1 for _, ok in results if ok)
total = len(results)
print(f'\n结果: {passed}/{total} 通过')
print('=' * 60)
scripts/test_llm_full.py
+72
View File
@@ -0,0 +1,72 @@
"""Test LLM basic chat and orchestrator"""
import sys, os, json, urllib.request
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
# Test 1: Basic chat
print('=== Test 1: Basic chat ===')
payload = json.dumps({
'model': 'deepseek-chat',
'messages': [
{'role': 'system', 'content': 'You are a data analysis assistant.'},
{'role': 'user', 'content': 'Profile dataset ds_001.'}
],
'max_tokens': 200,
}).encode()
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
headers={
'Content-Type': 'application/json',
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39',
},
method='POST')
with urllib.request.urlopen(req, timeout=15) as resp:
data = json.loads(resp.read())
print('Basic chat OK')
print('Response:', data['choices'][0]['message']['content'][:200])
# Test 2: LLM orchestrator
print('\n=== Test 2: LLM orchestrator ===')
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
import asyncio
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
store = SessionStore(); store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
cfg = LLMConfig(
base_url='https://api.deepseek.com',
api_key='sk-360ef76d59674d6b8bc2eb160327dd39',
model='deepseek-chat',
)
result = run_llm_pipeline('ds', config=cfg)
print('Orchestrator result:', json.dumps(result, ensure_ascii=False, indent=2)[:500])
# Test 3: Manual step-by-step (fallback if LLM fails)
if 'error' in result:
print('\n=== Test 3: Manual pipeline (fallback) ===')
entity_col = detect_entity_column('ds')['recommended']
print(f'Entity col: {entity_col}')
from analysis.entity_aggregator import aggregate_by_entity
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect()
print(f'Aggregated: {len(df_agg)} entities, {len(features)} features')
store.store_dataset('entity_data', agg_lf, schema=dict(zip(features, ['']*len(features))),
metadata={'row_count': len(df_agg), 'csv_glob': csv})
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
cluster_columns=[c for c in features if not c.startswith('_')][:8],
algorithm='hdbscan', params={}, random_state=42))
print(f'Clusters: {clust_result.get("n_clusters")}')
cid = clust_result.get('cluster_result_id', '')
if cid and 'error' not in clust_result:
feat_result = asyncio.run(_handle_extract_features(
dataset_id='entity_data', cluster_result_id=cid,
top_k=10, method='zscore', save_to_db=True))
print(f'Features: {feat_result.get("n_features")} db_saved={feat_result.get("db_saved")}')
scripts/test_orch.py
+64
View File
@@ -0,0 +1,64 @@
"""Test full LLM orchestrator with actual tools"""
import sys, os, json, asyncio
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.tool_registry import handle_call
import urllib.request
store = SessionStore()
store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
TOOLS = [{"type": "function", "function": {
"name": "profile_data", "description": "Get column statistics",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}
}}, {"type": "function", "function": {
"name": "build_entity_profiles", "description": "Auto-detect entity and aggregate",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "auto_detect": {"type": "boolean"}}, "required": ["dataset_id"]}
}}, {"type": "function", "function": {
"name": "run_clustering", "description": "Run HDBSCAN clustering",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_columns": {"type": "array", "items": {"type": "string"}}, "algorithm": {"type": "string", "enum": ["hdbscan", "kmeans"]}}, "required": ["dataset_id", "cluster_columns"]}
}}, {"type": "function", "function": {
"name": "extract_features", "description": "Extract distinguishing features",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_result_id": {"type": "string"}, "method": {"type": "string", "enum": ["zscore", "anova"]}}, "required": ["dataset_id", "cluster_result_id"]}
}}]
def call_llm(messages):
payload = json.dumps({"model": "deepseek-v4-flash", "messages": messages, "tools": TOOLS,
"tool_choice": "auto", "max_tokens": 4096}).encode()
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
headers={'Content-Type': 'application/json',
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST')
return json.loads(urllib.request.urlopen(req, timeout=30).read())
msgs = [{"role": "user", "content": "Analyze dataset ds. Profile it, build entity profiles, cluster, extract features."}]
for step in range(10):
print(f'\nStep {step + 1}:')
response = call_llm(msgs)
msg = response["choices"][0]["message"]
if "tool_calls" not in msg or not msg["tool_calls"]:
print(f'Done: {msg.get("content", "")[:200]}')
break
for tc in msg["tool_calls"]:
func = tc.get("function", {})
name = func.get("name", "")
try:
args = json.loads(func.get("arguments", "{}"))
except json.JSONDecodeError:
args = {}
print(f' -> {name}({args})')
result = asyncio.run(handle_call(name, args))
content = json.dumps(result, default=str, ensure_ascii=False)[:2000]
print(f' <- {name}: ok' if 'error' not in result else f' <- ERROR: {result["error"]}')
msgs.append({"role": "tool", "tool_call_id": tc["id"], "content": content})
scripts/test_orch2.py
+49
View File
@@ -0,0 +1,49 @@
"""Debug LLM orchestrator 400 on second call"""
import sys, os, json, asyncio
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.tool_registry import handle_call
import urllib.request
store = SessionStore(); store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
TOOLS = [{"type": "function", "function": {"name": "profile_data",
"description": "Get column statistics",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}}}]
# Step 1: Get tool call from LLM
msgs = [{"role": "user", "content": "Call profile_data with ds"}]
payload = json.dumps({"model": "deepseek-v4-flash", "messages": msgs, "tools": TOOLS,
"tool_choice": "auto", "max_tokens": 200}).encode()
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
headers={'Content-Type': 'application/json', 'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST')
resp = json.loads(urllib.request.urlopen(req, timeout=10).read())
msg = resp["choices"][0]["message"]
tc = msg["tool_calls"][0]
print(f'Got tool call: {tc["function"]["name"]}')
# Execute tool
result = asyncio.run(handle_call(tc["function"]["name"], json.loads(tc["function"]["arguments"])))
content = json.dumps(result, default=str, ensure_ascii=False)
print(f'Result length: {len(content)}')
# Step 2: Send back - with correct message format
msgs2 = msgs + [msg, {"role": "tool", "tool_call_id": tc["id"], "content": content[:2000]}]
payload2 = json.dumps({"model": "deepseek-v4-flash", "messages": msgs2, "tools": TOOLS,
"tool_choice": "auto", "max_tokens": 200}).encode()
req2 = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload2,
headers={'Content-Type': 'application/json', 'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST')
try:
resp2 = json.loads(urllib.request.urlopen(req2, timeout=10).read())
print(f'Step 2 OK: {resp2["choices"][0]["message"]["content"][:100]}')
except urllib.error.HTTPError as e:
body = e.read().decode()[:500]
print(f'Step 2 400: {body}')
scripts/test_pca.py
+51
View File
@@ -0,0 +1,51 @@
"""Test PCA embedding in _handle_extract_features"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
import asyncio
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
store = SessionStore(); store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
entity_col = detect_entity_column('ds')['recommended']
print(f'Entity col: {entity_col}')
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
df_agg = agg_lf.collect()
store.store_dataset('entity_data', agg_lf, schema=dict(zip(features, ['']*len(features))),
metadata={'row_count': len(df_agg), 'csv_glob': csv})
print(f'Aggregated: {len(df_agg)} rows, {len(features)} features')
# Create a dummy AnalysisRun so PCA embedding can save EntityProfiles
from analysis.models import AnalysisRun
run_for_pca, _ = AnalysisRun.objects.get_or_create(csv_glob=csv, defaults={'status': 'running'})
run_for_pca.status = 'running'
run_for_pca.save()
print(f'AnalysisRun: #{run_for_pca.id}')
clust = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
cluster_columns=[c for c in features if not c.startswith('_')][:8],
algorithm='hdbscan', params={}, random_state=42))
cid = clust.get('cluster_result_id', '')
print(f'Cluster result: {cid}')
feat = asyncio.run(_handle_extract_features(dataset_id='entity_data', cluster_result_id=cid,
top_k=10, method='zscore', save_to_db=True))
print(f'Result keys: {list(feat.keys())}')
if 'error' in feat:
print(f'ERROR: {feat["error"]}')
print(f'n_features={feat.get("n_features")} db_saved={feat.get("db_saved")}')
from analysis.models import EntityProfile
cnt = EntityProfile.objects.exclude(embedding_x=None).count()
total = EntityProfile.objects.count()
print(f'EntityProfile with PCA coords: {cnt}/{total}')
if cnt > 0:
ep = EntityProfile.objects.exclude(embedding_x=None).first()
print(f'Sample: {ep.entity_value} ({ep.embedding_x}, {ep.embedding_y}) cluster={ep.cluster_label}')
scripts/test_profile.py
+24
View File
@@ -0,0 +1,24 @@
"""Test profile_data fix"""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
import django; django.setup()
import asyncio
from analysis.session_store import SessionStore
from analysis.data_loader import load_csv_directory
from analysis.tool_registry import handle_call
store = SessionStore()
store.drop_all()
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
lf, schema, rc, fc, mem = load_csv_directory(csv)
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
result = asyncio.run(handle_call('profile_data', {'dataset_id': 'ds'}))
if 'error' in result:
print(f'FAIL: {result["error"]}')
else:
cols = result.get('column_stats', {})
print(f'OK: {len(cols)} columns profiled')
for name, stats in list(cols.items())[:3]:
print(f' {name}: dtype={stats.get("dtype")} mean={stats.get("mean","-")}')
scripts/test_tool_format.py
+48
View File
@@ -0,0 +1,48 @@
"""Test DeepSeek tool calling with follow-up"""
import json, urllib.request
def chat(messages, tools=None):
payload = {"model": "deepseek-v4-flash", "messages": messages, "max_tokens": 200}
if tools:
payload["tools"] = tools
payload["tool_choice"] = "auto"
req = urllib.request.Request(
'https://api.deepseek.com/chat/completions',
data=json.dumps(payload).encode(),
headers={'Content-Type': 'application/json',
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST'
)
with urllib.request.urlopen(req, timeout=15) as resp:
return json.loads(resp.read())
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get weather",
"parameters": {"type": "object", "properties": {"loc": {"type": "string"}}, "required": ["loc"]}
}}]
# Call 1: LLM returns tool_call
msgs = [{"role": "user", "content": "Get weather for Beijing"}]
resp1 = chat(msgs, tools)
msg1 = resp1["choices"][0]["message"]
print(f'Call 1: tool_calls={bool(msg1.get("tool_calls"))}')
if msg1.get("tool_calls"):
tc = msg1["tool_calls"][0]
print(f' tool={tc["function"]["name"]} args={tc["function"]["arguments"]}')
# Call 2: Send tool result back (test different formats)
print('\nTest 1: with "name" field:')
msgs2 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "name": tc["function"]["name"], "content": '{"temp": 22}'}]
try:
resp2 = chat(msgs2)
print(f' OK: {resp2["choices"][0]["message"]["content"][:100]}')
except urllib.error.HTTPError as e:
print(f' 400: {e.read().decode()[:200]}')
print('\nTest 2: without "name" field:')
msgs3 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "content": '{"temp": 22}'}]
try:
resp3 = chat(msgs3)
print(f' OK: {resp3["choices"][0]["message"]["content"][:100]}')
except urllib.error.HTTPError as e:
print(f' 400: {e.read().decode()[:200]}')
scripts/test_tool_loop.py
+43
View File
@@ -0,0 +1,43 @@
"""Test tool calling loop with DeepSeek"""
import json, urllib.request
tools = [{"type": "function", "function": {
"name": "profile_data", "description": "Profile dataset",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}
}}]
def call(messages):
payload = json.dumps({"model": "deepseek-v4-flash", "messages": messages, "tools": tools,
"tool_choice": "auto", "max_tokens": 200}).encode()
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
headers={'Content-Type': 'application/json',
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST')
return json.loads(urllib.request.urlopen(req, timeout=10).read())
# Call 1: Get tool call
msgs = [{"role": "user", "content": "Call profile_data with ds"}]
r1 = call(msgs)
msg1 = r1["choices"][0]["message"]
tc = msg1["tool_calls"][0]
print(f'Call 1: id={tc["id"][:20]} tool={tc["function"]["name"]}')
# Call 2: Send result back (with tools still in payload)
msgs2 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "content": '{"status": "ok"}'}]
try:
r2 = call(msgs2)
print(f'Call 2: {r2["choices"][0]["message"]["content"][:100]}')
except urllib.error.HTTPError as e:
print(f'Call 2 400: body={e.read().decode()[:300]}')
# Call 3: Without tools in payload
try:
payload3 = json.dumps({"model": "deepseek-v4-flash", "messages": msgs2, "max_tokens": 200}).encode()
req3 = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload3,
headers={'Content-Type': 'application/json',
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
method='POST')
r3 = json.loads(urllib.request.urlopen(req3, timeout=10).read())
print(f'Call 3 (no tools): {r3["choices"][0]["message"]["content"][:100]}')
except urllib.error.HTTPError as e:
print(f'Call 3 400: body={e.read().decode()[:300]}')
scripts/verify_runtime.py
+42
View File
@@ -0,0 +1,42 @@
"""
天璇运行时隔离验证脚本。在不同机器上运行，输出应完全一致。
"""
import sys, site, importlib
ok = True
# 1. 检查 Python 路径
print(f'Python: {sys.version}')
print(f'base_exec_prefix: {sys.base_exec_prefix}')
if '天璇' not in sys.base_exec_prefix and 'runtime' not in sys.base_exec_prefix:
print('WARN: base_exec_prefix does not point to runtime Python')
# This is expected when running via uv, so not a failure
# 2. 检查 site-packages 隔离
print(f'ENABLE_USER_SITE: {site.ENABLE_USER_SITE}')
# 3. 检查核心依赖版本
checks = [
('django', '4.2'),
('polars', '1.42'),
('sklearn', '1.5'),
('numpy', '1.26'),
('yaml', ''),
('pydantic', '2.'),
]
for mod_name, ver_prefix in checks:
try:
mod = importlib.import_module(mod_name)
ver = getattr(mod, '__version__', 'unknown')
match = 'YES' if not ver_prefix or ver.startswith(ver_prefix) else f'NEEDS: {ver_prefix}xxx, GOT: {ver}'
print(f' {mod_name:15s} {ver:20s} {match}')
except Exception as e:
print(f' {mod_name:15s} FAIL: {e}')
ok = False
# 4. 检查字符编码
import sys
print(f'defaultencoding: {sys.getdefaultencoding()}')
print(f'filesystemencoding: {sys.getfilesystemencoding()}')
print(f'\nOverall: {"PASS" if ok else "FAIL"}')
scripts/wnl_to_user_csv.py
+418
View File
@@ -0,0 +1,418 @@
#!/usr/bin/env python
"""Convert WNL QoS TLS dataset to the user's 44-column CSV format.
The WNL dataset (tab-separated) has columns:
SNI\tservice_label\tPKT_1_payload\tPKT_2_payload
PKT_1_payload contains TLS record bytes as comma-separated decimal ints.
This script parses the TLS ClientHello and generates the user's 44-column
format, appending the ground-truth label as the last column.
Usage:
runtime\python\python.exe scripts\wnl_to_user_csv.py
"""
import csv
import os
import random
import struct
from datetime import datetime, timedelta
from pathlib import Path
random.seed(42)
# ── Paths ──────────────────────────────────────────────────────────────────
BASE = Path(r"C:\Users\25044\Desktop\Proj")
WNL_CSV = BASE / "qos-tls-dataset-of-enc-traffic" / "Preprocessed_CSV" / "Preprocessed_CSV" / "WNL_TLS_Dataset_Origin.csv"
OUT_CSV = BASE / "天璇" / "data" / "wnl_converted.csv"
OUT_CSV.parent.mkdir(parents=True, exist_ok=True)
# ── User 44-column header ──────────────────────────────────────────────────
USER_HEADER = [
":ips", ":ipd", ":prs", ":prd", "scnt", "dcnt",
"server-ip", "client-ip",
"0ver", "snam", "cnam",
"4dur", "8ses", "2tmo", "4ksz",
"cnrs", "isrs",
"8ack", "8ppk", "8dbd", "1ipp",
"4dbn", "tabl", "name", "source-node",
"cipher-suite", "ecdhe-named-curve", "0cph", "0crv",
"0rnd", "0rnt",
"row", "time", "timestamp",
":ips.latd", ":ips.lond", ":ipd.latd", ":ipd.lond",
":ips.ispn", ":ipd.ispn", ":ips.orgn", ":ipd.orgn",
":ips.city", ":ipd.city",
]
# ── Cipher suite hex → IANA name map ──────────────────────────────────────
CIPHER_HEX_MAP = {
0x1301: "TLS_AES_128_GCM_SHA256",
0x1302: "TLS_AES_256_GCM_SHA384",
0x1303: "TLS_CHACHA20_POLY1305_SHA256",
0xC02B: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
0xC02C: "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
0xC02F: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
0xC030: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
0xCCA9: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
0xCCAC: "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
0x009C: "TLS_RSA_WITH_AES_128_GCM_SHA256",
0x009D: "TLS_RSA_WITH_AES_256_GCM_SHA384",
0x003C: "TLS_RSA_WITH_AES_128_CBC_SHA256",
0x003D: "TLS_RSA_WITH_AES_256_CBC_SHA256",
0xC009: "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
0xC013: "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
0x002F: "TLS_RSA_WITH_AES_128_CBC_SHA",
0x0035: "TLS_RSA_WITH_AES_128_CBC_SHA",
0x0005: "TLS_RSA_WITH_RC4_128_SHA",
0x00FF: "TLS_RSA_WITH_NULL_MD5",
}
# ── Named curve hex → name map ────────────────────────────────────────────
CURVE_HEX_MAP = {
0x0017: "secp256r1",
0x0018: "secp384r1",
0x0019: "secp521r1",
0x001D: "x25519",
0x001E: "x448",
}
# ── Service class → IP / geo / duration profiles ──────────────────────────
SERVICE_PROFILES = {
"Web": {"ip8": "10", "cnt": "US", "dur_range": (0.5, 15), "isp": "Cloudflare", "org": "Cloudflare Inc.", "city": "San Francisco"},
"Spotify": {"ip8": "20", "cnt": "SE", "dur_range": (30, 600), "isp": "Telia", "org": "Spotify AB", "city": "Stockholm"},
"Netflix": {"ip8": "30", "cnt": "US", "dur_range": (60, 1200), "isp": "Comcast", "org": "Netflix Inc.", "city": "Los Gatos"},
"YouTube": {"ip8": "40", "cnt": "US", "dur_range": (30, 900), "isp": "Google Fiber", "org": "Google LLC", "city": "Mountain View"},
"AppleMusic": {"ip8": "50", "cnt": "US", "dur_range": (30, 600), "isp": "AT&T", "org": "Apple Inc.", "city": "Cupertino"},
"SoundCloud": {"ip8": "60", "cnt": "DE", "dur_range": (30, 500), "isp": "Deutsche Telekom", "org": "SoundCloud Ltd.", "city": "Berlin"},
"PrimeVideo": {"ip8": "70", "cnt": "US", "dur_range": (60, 1200), "isp": "Amazon", "org": "Amazon.com Inc.", "city": "Seattle"},
"LiveYouTube": {"ip8": "80", "cnt": "US", "dur_range": (60, 1800), "isp": "Google Fiber", "org": "Google LLC", "city": "Mountain View"},
"LiveFacebook": {"ip8": "90", "cnt": "US", "dur_range": (60, 1800), "isp": "Facebook", "org": "Meta Platforms Inc.", "city": "Menlo Park"},
"Kinopoisk": {"ip8": "100", "cnt": "RU", "dur_range": (30, 900), "isp": "Rostelecom", "org": "Yandex LLC", "city": "Moscow"},
"YandexMusic": {"ip8": "110", "cnt": "RU", "dur_range": (30, 600), "isp": "Rostelecom", "org": "Yandex LLC", "city": "Moscow"},
"Vimeo": {"ip8": "120", "cnt": "US", "dur_range": (30, 600), "isp": "Akamai", "org": "Vimeo Inc.", "city": "New York"},
}
# ── TLS version hex → display ─────────────────────────────────────────────
TLS_VER_MAP = {
(3, 4): "03 04",
(3, 3): "03 03",
(3, 2): "03 02",
(3, 1): "03 01",
}
def parse_tls_clienthello(payload_bytes: bytes) -> dict:
"""Parse TLS ClientHello from raw bytes, returning extracted fields.
Returns dict with keys:
record_version, ch_version, cipher_suites, cipher_suite_hex,
sni, supported_groups
Missing/parse-failure values are empty strings or empty lists.
"""
result = {
"record_version": "",
"ch_version": "",
"cipher_suites": [],
"cipher_suite_hex": "",
"sni": "",
"supported_groups": [],
"curves_hex": "",
}
if len(payload_bytes) < 5:
return result
content_type = payload_bytes[0]
if content_type != 22: # Handshake
return result
# Record layer version
rv_major = payload_bytes[1]
rv_minor = payload_bytes[2]
result["record_version"] = TLS_VER_MAP.get((rv_major, rv_minor), f"{rv_major:02x} {rv_minor:02x}")
if len(payload_bytes) < 6:
return result
hs_type = payload_bytes[5]
if hs_type != 1: # ClientHello
return result
# ClientHello version at offset 9-10
if len(payload_bytes) < 11:
return result
ch_major = payload_bytes[9]
ch_minor = payload_bytes[10]
result["ch_version"] = TLS_VER_MAP.get((ch_major, ch_minor), f"{ch_major:02x} {ch_minor:02x}")
# Session ID length at offset 43
if len(payload_bytes) < 44:
return result
sess_len = payload_bytes[43]
offset = 44 + sess_len
# Cipher suites: 2 bytes length, then N bytes (2-byte each)
if len(payload_bytes) < offset + 2:
return result
cs_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
offset += 2
cs_end = offset + cs_len
if cs_end > len(payload_bytes):
cs_end = len(payload_bytes)
cipher_suite_hex_list = []
for i in range(offset, cs_end, 2):
if i + 1 < cs_end:
cs_val = (payload_bytes[i] << 8) | payload_bytes[i + 1]
cs_hex = f"{payload_bytes[i]:02x} {payload_bytes[i+1]:02x}"
cipher_suite_hex_list.append(cs_hex)
result["cipher_suites"].append(cs_val)
result["cipher_suite_hex"] = cipher_suite_hex_list[0] if cipher_suite_hex_list else ""
offset = cs_end
# Compression methods
if offset >= len(payload_bytes):
return result
comp_len = payload_bytes[offset]
offset += 1 + comp_len
# Extensions
if offset + 1 >= len(payload_bytes):
return result
ext_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
offset += 2
ext_end = offset + ext_len
if ext_end > len(payload_bytes):
ext_end = len(payload_bytes)
while offset + 3 < ext_end:
ext_type = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
ext_data_len = (payload_bytes[offset + 2] << 8) | payload_bytes[offset + 3]
offset += 4
ext_data_end = offset + ext_data_len
if ext_data_end > ext_end:
ext_data_end = ext_end
if ext_type == 0 and ext_data_len > 5: # server_name
# Skip list length (2) + type (1) + name length (2)
sni_name_len = (payload_bytes[offset + 3] << 8) | payload_bytes[offset + 4]
if offset + 5 + sni_name_len <= ext_data_end:
sni_bytes = payload_bytes[offset + 5: offset + 5 + sni_name_len]
try:
result["sni"] = sni_bytes.decode("ascii", errors="replace")
except Exception:
result["sni"] = sni_bytes.decode("ascii", errors="replace")
elif ext_type == 10 and ext_data_len > 1: # supported_groups
groups_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
groups_hex = []
for gi in range(0, groups_len, 2):
idx = offset + 2 + gi
if idx + 1 < ext_data_end:
gv = (payload_bytes[idx] << 8) | payload_bytes[idx + 1]
groups_hex.append(f"{payload_bytes[idx]:02x} {payload_bytes[idx+1]:02x}")
result["supported_groups"].append(gv)
result["curves_hex"] = groups_hex[0] if groups_hex else ""
offset = ext_data_end
return result
def make_ip(ip8: str, idx: int) -> str:
"""Generate a plausible IP from a /8 prefix."""
return f"{ip8}.{random.randint(1, 254)}.{random.randint(1, 254)}.{random.randint(1, 254)}"
def make_server_ip(cnt: str) -> str:
"""Generate a server IP that looks like a CDN/cloud provider."""
prefixes = {
"US": "203.0.113", # example/documentation range
"SE": "198.51.100",
"DE": "203.0.114",
"RU": "198.52.100",
"GB": "203.0.115",
}
p = prefixes.get(cnt, "203.0.113")
return f"{p}.{random.randint(1, 254)}"
def main():
print(f"[1/3] 读取 WNL 数据集: {WNL_CSV}")
# Read the origin file and parse all rows
rows_data = [] # list of (sni, service_label, pkt1_bytes)
with open(WNL_CSV, encoding="utf-8") as f:
# CSV with tab delimiter
reader = csv.reader(f, delimiter="\t")
headers = next(reader)
print(f" Headers: {headers}")
for row in reader:
if not row or len(row) < 4:
continue
sni = row[0].strip()
label = row[1].strip()
# PKT_1_payload is comma-separated decimal ints
pkt1_str = row[2].strip()
if not pkt1_str:
continue
try:
pkt1_bytes = bytes(int(b.strip()) for b in pkt1_str.split(",") if b.strip())
except ValueError:
continue
rows_data.append((sni, label, pkt1_bytes))
print(f" 总共 {len(rows_data)} 行")
# Count labels
label_counts = {}
for _, lbl, _ in rows_data:
label_counts[lbl] = label_counts.get(lbl, 0) + 1
print(f" Service classes ({len(label_counts)}):")
for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]):
print(f" {lbl}: {cnt}")
print(f"\n[2/3] 解析 TLS ClientHello 并生成 44 列格式")
# Generate output
out_rows = []
parse_errors = 0
for idx, (sni, label, pkt1_bytes) in enumerate(rows_data):
tls_info = parse_tls_clienthello(pkt1_bytes)
if not tls_info["record_version"]:
parse_errors += 1
prof = SERVICE_PROFILES.get(label, SERVICE_PROFILES["Web"])
ip8 = prof["ip8"]
cnt = prof["cnt"]
dur_range = prof["dur_range"]
isp = prof["isp"]
org = prof["org"]
city = prof["city"]
# Synthetic IPs
src_ip = make_ip(ip8, idx) # client IP (:ips)
dst_ip = make_server_ip(cnt) # server IP (:ipd)
# TLS version: prefer CH version, fallback to record version
tls_ver = tls_info["ch_version"] or tls_info["record_version"] or "03 03"
# Cipher suite
cipher_suites = tls_info["cipher_suites"]
cipher_suite_hex = tls_info["cipher_suite_hex"]
cipher_suite_name = ""
if cipher_suites:
cs_val = cipher_suites[0]
cipher_suite_name = CIPHER_HEX_MAP.get(cs_val, f"UNKNOWN_0x{cs_val:04X}")
else:
# Default for TLS 1.3
cipher_suite_hex = "13 01"
cipher_suite_name = "TLS_AES_128_GCM_SHA256"
# Named curve
supported_groups = tls_info["supported_groups"]
curves_hex = tls_info["curves_hex"]
curve_name = ""
if supported_groups:
cg_val = supported_groups[0]
curve_name = CURVE_HEX_MAP.get(cg_val, f"unknown_curve_0x{cg_val:04X}")
else:
curves_hex = "00 1d" # default x25519
curve_name = "x25519"
# SNI: use parsed or the raw SNI column
sni_val = tls_info.get("sni") or sni
# Duration based on service class
duration = round(random.uniform(*dur_range), 2)
# Random values for synthetic columns
rand_bytes = random.randint(1000, 200000)
rand_packets = random.randint(5, 100)
ses_val = round(random.uniform(100, 1000), 4)
tmo_val = round(random.uniform(100, 500), 4)
ksz_val = random.choice([128, 256, 384, 512])
cnrs_val = random.choice(["True", ""])
isrs_val = random.choice(["True", ""])
# Timestamp: base it on row index spread over a month
base_dt = datetime(2024, 6, 1, 0, 0, 0)
dt = base_dt + timedelta(seconds=random.randint(0, 30 * 24 * 3600))
ts_str = dt.strftime("%Y-%m-%d %H:%M:%S")
unix_ts = int(dt.timestamp())
# Random hex random bytes
rnd_hex = " ".join(f"{random.randint(0, 255):02x}" for _ in range(32))
row_data = [
src_ip, # :ips
dst_ip, # :ipd
str(random.randint(40000, 65000)), # :prs (source port)
"443", # :prd (destination port, typically 443)
"US", # scnt (source country)
cnt, # dcnt (destination country)
dst_ip, # server-ip
src_ip, # client-ip
tls_ver, # 0ver
sni_val, # snam
sni_val, # cnam
str(duration), # 4dur
str(ses_val), # 8ses
str(tmo_val), # 2tmo
str(ksz_val), # 4ksz
cnrs_val, # cnrs
isrs_val, # isrs
str(rand_bytes), # 8ack
str(rand_packets), # 8ppk
ts_str, # 8dbd
"6", # 1ipp (TCP)
"packet_capture", # 4dbn
"wan-link", # tabl
"wnl_tls", # name
"gw-01", # source-node
cipher_suite_name, # cipher-suite
curve_name, # ecdhe-named-curve
cipher_suite_hex, # 0cph
curves_hex, # 0crv
rnd_hex, # 0rnd
str(unix_ts), # 0rnt
str(idx + 1), # row
dt.strftime("%H:%M:%S"), # time
ts_str, # timestamp
"", # :ips.latd (leave blank)
"", # :ips.lond (leave blank)
"", # :ipd.latd (leave blank)
"", # :ipd.lond (leave blank)
"", # :ips.ispn (leave blank)
"", # :ipd.ispn (leave blank)
"", # :ips.orgn (leave blank)
"", # :ipd.orgn (leave blank)
"", # :ips.city (leave blank)
"", # :ipd.city (leave blank)
]
# Append label as the LAST column
row_data.append(label)
out_rows.append(row_data)
print(f" TLS 解析失败: {parse_errors}/{len(rows_data)}")
print(f" 成功生成: {len(out_rows)} 行")
# Write output CSV
print(f"\n[3/3] 写入输出: {OUT_CSV}")
header = USER_HEADER + ["label"]
with open(OUT_CSV, "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(header)
writer.writerows(out_rows)
print(f" 完成! 输出 {len(out_rows)} 行 x {len(header)} 列")
print(f" Columns: {header}")
if __name__ == "__main__":
main()
shell.bat
+4
View File
@@ -0,0 +1,4 @@
@echo off
cd /d "%~dp0"
runtime\python\python.exe manage.py shell
pause
static/tianxuan/earth_atmos_2048.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 MiB

static/tianxuan/favicon.svg
+10
View File
@@ -0,0 +1,10 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">
<defs>
<linearGradient id="g" x1="0" y1="0" x2="1" y2="1">
<stop offset="0%" stop-color="#4361ee"/>
<stop offset="100%" stop-color="#7b2ff7"/>
</linearGradient>
</defs>
<circle cx="32" cy="32" r="28" fill="url(#g)"/>
<text x="32" y="42" text-anchor="middle" font-size="32" font-family="sans-serif" fill="#fff">璇</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 417 B

static/tianxuan/three.min.js
Vendored
+6
View File
File diff suppressed because one or more lines are too long
static/tianxuan/world_borders.js
+11148
View File
File diff suppressed because it is too large Load Diff
templates/analysis/cluster_detail.html
+62
View File
@@ -0,0 +1,62 @@
{% extends 'base.html' %}
{% block title %}Cluster #{{ cluster.cluster_label }} - Run #{{ run.id }}{% endblock %}
{% block content %}
<div class="card">
<a href="{% url 'analysis:cluster_overview' run.id %}" style="color:#4361ee;text-decoration:none;font-size:0.9rem;">← Back to overview</a>
<h2 style="margin-top:0.5rem;">Cluster #{{ cluster.cluster_label }}</h2>
<p>{{ cluster.size }} entities ({{ cluster.proportion|floatformat:2 }}% of total)</p>
<p>Silhouette Score: {{ cluster.silhouette_score|floatformat:4|default:"-" }}</p>
</div>
<div class="grid-2">
<div class="card">
<h2>Feature Summary (top 50)</h2>
<table>
<thead>
<tr>
<th>Feature</th>
<th>Mean</th>
<th>Std</th>
<th>Median</th>
<th>Dist. Score</th>
</tr>
</thead>
<tbody>
{% for f in features %}
<tr>
<td><code>{{ f.feature_name }}</code></td>
<td>{{ f.mean|floatformat:3|default:"-" }}</td>
<td>{{ f.std|floatformat:3|default:"-" }}</td>
<td>{{ f.median|floatformat:3|default:"-" }}</td>
<td>{{ f.distinguishing_score|floatformat:3|default:"-" }}</td>
</tr>
{% empty %}
<tr><td colspan="5">No features</td></tr>
{% endfor %}
</tbody>
</table>
</div>
<div class="card">
<h2>Entities (top 50)</h2>
<table>
<thead>
<tr>
<th>Entity</th>
<th></th>
</tr>
</thead>
<tbody>
{% for e in entities %}
<tr>
<td><code>{{ e.entity_value }}</code></td>
<td><a href="{% url 'analysis:entity_profile' e.id %}" class="btn btn-primary">Profile</a></td>
</tr>
{% empty %}
<tr><td colspan="2">No entities</td></tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}
templates/analysis/cluster_overview.html
+286
View File
@@ -0,0 +1,286 @@
{% extends 'base.html' %}
{% load static %}
{% block title %}Cluster Overview - Run #{{ run.id }}{% endblock %}
{% block content %}
<style>
.feature-chart { width: 100%; display: block; }
#featureChartsContainer canvas { max-height: 400px; }
</style>
<div class="card">
<h2>Cluster Overview — Run #{{ run.id }}</h2>
<p>Entity column: <code>{{ run.entity_column }}</code> | {{ run.entity_count }} entities, {{ run.cluster_count }} clusters</p>
</div>
<div class="card">
<h2>PCA-2D Embedding</h2>
<canvas id="scatterChart" class="scatter-canvas"></canvas>
</div>
{% if geo_count > 0 %}
<div class="card">
<h2>地理位置分布 <span class="badge badge-info">{{ geo_count }} entities</span></h2>
{% if geo_skipped > 0 %}
<p style="color:#888;font-size:0.85rem;">{{ geo_skipped }} entities excluded (missing lat/lon)</p>
{% endif %}
<canvas id="geoChart" class="scatter-canvas"></canvas>
</div>
{% endif %}
<div class="card" id="clusterSizesCard">
<h2>Cluster Size Distribution</h2>
<canvas id="clusterSizesChart" class="scatter-canvas"></canvas>
</div>
<div class="card" id="silhouetteCard">
<h2>Silhouette Score Comparison</h2>
<canvas id="silhouetteChart" class="scatter-canvas"></canvas>
</div>
<div class="card" id="topFeaturesCard">
<h2>Top Distinguishing Features per Cluster</h2>
<div id="featureChartsContainer"></div>
</div>
<div class="grid-2">
{% for c in clusters %}
<div class="card">
<h3>Cluster #{{ c.cluster_label }} <span class="badge badge-info">{{ c.size }} entities</span></h3>
<p>Proportion: {{ c.proportion|floatformat:3 }}</p>
<p>Silhouette: {{ c.silhouette_score|floatformat:4|default:"-" }}</p>
<h4 style="margin-top:0.75rem;font-size:0.9rem;">Top Distinguishing Features</h4>
<table>
<thead>
<tr><th>Feature</th><th>Score</th><th>Mean</th></tr>
</thead>
<tbody>
{% for f in c.features.all|slice:":5" %}
<tr>
<td>{{ f.feature_name }}</td>
<td>{{ f.distinguishing_score|floatformat:3|default:"-" }}</td>
<td>{{ f.mean|floatformat:3|default:"-" }}</td>
</tr>
{% empty %}
<tr><td colspan="3">No features</td></tr>
{% endfor %}
</tbody>
</table>
<a href="{% url 'analysis:cluster_detail' run.id c.cluster_label %}" class="btn btn-primary" style="margin-top:0.75rem;">Detail</a>
</div>
{% empty %}
<div class="card">
<div class="empty-state"><p>No clusters found.</p></div>
</div>
{% endfor %}
</div>
{% if noise %}
<div class="card">
<h3>Noise (Cluster -1)</h3>
<p>{{ noise.size }} entities classified as noise</p>
</div>
{% endif %}
<script>
// Pure Canvas scatter plot — no Chart.js needed (offline compatible)
const scatterData = {{ scatter_data_json|safe }};
const colors = ['#4361ee','#f72585','#7209b7','#4cc9f0','#f8961e','#43aa8b','#f94144','#577590','#e9c46a','#9b5de5'];
function drawScatter(canvasId, data, xLabel, yLabel) {
const canvas = document.getElementById(canvasId);
if (!canvas || data.length === 0) return;
const ctx = canvas.getContext('2d');
const W = canvas.parentElement.clientWidth || 600;
const H = 400;
canvas.width = W; canvas.height = H;
const pad = { top: 30, bottom: 50, left: 60, right: 30 };
const plotW = W - pad.left - pad.right;
const plotH = H - pad.top - pad.bottom;
// Compute bounds
const xs = data.map(d => d.x); const ys = data.map(d => d.y);
const xMin = Math.min(...xs), xMax = Math.max(...xs);
const yMin = Math.min(...ys), yMax = Math.max(...ys);
const xRange = xMax - xMin || 1; const yRange = yMax - yMin || 1;
const toX = v => pad.left + (v - xMin) / xRange * plotW;
const toY = v => pad.top + plotH - (v - yMin) / yRange * plotH;
// Clear
ctx.clearRect(0, 0, W, H);
// Grid lines
ctx.strokeStyle = '#eee'; ctx.lineWidth = 1;
for (let i = 0; i <= 5; i++) {
const y = pad.top + i * plotH / 5;
ctx.beginPath(); ctx.moveTo(pad.left, y); ctx.lineTo(W - pad.right, y); ctx.stroke();
}
// Axes
ctx.strokeStyle = '#333'; ctx.lineWidth = 2;
ctx.beginPath(); ctx.moveTo(pad.left, pad.top); ctx.lineTo(pad.left, H - pad.bottom); ctx.stroke();
ctx.beginPath(); ctx.moveTo(pad.left, H - pad.bottom); ctx.lineTo(W - pad.right, H - pad.bottom); ctx.stroke();
// Labels
ctx.fillStyle = '#666'; ctx.font = '13px sans-serif'; ctx.textAlign = 'center';
ctx.fillText(xLabel, W / 2, H - 5);
ctx.save(); ctx.translate(15, H / 2); ctx.rotate(-Math.PI / 2); ctx.fillText(yLabel, 0, 0); ctx.restore();
// Points
data.forEach(d => {
const cx = toX(d.x), cy = toY(d.y);
const color = d.label === -1 ? 'rgba(150,150,150,0.5)' : colors[Math.abs(d.label) % colors.length];
const r = d.label === -1 ? 2 : 4;
ctx.beginPath(); ctx.arc(cx, cy, r, 0, Math.PI * 2);
ctx.fillStyle = color; ctx.fill();
});
// Tooltip on hover
canvas.onmousemove = function(e) {
const rect = canvas.getBoundingClientRect();
const mx = e.clientX - rect.left, my = e.clientY - rect.top;
const hit = data.find(d => Math.abs(toX(d.x) - mx) < 6 && Math.abs(toY(d.y) - my) < 6);
if (hit) {
canvas.title = `${hit.entity} (${hit.label === -1 ? 'Noise' : 'Cluster ' + hit.label})`;
}
};
// Legend
const uniqueLabels = [...new Set(data.map(d => d.label))];
let lx = W - pad.right - 120, ly = pad.top + 5;
ctx.font = '11px sans-serif'; ctx.textAlign = 'left';
uniqueLabels.forEach(label => {
const color = label === -1 ? 'rgba(150,150,150,0.8)' : colors[Math.abs(label) % colors.length];
ctx.fillStyle = color; ctx.fillRect(lx, ly, 10, 10);
ctx.fillStyle = '#333'; ctx.fillText(label === -1 ? 'Noise' : `Cluster ${label}`, lx + 15, ly + 10);
ly += 18;
});
}
drawScatter('scatterChart', scatterData, 'PCA-1', 'PCA-2');
{% if geo_count > 0 %}
const geoData = {{ geo_data_json|safe }};
drawScatter('geoChart', geoData, 'Longitude', 'Latitude');
{% endif %}
// ── Cluster size horizontal bar chart ──
const clusterSizes = {{ cluster_sizes_json|safe }};
const topFeaturesAll = {{ top_features_json|safe }};
const chartColors = ['#4361ee','#f72585','#7209b7','#4cc9f0','#f8961e','#43aa8b','#f94144','#577590','#e9c46a','#9b5de5'];
if (clusterSizes.length > 0) {
// Cluster size bar chart
(function() {
const canvas = document.getElementById('clusterSizesChart');
if (!canvas) return;
const ctx = canvas.getContext('2d');
const W = canvas.parentElement.clientWidth || 600;
const H = Math.max(200, clusterSizes.length * 40 + 60);
canvas.width = W; canvas.height = H;
const pad = { top: 20, bottom: 30, left: 80, right: 30 };
const plotW = W - pad.left - pad.right;
const plotH = H - pad.top - pad.bottom;
const maxSize = Math.max(...clusterSizes.map(d => d.size));
ctx.clearRect(0, 0, W, H);
clusterSizes.forEach((d, i) => {
const barH = Math.min(30, plotH / clusterSizes.length - 4);
const y = pad.top + i * (barH + 4);
const bw = (d.size / maxSize) * plotW;
ctx.fillStyle = chartColors[i % chartColors.length];
ctx.fillRect(pad.left, y, bw, barH);
ctx.fillStyle = '#333';
ctx.font = '12px sans-serif';
ctx.textAlign = 'right';
ctx.fillText('#' + d.label, pad.left - 5, y + barH / 2 + 4);
ctx.textAlign = 'left';
ctx.fillStyle = '#666';
ctx.fillText(d.size + ' (' + (d.size / clusterSizes.reduce((a,b) => a+b.size, 0) * 100).toFixed(1) + '%)', pad.left + bw + 5, y + barH / 2 + 4);
});
})();
// Silhouette score bar chart
(function() {
const canvas = document.getElementById('silhouetteChart');
if (!canvas) return;
const silData = clusterSizes.filter(d => d.silhouette !== null && d.silhouette !== undefined);
if (silData.length === 0) {
canvas.parentElement.style.display = 'none';
return;
}
const ctx = canvas.getContext('2d');
const W = canvas.parentElement.clientWidth || 600;
const H = Math.max(200, silData.length * 40 + 60);
canvas.width = W; canvas.height = H;
const pad = { top: 20, bottom: 30, left: 80, right: 40 };
const plotW = W - pad.left - pad.right;
const plotH = H - pad.top - pad.bottom;
const maxScore = Math.max(1, ...silData.map(d => Math.abs(d.silhouette)));
ctx.clearRect(0, 0, W, H);
silData.forEach((d, i) => {
const barH = Math.min(30, plotH / silData.length - 4);
const y = pad.top + i * (barH + 4);
const bw = (d.silhouette / maxScore) * plotW * 0.5;
const x = pad.left + plotW / 2;
ctx.fillStyle = d.silhouette >= 0 ? '#43aa8b' : '#f94144';
ctx.fillRect(x, y, d.silhouette >= 0 ? bw : -bw, barH);
ctx.fillStyle = '#333';
ctx.font = '12px sans-serif';
ctx.textAlign = 'right';
ctx.fillText('#' + d.label, pad.left - 5, y + barH / 2 + 4);
ctx.textAlign = 'left';
ctx.fillStyle = '#666';
ctx.fillText(d.silhouette.toFixed(4), d.silhouette >= 0 ? x + bw + 5 : x - bw - 40, y + barH / 2 + 4);
});
// Zero line
ctx.strokeStyle = '#999'; ctx.lineWidth = 1;
ctx.beginPath(); ctx.moveTo(pad.left + plotW / 2, pad.top); ctx.lineTo(pad.left + plotW / 2, H - pad.bottom); ctx.stroke();
ctx.fillStyle = '#999'; ctx.font = '11px sans-serif'; ctx.textAlign = 'center';
ctx.fillText('0', pad.left + plotW / 2, H - 5);
ctx.fillText('+1', W - pad.right, H - 5);
ctx.fillText('-1', pad.left, H - 5);
})();
// Top features per cluster horizontal bar charts
(function() {
const container = document.getElementById('featureChartsContainer');
if (!container) return;
clusterSizes.forEach(d => {
const label = d.label;
const features = topFeaturesAll[String(label)] || [];
if (features.length === 0) return;
const card = document.createElement('div');
card.style.marginBottom = '1.5rem';
card.innerHTML = '<h4 style="margin-bottom:0.5rem;">Cluster #' + label + ' - Top Features</h4><canvas class="feature-chart" data-label="' + label + '" style="width:100%;height:' + Math.max(120, features.length * 28 + 30) + 'px;"></canvas>';
container.appendChild(card);
const canvas = card.querySelector('canvas');
if (!canvas) return;
const ctx = canvas.getContext('2d');
const W2 = canvas.parentElement.clientWidth || 600;
const H2 = Math.max(120, features.length * 28 + 30);
canvas.width = W2; canvas.height = H2;
const pad2 = { top: 10, bottom: 20, left: 100, right: 60 };
const plotW2 = W2 - pad2.left - pad2.right;
const plotH2 = H2 - pad2.top - pad2.bottom;
const maxScore = Math.max(0.01, ...features.map(f => Math.abs(f.score || 0)));
ctx.clearRect(0, 0, W2, H2);
features.forEach((f, i) => {
const barH = Math.min(22, plotH2 / features.length - 3);
const y = pad2.top + i * (barH + 3);
const score = f.score || 0;
const bw = (score / maxScore) * plotW2;
ctx.fillStyle = chartColors[label % chartColors.length];
ctx.fillRect(pad2.left, y, bw, barH);
ctx.fillStyle = '#333';
ctx.font = '11px sans-serif';
ctx.textAlign = 'right';
const name = f.feature_name.length > 18 ? f.feature_name.slice(0, 16) + '..' : f.feature_name;
ctx.fillText(name, pad2.left - 5, y + barH / 2 + 4);
ctx.textAlign = 'left';
ctx.fillStyle = '#666';
ctx.fillText(score.toFixed(3), pad2.left + bw + 5, y + barH / 2 + 4);
});
});
})();
}
</script>
{% endblock %}
templates/analysis/dashboard.html
+54
View File
@@ -0,0 +1,54 @@
{% extends 'base.html' %}
{% block title %}首页 - 天璇{% endblock %}
{% block content %}
<div class="card">
<h2>TLS Flow Analysis Dashboard</h2>
<p style="color:#666;margin-bottom:1rem;">Upload CSV flow data, detect entities, cluster behavior profiles, and visualize results.</p>
</div>
{% if runs %}
<div class="card">
<h2>Recent Analysis Runs</h2>
<table>
<thead>
<tr>
<th>ID</th>
<th>Started</th>
<th>Status</th>
<th>Entity Column</th>
<th>Entities</th>
<th>Clusters</th>
<th></th>
</tr>
</thead>
<tbody>
{% for run in runs %}
<tr>
<td>#{{ run.id }}</td>
<td>{{ run.created_at|date:"Y-m-d H:i" }}</td>
<td>
{% if run.status == 'completed' %}
<span class="badge badge-success">Completed</span>
{% elif run.status == 'failed' %}
<span class="badge badge-danger">Failed</span>
{% else %}
<span class="badge badge-info">{{ run.get_status_display }}</span>
{% endif %}
</td>
<td>{{ run.entity_column|default:"-" }}</td>
<td>{{ run.entity_count|default:"-" }}</td>
<td>{{ run.cluster_count|default:"-" }}</td>
<td><a href="{% url 'analysis:run_detail' run.id %}" class="btn btn-primary">View</a></td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<div class="card empty-state">
<div style="font-size:3rem;">📊</div>
<p>No analysis runs yet. Use MCP tools to load data and start analysis.</p>
<p style="font-size:0.9rem;color:#bbb;margin-top:0.5rem;">Call <code>load_data()</code> via MCP to begin.</p>
</div>
{% endif %}
{% endblock %}
templates/analysis/entity_profile.html
+74
View File
@@ -0,0 +1,74 @@
{% extends 'base.html' %}
{% block title %}Entity: {{ entity.entity_value }} - TLS Analyzer{% endblock %}
{% block content %}
<div class="card">
<a href="javascript:history.back()" style="color:#4361ee;text-decoration:none;font-size:0.9rem;">← Back</a>
<h2 style="margin-top:0.5rem;">Entity: <code>{{ entity.entity_value }}</code></h2>
<p>Run #{{ run.id }} | Cluster: <span class="badge badge-info">#{{ entity.cluster_label|default:"unassigned" }}</span></p>
</div>
<div class="card">
<h2>Aggregated Features</h2>
<table>
<thead>
<tr>
<th>Feature</th>
<th>Value</th>
<th>Cluster Mean</th>
<th>Deviation (Z-score)</th>
</tr>
</thead>
<tbody>
{% for col, dev in deviations.items %}
<tr>
<td><code>{{ col }}</code></td>
<td>{{ dev.value|floatformat:3 }}</td>
<td>{{ dev.cluster_mean|floatformat:3 }}</td>
<td style="color:{% if dev.zscore > 2 %}#f72585{% elif dev.zscore < -2 %}#4361ee{% else %}#666{% endif %};">
{{ dev.zscore|floatformat:2 }}
</td>
</tr>
{% empty %}
<tr><td colspan="4">No feature data available.</td></tr>
{% endfor %}
</tbody>
</table>
</div>
{% if entity.embedding_x is not None and entity.embedding_y is not None %}
<div class="card">
<h2>PCA Position</h2>
<p>X: {{ entity.embedding_x|floatformat:3 }}, Y: {{ entity.embedding_y|floatformat:3 }}</p>
<canvas id="entityScatter" class="scatter-canvas" height="300"></canvas>
</div>
<script>
const canvas = document.getElementById('entityScatter');
if (canvas) {
const ctx = canvas.getContext('2d');
const W = canvas.parentElement.clientWidth || 400, H = 300;
canvas.width = W; canvas.height = H;
const pad = { top: 30, bottom: 40, left: 50, right: 20 };
const x = {{ entity.embedding_x }}, y = {{ entity.embedding_y }};
const margin = 0.2; // 20% padding around the point
const xRange = Math.abs(x) * margin * 2 || 1;
const yRange = Math.abs(y) * margin * 2 || 1;
const xMin = x - xRange, xMax = x + xRange;
const yMin = y - yRange, yMax = y + yRange;
const toX = v => pad.left + (v - xMin) / (xMax - xMin) * (W - pad.left - pad.right);
const toY = v => pad.top + H - pad.bottom - (v - yMin) / (yMax - yMin) * (H - pad.top - pad.bottom);
ctx.clearRect(0, 0, W, H);
ctx.strokeStyle = '#ddd'; ctx.lineWidth = 1;
ctx.beginPath(); ctx.moveTo(pad.left, toY(0)); ctx.lineTo(W - pad.right, toY(0)); ctx.stroke();
ctx.beginPath(); ctx.moveTo(toX(0), pad.top); ctx.lineTo(toX(0), H - pad.bottom); ctx.stroke();
ctx.fillStyle = '#f72585';
ctx.beginPath(); ctx.arc(toX(x), toY(y), 8, 0, Math.PI * 2); ctx.fill();
ctx.fillStyle = '#fff'; ctx.font = 'bold 10px sans-serif'; ctx.textAlign = 'center';
ctx.fillText('●', toX(x), toY(y) + 3);
}
</script>
{% endif %}
{% endblock %}
templates/analysis/run_detail.html
+56
View File
@@ -0,0 +1,56 @@
{% extends 'base.html' %}
{% block title %}Run #{{ run.id }} - TLS Analyzer{% endblock %}
{% block content %}
<div class="card">
<h2>Run #{{ run.id }} — {{ run.created_at|date:"Y-m-d H:i" }}</h2>
<p>Status: <span class="badge {% if run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-info{% endif %}">{{ run.get_status_display }}</span></p>
<p>CSV: <code>{{ run.csv_glob }}</code></p>
<p>Entity Column: {{ run.entity_column|default:"(not detected)" }}</p>
</div>
<div class="grid-3">
<div class="card" style="text-align:center;">
<div class="stat">{{ run.total_flows|default:"0" }}</div>
<div class="stat-label">Total Flows</div>
</div>
<div class="card" style="text-align:center;">
<div class="stat">{{ run.entity_count|default:"0" }}</div>
<div class="stat-label">Entities</div>
</div>
<div class="card" style="text-align:center;">
<div class="stat">{{ run.cluster_count|default:"0" }}</div>
<div class="stat-label">Clusters</div>
</div>
</div>
<div class="card">
<h2>Clusters</h2>
<a href="{% url 'analysis:cluster_overview' run.id %}" class="btn btn-primary" style="margin-bottom:1rem;">View Cluster Overview</a>
{% if clusters %}
<table>
<thead>
<tr>
<th>Cluster</th>
<th>Size</th>
<th>Proportion</th>
<th>Silhouette</th>
<th></th>
</tr>
</thead>
<tbody>
{% for c in clusters %}
<tr>
<td><span class="badge badge-info">#{{ c.cluster_label }}</span></td>
<td>{{ c.size }}</td>
<td>{{ c.proportion|floatformat:2 }}</td>
<td>{{ c.silhouette_score|floatformat:4|default:"-" }}</td>
<td><a href="{% url 'analysis:cluster_detail' run.id c.cluster_label %}" class="btn btn-primary">Detail</a></td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<div class="empty-state"><p>No clusters yet.</p></div>
{% endif %}
</div>
{% endblock %}
templates/analysis/run_list.html
+42
View File
@@ -0,0 +1,42 @@
{% extends 'base.html' %}
{% block title %}Analysis Runs - TLS Analyzer{% endblock %}
{% block content %}
<div class="card">
<h2>All Analysis Runs</h2>
{% if runs %}
<table>
<thead>
<tr>
<th>ID</th>
<th>Started</th>
<th>Status</th>
<th>CSV Glob</th>
<th>Entities</th>
<th>Clusters</th>
<th></th>
</tr>
</thead>
<tbody>
{% for run in runs %}
<tr>
<td>#{{ run.id }}</td>
<td>{{ run.created_at|date:"Y-m-d H:i" }}</td>
<td><span class="badge {% if run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-info{% endif %}">{{ run.get_status_display }}</span></td>
<td style="max-width:300px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
<td>{{ run.entity_count|default:"-" }}</td>
<td>{{ run.cluster_count|default:"-" }}</td>
<td><a href="{% url 'analysis:run_detail' run.id %}" class="btn btn-primary">Detail</a></td>
</tr>
{% endfor %}
</tbody>
</table>
<div class="pagination">
{% if page > 1 %}<a href="?page={{ page|add:-1 }}">← Prev</a>{% endif %}
<span class="active">{{ page }}</span>
{% if page < total_pages %}<a href="?page={{ page|add:1 }}">Next →</a>{% endif %}
</div>
{% else %}
<div class="empty-state"><p>No runs yet.</p></div>
{% endif %}
</div>
{% endblock %}
templates/base.html
+55
View File
@@ -0,0 +1,55 @@
<!DOCTYPE html>
<html lang="zh-hans">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}天璇{% endblock %}</title>
<link rel="icon" type="image/svg+xml" href="/static/tianxuan/favicon.svg">
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f7fa; color: #1a1a2e; }
nav { background: #1a1a2e; padding: 1rem 2rem; display: flex; align-items: center; gap: 2rem; }
nav a { color: #e0e0e0; text-decoration: none; font-size: 0.95rem; }
nav a:hover { color: #fff; }
nav .brand { color: #fff; font-size: 1.2rem; font-weight: 700; }
.container { max-width: 1200px; margin: 0 auto; padding: 2rem; }
.card { background: #fff; border-radius: 8px; padding: 1.5rem; margin-bottom: 1.5rem; box-shadow: 0 1px 3px rgba(0,0,0,0.1); }
.card h2 { margin-bottom: 1rem; color: #1a1a2e; font-size: 1.1rem; }
table { width: 100%; border-collapse: collapse; }
th, td { text-align: left; padding: 0.75rem 0.5rem; border-bottom: 1px solid #eee; font-size: 0.9rem; }
th { font-weight: 600; color: #666; }
.badge { display: inline-block; padding: 0.2rem 0.6rem; border-radius: 12px; font-size: 0.8rem; font-weight: 600; }
.badge-success { background: #d4edda; color: #155724; }
.badge-warning { background: #fff3cd; color: #856404; }
.badge-info { background: #d1ecf1; color: #0c5460; }
.badge-danger { background: #f8d7da; color: #721c24; }
.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 1.5rem; }
.grid-3 { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 1.5rem; }
.stat { font-size: 2rem; font-weight: 700; color: #1a1a2e; }
.stat-label { font-size: 0.85rem; color: #666; margin-top: 0.25rem; }
.btn { display: inline-block; padding: 0.5rem 1rem; border-radius: 6px; text-decoration: none; font-size: 0.9rem; cursor: pointer; border: none; }
.btn-primary { background: #4361ee; color: #fff; }
.btn-primary:hover { background: #3a56d4; }
.scatter-canvas { max-height: 500px; }
.pagination { display: flex; justify-content: center; gap: 0.5rem; margin-top: 1.5rem; }
.pagination a, .pagination span { padding: 0.4rem 0.8rem; border-radius: 4px; text-decoration: none; color: #4361ee; background: #fff; border: 1px solid #ddd; }
.pagination .active { background: #4361ee; color: #fff; border-color: #4361ee; }
.empty-state { text-align: center; padding: 3rem; color: #999; }
.empty-state p { font-size: 1rem; margin-top: 0.5rem; }
</style>
</head>
<body>
<nav>
<span class="brand">天璇</span>
<a href="{% url 'analysis:dashboard' %}">首页</a>
<a href="{% url 'analysis:upload' %}">上传数据</a>
<a href="{% url 'analysis:manual' %}">手动分析</a>
<a href="{% url 'analysis:auto' %}">LLM分析</a>
<a href="{% url 'analysis:run_list' %}">运行记录</a>
<a href="{% url 'analysis:globe' %}">地球</a>
</nav>
<div class="container">
{% block content %}{% endblock %}
</div>
</body>
</html>
templates/tianxuan/auto.html
+108
View File
@@ -0,0 +1,108 @@
{% extends 'base.html' %}
{% block title %}LLM 自动分析 - 天璇{% endblock %}
{% block content %}
<div class="card">
<h2>LLM 自动分析</h2>
<p style="color:#666;">LLM 将自动编排工具链完成：加载 → 检测实体 → 聚合 → 聚类 → 特征提取</p>
{% if not llm_configured %}
<div style="background:#fff3cd;padding:1rem;border-radius:6px;margin:1rem 0;">
⚠️ LLM 尚未配置。请先在 <a href="/config/">配置页面</a> 设置 base_url 和 API Key。
</div>
{% endif %}
</div>
<div class="card">
<h3>选择数据集</h3>
{% if runs %}
<table>
<thead><tr><th></th><th>ID</th><th>文件</th><th>实体列</th><th>状态</th></tr></thead>
<tbody>
{% for run in runs %}
<tr>
<td><input type="radio" name="auto_run_id" value="{{ run.id }}"></td>
<td>#{{ run.id }}</td>
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
<td>{{ run.entity_column|default:"-" }}</td>
<td><span class="badge badge-success">{{ run.status }}</span></td>
</tr>
{% empty %}
<tr><td colspan="5" style="text-align:center;">暂无数据。先去 <a href="/upload/">上传 CSV</a>。</td></tr>
{% endfor %}
</tbody>
</table>
<button class="btn btn-primary" style="margin-top:1rem;" onclick="startAuto()" {% if not llm_configured %}disabled{% endif %}>开始自动分析</button>
{% endif %}
<div id="autoProgress" style="display:none;margin-top:1rem;">
<p>LLM 编排中... <span id="autoStatus"></span></p>
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
<div id="autoBar" style="background:#4361ee;height:100%;width:0%;"></div>
</div>
<button id="cancelBtn" class="btn" style="background:#f72585;color:white;display:none;" onclick="cancelAnalysis()">取消分析</button>
<div id="liveLog" style="background:#1a1a2e;color:#e0e0e0;padding:1rem;border-radius:6px;margin-top:1rem;max-height:300px;overflow:auto;font-size:0.8rem;display:none;"></div>
</div>
</div>
<script>
let logPoll = null;
let statusPoll = null;
async function startAuto() {
const selected = document.querySelector('input[name="auto_run_id"]:checked');
if (!selected) { alert('请先选择一个数据集'); return; }
document.getElementById('autoProgress').style.display = 'block';
document.getElementById('cancelBtn').style.display = 'inline-block';
document.getElementById('liveLog').style.display = 'block';
document.getElementById('liveLog').innerHTML = '';
const resp = await fetch('/analyze/llm/', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-CSRFToken': '{{ csrf_token }}' },
body: JSON.stringify({ run_id: parseInt(selected.value), entity_column: '' }),
});
const data = await resp.json();
const runId = selected.value;
statusPoll = setInterval(async () => {
const r = await fetch(`/runs/${runId}/status/`);
const s = await r.json();
const pct = s.progress_pct || 0;
const msg = s.progress_msg || s.status;
document.getElementById('autoStatus').textContent = `${msg} (${pct}%)`;
document.getElementById('autoBar').style.width = pct + '%';
if (s.run_log) {
document.getElementById('liveLog').innerHTML = s.run_log.split('\n').map(l => l.trim()).filter(l => l).join('\n') + '\n';
document.getElementById('liveLog').scrollTop = document.getElementById('liveLog').scrollHeight;
}
if (s.status === 'completed') { clearInterval(statusPoll); clearInterval(logPoll); document.getElementById('autoStatus').textContent = '分析完成！'; }
if (s.status === 'failed') { clearInterval(statusPoll); clearInterval(logPoll); document.getElementById('autoStatus').textContent = '失败: ' + (s.error_message || ''); }
}, 1000);
// Poll live logs every 3 seconds
let lastPos = 0;
logPoll = setInterval(async () => {
try {
const r = await fetch(`/logs/?pos=${lastPos}`);
const txt = await r.text();
const logDiv = document.getElementById('liveLog');
// Check if response is a number (position marker) followed by new content
const parts = txt.split('\n', 2);
if (parts.length >= 2) {
const newPos = parseInt(parts[0], 10);
if (!isNaN(newPos)) { lastPos = newPos; }
if (parts[1]) {
logDiv.innerHTML += parts[1] + '\n';
logDiv.scrollTop = logDiv.scrollHeight;
}
}
} catch (e) { /* ignore fetch errors during polling */ }
}, 3000);
}
function cancelAnalysis() {
if (statusPoll) { clearInterval(statusPoll); statusPoll = null; }
if (logPoll) { clearInterval(logPoll); logPoll = null; }
document.getElementById('autoStatus').textContent = '已取消';
document.getElementById('cancelBtn').style.display = 'none';
document.getElementById('liveLog').innerHTML += '\n--- 分析已取消 ---\n';
}
</script>
{% endblock %}
templates/tianxuan/config.html
+174
View File
@@ -0,0 +1,174 @@
{% extends 'base.html' %}
{% block title %}配置 - TLS Analyzer{% endblock %}
{% block content %}
<div class="card">
<h2>系统配置</h2>
<p style="color:#666;margin-bottom:1rem;">修改配置后点击「保存配置」持久化到 <code>config/config.yaml</code>。</p>
</div>
<form id="config-form" method="post">
{% csrf_token %}
{# ── Server ── #}
<div class="card">
<h2>服务器</h2>
<div class="grid-3">
<div class="form-group">
<label for="server_host">Host</label>
<input type="text" id="server_host" name="server_host" value="{{ config.server.host }}" class="form-input">
</div>
<div class="form-group">
<label for="server_port">Port</label>
<input type="number" id="server_port" name="server_port" value="{{ config.server.port }}" class="form-input">
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" name="server_debug" value="true" {% if config.server.debug %}checked{% endif %}>
Debug 模式
</label>
</div>
</div>
</div>
{# ── Data ── #}
<div class="card">
<h2>数据</h2>
<div class="grid-2">
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" name="data_schema_strict" value="true" {% if config.data.schema_strict %}checked{% endif %}>
严格 Schema 校验
</label>
</div>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" name="data_recursive" value="true" {% if config.data.recursive %}checked{% endif %}>
递归扫描
</label>
</div>
</div>
</div>
{# ── Clustering ── #}
<div class="card">
<h2>聚类</h2>
<div class="grid-2">
<div class="form-group">
<label for="clustering_algorithm">算法</label>
<select id="clustering_algorithm" name="clustering_algorithm" class="form-input">
<option value="hdbscan" {% if config.clustering.algorithm == 'hdbscan' %}selected{% endif %}>HDBSCAN</option>
<option value="kmeans" {% if config.clustering.algorithm == 'kmeans' %}selected{% endif %}>K-Means</option>
<option value="dbscan" {% if config.clustering.algorithm == 'dbscan' %}selected{% endif %}>DBSCAN</option>
</select>
</div>
<div class="form-group">
<label for="clustering_min_cluster_size">最小簇大小</label>
<input type="number" id="clustering_min_cluster_size" name="clustering_min_cluster_size"
value="{{ config.clustering.min_cluster_size }}" min="2" class="form-input">
</div>
</div>
</div>
{# ── LLM ── #}
<div class="card">
<h2>LLM 连接</h2>
<div class="form-group">
<label class="checkbox-label">
<input type="checkbox" name="llm_enabled" value="true" {% if config.llm.enabled %}checked{% endif %}>
启用 LLM
</label>
</div>
<div class="form-group">
<label for="llm_base_url">Base URL</label>
<input type="text" id="llm_base_url" name="llm_base_url" value="{{ config.llm.base_url }}"
placeholder="https://api.openai.com/v1" class="form-input">
</div>
<div class="form-group">
<label for="llm_api_key">API Key</label>
<input type="password" id="llm_api_key" name="llm_api_key" value="{{ config.llm.api_key }}"
placeholder="sk-..." class="form-input">
</div>
<div class="form-group">
<label for="llm_model">Model</label>
<input type="text" id="llm_model" name="llm_model" value="{{ config.llm.model }}"
placeholder="gpt-4" class="form-input">
</div>
{# LLM connectivity test button + result area #}
<div style="margin-top:1rem; display:flex; align-items:center; gap:1rem; flex-wrap:wrap;">
<button type="button" id="btn-llm-test" class="btn btn-primary" onclick="testLLM()">测试连通性</button>
<span id="llm-test-result" style="font-size:0.9rem;"></span>
</div>
</div>
{# ── Save button ── #}
<div class="card" style="text-align:right;">
<button type="submit" class="btn btn-primary" style="padding:0.6rem 2rem;">保存配置</button>
</div>
</form>
{% if saved %}
<script>
// Flash-style feedback: briefly highlight the message
const el = document.createElement('div');
el.style.cssText = 'position:fixed;top:1rem;right:1rem;background:#155724;color:#fff;padding:0.8rem 1.5rem;border-radius:6px;font-size:0.9rem;z-index:999;box-shadow:0 2px 8px rgba(0,0,0,0.2);';
el.textContent = '✅ 配置已保存';
document.body.appendChild(el);
setTimeout(() => el.remove(), 3000);
</script>
{% endif %}
<script>
async function testLLM() {
const btn = document.getElementById('btn-llm-test');
const result = document.getElementById('llm-test-result');
btn.disabled = true;
btn.textContent = '测试中...';
result.textContent = '';
result.style.color = '';
const payload = {
base_url: document.getElementById('llm_base_url').value || '',
api_key: document.getElementById('llm_api_key').value || '',
model: document.getElementById('llm_model').value || '',
};
try {
const resp = await fetch('/api/llm/test/', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-CSRFToken': getCSRF() },
body: JSON.stringify(payload),
});
const data = await resp.json();
if (data.success) {
result.style.color = '#155724';
result.textContent = `✅ ${data.message} (${data.latency_ms}ms)`;
} else {
result.style.color = '#721c24';
result.textContent = `❌ ${data.message}`;
}
} catch (err) {
result.style.color = '#721c24';
result.textContent = `❌ 请求失败: ${err.message}`;
} finally {
btn.disabled = false;
btn.textContent = '测试连通性';
}
}
function getCSRF() {
return document.querySelector('[name=csrfmiddlewaretoken]')?.value || '';
}
</script>
<style>
.form-group { margin-bottom: 1rem; }
.form-group label { display: block; font-size: 0.85rem; color: #555; margin-bottom: 0.3rem; font-weight: 600; }
.form-input { width: 100%; padding: 0.5rem 0.75rem; border: 1px solid #ddd; border-radius: 6px; font-size: 0.9rem; }
.form-input:focus { outline: none; border-color: #4361ee; box-shadow: 0 0 0 2px rgba(67,97,238,0.15); }
.checkbox-label { display: flex; align-items: center; gap: 0.5rem; font-size: 0.9rem; color: #333; cursor: pointer; }
.checkbox-label input[type="checkbox"] { width: 1rem; height: 1rem; cursor: pointer; }
.grid-2 .form-group, .grid-3 .form-group { min-width: 0; }
.card h2 { font-size: 1rem; border-bottom: 1px solid #eee; padding-bottom: 0.5rem; margin-bottom: 1rem; }
</style>
{% endblock %}
templates/tianxuan/globe.html
+235
View File
@@ -0,0 +1,235 @@
{% extends 'base.html' %}
{% load static %}
{% block title %}TLS 流量地球 - 天璇{% endblock %}
{% block content %}
<div class="card">
<h2>TLS 流量地球可视化</h2>
<p style="color:#666;">三维地球上的流量弧线，显示源和目标之间的 TLS 连接</p>
<p style="font-size:0.8rem;color:#999;">
<span id="arcCount">0</span> 条流量 |
颜色: <span style="color:#4361ee;">TLSv1.3</span>
<span style="color:#f72585;">TLSv1.2</span>
<span style="color:#7209b7;">TLSv1.1/1.0</span>
<span style="color:#4cc9f0;">其他</span>
<span style="color:#888;margin-left:0.5rem;">| 支持 hex 格式 (03 03 → TLSv1.2)</span>
</p>
</div>
{% if all_runs %}
<div class="card" style="margin-bottom:1rem;">
<h3>选择数据运行</h3>
<form method="get" action="{% url 'analysis:globe' %}" id="runSelectForm">
<div style="display:flex;flex-wrap:wrap;gap:0.5rem;align-items:center;">
{% for r in all_runs %}
<label style="font-size:0.85rem;cursor:pointer;">
<input type="checkbox" name="runs" value="{{ r.id }}"
{% if r.id in selected_ids %}checked{% endif %}>
Run #{{ r.id }}
</label>
{% endfor %}
<button type="submit" class="btn btn-primary" style="margin-left:auto;">更新地球</button>
</div>
</form>
</div>
{% endif %}
<div class="card" style="margin-bottom:1rem;padding:0.8rem;">
<div style="display:flex;gap:1.5rem;align-items:center;flex-wrap:wrap;">
<label style="font-size:0.85rem;cursor:pointer;">
<input type="checkbox" id="toggleGrid" checked onchange="gridGroup.visible = this.checked;">
经纬网格
</label>
<label style="font-size:0.85rem;cursor:pointer;">
<input type="checkbox" id="toggleBorders" checked onchange="borderGroup.visible = this.checked;">
国境线
</label>
<span style="font-size:0.85rem;color:#666;">鼠标拖拽旋转 · 滚轮缩放</span>
</div>
</div>
<div class="card" style="padding:0;overflow:hidden;background:radial-gradient(ellipse at center, #0a0a2e 0%, #000 100%);">
<div id="globeContainer" style="width:100%;height:70vh;min-height:400px;cursor:grab;"></div>
</div>
<script src="{% static 'tianxuan/three.min.js' %}"></script>
<script src="{% static 'tianxuan/world_borders.js' %}"></script>
<script>
var flowData = {{ geo_flows|safe }};
var container = document.getElementById('globeContainer');
var H = container.clientHeight || window.innerHeight * 0.8;
var W = container.clientWidth;
var scene = new THREE.Scene();
var camera = new THREE.PerspectiveCamera(45, W/H, 0.1, 1000);
camera.position.set(0, 0.5, 12);
var renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.setSize(W, H);
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
renderer.setClearColor(0x000000, 0);
container.appendChild(renderer.domElement);
var texLoader = new THREE.TextureLoader();
var earthMap = texLoader.load("{% static 'tianxuan/earth_atmos_2048.jpg' %}");
var earthGeo = new THREE.SphereGeometry(5, 96, 96);
var earthMat = new THREE.MeshPhongMaterial({ map: earthMap, specular: new THREE.Color(0x333333), shininess: 5 });
var earth = new THREE.Mesh(earthGeo, earthMat);
scene.add(earth);
var glowGeo = new THREE.SphereGeometry(5.08, 64, 64);
var glowMat = new THREE.MeshPhongMaterial({ color: 0x4361ee, transparent: true, opacity: 0.12, side: THREE.BackSide });
scene.add(new THREE.Mesh(glowGeo, glowMat));
var gridMat = new THREE.LineBasicMaterial({ color: 0xffffff, transparent: true, opacity: 0.12 });
var gridGroup = new THREE.Group();
for (var lat = -80; lat <= 80; lat += 10) {
var phi = (90 - lat) * Math.PI / 180; var pts = [];
for (var i = 0; i <= 64; i++) {
var theta = i / 64 * Math.PI * 2;
pts.push(new THREE.Vector3(5.02 * Math.sin(phi) * Math.cos(theta), 5.02 * Math.cos(phi), 5.02 * Math.sin(phi) * Math.sin(theta)));
}
gridGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), gridMat));
}
for (var lon = 0; lon < 360; lon += 10) {
var pts = [];
for (var i = 0; i <= 32; i++) {
var phi = i / 32 * Math.PI; var theta = lon * Math.PI / 180;
pts.push(new THREE.Vector3(5.02 * Math.sin(phi) * Math.cos(theta), 5.02 * Math.cos(phi), 5.02 * Math.sin(phi) * Math.sin(theta)));
}
gridGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), gridMat));
}
scene.add(gridGroup);
// ── World borders (loaded from static file) ──
var borderGroup = new THREE.Group();
var borderMat = new THREE.LineBasicMaterial({ color: 0x88dd88, transparent: true, opacity: 0.4, depthTest: true, depthWrite: false });
var borderData = []; // { line, center }
function renderBorders(borders) {
if (!borders) return;
for (var c = 0; c < borders.length; c++) {
var pts = [];
for (var p = 0; p < borders[c].length; p++) {
var lat = borders[c][p][0], lon = borders[c][p][1];
var phi = (90 - lat) * Math.PI / 180;
var theta = (lon + 180) * Math.PI / 180;
pts.push(new THREE.Vector3(5.01 * Math.sin(phi) * (-Math.cos(theta)), 5.01 * Math.cos(phi), 5.01 * Math.sin(phi) * Math.sin(theta)));
}
if (pts.length > 2) {
var line = new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), borderMat);
borderGroup.add(line);
// Compute center of polygon (average of all points)
var cx = 0, cy = 0, cz = 0;
for (var p = 0; p < pts.length; p++) { cx += pts[p].x; cy += pts[p].y; cz += pts[p].z; }
borderData.push({ line: line, center: new THREE.Vector3(cx / pts.length, cy / pts.length, cz / pts.length) });
}
}
}
if (typeof worldBorders !== 'undefined') renderBorders(worldBorders);
scene.add(borderGroup);
var starsGeo = new THREE.BufferGeometry();
var starPos = new Float32Array(5000 * 3);
for (var i = 0; i < 5000 * 3; i++) starPos[i] = (Math.random() - 0.5) * 400;
starsGeo.setAttribute('position', new THREE.BufferAttribute(starPos, 3));
scene.add(new THREE.Points(starsGeo, new THREE.PointsMaterial({ color: 0xffffff, size: 0.2, transparent: true, opacity: 0.7 })));
scene.add(new THREE.AmbientLight(0x222244));
var dl = new THREE.DirectionalLight(0xffffff, 1.2); dl.position.set(10, 15, 10); scene.add(dl);
scene.add(new THREE.DirectionalLight(0x8888ff, 0.3));
function ll2pos(lat, lon, r) {
var phi = (90 - lat) * Math.PI / 180;
var theta = (lon + 180) * Math.PI / 180;
r = r || 5.5; // well above earth surface (5.0) for depth testing
return new THREE.Vector3(-r * Math.sin(phi) * Math.cos(theta), r * Math.cos(phi), r * Math.sin(phi) * Math.sin(theta));
}
// ── Pulse arcs: faint tube + moving trail spheres ──
// Uses depthTest:true — Three.js naturally hides back-face arcs behind earth
var arcGroup = new THREE.Group();
var arcColors = { 'TLSv1.3': 0x4361ee, 'TLSv1.2': 0xf72585, 'TLSv1.1': 0x7209b7, 'TLSv1.0': 0x7209b7 };
var TRAIL = 16;
var SPACING = 0.018;
var arcData = [];
flowData.slice(0, 120).forEach(function(f) {
var src = ll2pos(f.slat, f.slon, 5.03);
var dst = ll2pos(f.dlat, f.dlon, 5.03);
if (src.distanceTo(dst) < 1.0) return;
var mid = new THREE.Vector3().addVectors(src, dst).multiplyScalar(0.5);
var dist = src.distanceTo(dst);
mid.normalize().multiplyScalar(6.5 + dist * 0.2);
var curve = new THREE.QuadraticBezierCurve3(src, mid, dst);
var c = arcColors[f.tls] || 0x4cc9f0;
var col = new THREE.Color(c);
var tube = new THREE.Mesh(
new THREE.TubeGeometry(curve, 24, 0.03, 6, false),
new THREE.MeshBasicMaterial({ color: c, transparent: true, opacity: 0.15, depthTest: true, depthWrite: false })
);
arcGroup.add(tube);
var spheres = [];
for (var i = 0; i < TRAIL; i++) {
spheres.push(new THREE.Mesh(
new THREE.SphereGeometry(0.04, 6, 6),
new THREE.MeshBasicMaterial({ color: col, transparent: true, opacity: 1 - i / TRAIL, depthTest: true, depthWrite: false })
));
}
for (var i = 0; i < spheres.length; i++) arcGroup.add(spheres[i]);
arcData.push({ curve: curve, spheres: spheres, tube: tube, offset: Math.random() * 1000 });
});
scene.add(arcGroup);
document.getElementById('arcCount').textContent = arcData.length;
function updatePulses(time) {
var speed = 0.6;
for (var a = 0; a < arcData.length; a++) {
var d = arcData[a];
var phase = ((time * speed + d.offset / 1000) % 1 + 1) % 1;
for (var s = 0; s < d.spheres.length; s++) {
var t = phase - s * SPACING; if (t < 0) t += 1;
var pt = d.curve.getPoint(t);
d.spheres[s].position.copy(pt);
d.spheres[s].material.opacity = 1 - s / TRAIL;
}
}
}
var inertiaY = 0.0005, inertiaX = 0;
function animate() {
requestAnimationFrame(animate);
inertiaY *= 0.97; inertiaX *= 0.97;
if (Math.abs(inertiaY) < 0.0005) inertiaY = Math.sign(inertiaY || 1) * 0.0005;
if (Math.abs(inertiaX) < 0.0003) inertiaX = 0;
earth.rotation.y += inertiaY; arcGroup.rotation.y += inertiaY; gridGroup.rotation.y += inertiaY; borderGroup.rotation.y += inertiaY;
earth.rotation.x += inertiaX; arcGroup.rotation.x += inertiaX; gridGroup.rotation.x += inertiaX; borderGroup.rotation.x += inertiaX;
updatePulses(Date.now() / 1000);
renderer.render(scene, camera);
}
animate();
window.addEventListener('resize', function() {
H = container.clientHeight || window.innerHeight * 0.8;
var w = container.clientWidth; renderer.setSize(w, H); camera.aspect = w / H; camera.updateProjectionMatrix();
});
var dragging = false, px = 0, py = 0;
renderer.domElement.addEventListener('mousedown', function(e) {
if (e.button === 1) { e.preventDefault(); camera.position.set(0, 0.5, 12); return; }
dragging = true; px = e.clientX; py = e.clientY; inertiaY = 0; inertiaX = 0;
});
renderer.domElement.addEventListener('wheel', function(e) { e.preventDefault(); camera.position.z += e.deltaY * 0.01; if (camera.position.z < 6) camera.position.z = 6; if (camera.position.z > 30) camera.position.z = 30; }, { passive: false });
window.addEventListener('mousemove', function(e) {
if (!dragging) return;
var dx = e.clientX - px, dy = e.clientY - py;
inertiaY = dx * 0.008; inertiaX = dy * 0.005;
px = e.clientX; py = e.clientY;
});
window.addEventListener('mouseup', function() { dragging = false; });
</script>
{% endblock %}
templates/tianxuan/log_viewer.html
+5
View File
@@ -0,0 +1,5 @@
{% extends 'base.html' %}{% block content %}
<div class="card"><h2>系统日志 (最近200行)</h2>
<pre style="background:#1a1a2e;color:#e0e0e0;padding:1rem;overflow:auto;max-height:80vh;font-size:0.8rem;">{% for line in lines %}{{ line }}{% endfor %}</pre>
<meta http-equiv="refresh" content="10">
{% endblock %}
templates/tianxuan/manual.html
+145
View File
@@ -0,0 +1,145 @@
{% extends 'base.html' %}
{% load query_helpers %}
{% block title %}手动分析 - 天璇{% endblock %}
{% block content %}
<div class="card">
<h2>手动分析 - 第 {{ step }} 步 / 共 3 步</h2>
</div>
{% if step == 1 %}
<div class="card">
<h3>Step 1: 选择数据集</h3>
<p>选择已上传且预处理完成的数据集：</p>
{% if runs %}
<form method="get" action="/analyze/manual/" style="margin-top:1rem;">
<input type="hidden" name="step" value="2">
<table>
<thead><tr><th></th><th>ID</th><th>文件</th><th>行数</th><th>实体列</th><th>状态</th></tr></thead>
<tbody>
{% for run in runs %}
<tr>
<td><input type="radio" name="run_id" value="{{ run.id }}" required></td>
<td>#{{ run.id }}</td>
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
<td>{{ run.total_flows|default:"-" }}</td>
<td>{{ run.entity_column|default:"-" }}</td>
<td><span class="badge badge-success">{{ run.status }}</span></td>
</tr>
{% empty %}
<tr><td colspan="6" style="text-align:center;">暂无已预处理的数据集。先去 <a href="/upload/">上传页面</a> 上传 CSV。</td></tr>
{% endfor %}
</tbody>
</table>
<button type="submit" class="btn btn-primary" style="margin-top:1rem;">下一步 →</button>
</form>
{% endif %}
</div>
{% endif %}
{% if step == 2 %}
<div class="card">
<h3>Step 2: 配置聚类参数</h3>
<form method="get" action="/analyze/manual/">
<input type="hidden" name="step" value="3">
<input type="hidden" name="run_id" value="{{ request.GET.run_id }}">
<div style="margin-bottom:1rem;">
<label>实体列（entity_columns）：</label>
<select name="entity_columns" multiple style="width:100%;min-height:100px;">
{% if entity_candidates %}
{% for c in entity_candidates %}
<option value="{{ c.column }}" {% if c.column in selected_entity_cols %}selected{% endif %}>{{ c.column }} (score: {{ c.score }})</option>
{% endfor %}
{% elif schema_keys %}
{% for col in schema_keys %}
<option value="{{ col }}" {% if col in selected_entity_cols %}selected{% endif %}>{{ col }}</option>
{% endfor %}
{% else %}
<option value="">暂无可选列</option>
{% endif %}
</select>
<small style="color:#666;">按住 Ctrl 多选，留空则使用自动检测结果</small>
</div>
<div style="margin-bottom:1rem;">
<label>特征列（feature_columns）：</label>
<select name="feature_columns" multiple style="width:100%;min-height:100px;">
{% if feature_options %}
{% for opt in feature_options %}
<option value="{{ opt.name }}">{{ opt.name }} ({{ opt.dtype }})</option>
{% endfor %}
{% else %}
<option value="">暂无可选数值列</option>
{% endif %}
</select>
<small style="color:#666;">按住 Ctrl 多选，留空则自动选择前10个数值列。仅显示数值类型列。</small>
</div>
<div style="margin-bottom:1rem;">
<label>聚类算法：</label>
<select name="algorithm">
<option value="hdbscan" {% if default_algo == 'hdbscan' %}selected{% endif %}>HDBSCAN（自动确定簇数）</option>
<option value="kmeans" {% if default_algo == 'kmeans' %}selected{% endif %}>KMeans（需预设簇数）</option>
</select>
</div>
<div style="margin-bottom:1rem;">
<label>最小簇大小（min_cluster_size）：</label>
<input type="number" name="min_cluster_size" value="{{ default_min_size }}" min="2" max="100">
</div>
<a href="/analyze/manual/?step=1" class="btn" style="background:#ddd;margin-right:0.5rem;">← 上一步</a>
<button type="submit" class="btn btn-primary">下一步 →</button>
</form>
</div>
{% endif %}
{% if step == 3 %}
<div class="card">
<h3>Step 3: 确认并运行</h3>
<p>请确认以下配置，确认无误后点击"开始分析"：</p>
<table>
<tr><td>数据集 ID</td><td>#{{ request.GET.run_id }}</td></tr>
<tr><td>实体列</td><td>{{ request.GET|getlist:"entity_columns"|default:"自动检测" }}</td></tr>
<tr><td>特征列</td><td>{{ request.GET.feature_columns|default:"自动选择" }}</td></tr>
<tr><td>聚类算法</td><td>{{ request.GET.algorithm }}</td></tr>
<tr><td>最小簇大小</td><td>{{ request.GET.min_cluster_size }}</td></tr>
</table>
<div style="margin-top:1rem;">
<a href="/analyze/manual/?step=2&run_id={{ request.GET.run_id }}" class="btn" style="background:#ddd;margin-right:0.5rem;">← 上一步</a>
<button class="btn btn-primary" onclick="runAnalysis()">开始分析</button>
</div>
<div id="runProgress" style="display:none;margin-top:1rem;">
<p>分析进行中... <span id="runStatus"></span></p>
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
<div id="runBar" style="background:#4361ee;height:100%;width:0%;"></div>
</div>
</div>
</div>
<script>
async function runAnalysis() {
document.getElementById('runProgress').style.display = 'block';
const params = new URLSearchParams(window.location.search);
const entityCols = params.getAll('entity_columns');
const featureCols = params.getAll('feature_columns');
const resp = await fetch('/analyze/run/', {
method: 'POST',
headers: { 'Content-Type': 'application/json',
'X-CSRFToken': '{{ csrf_token }}' },
body: JSON.stringify({
run_id: parseInt('{{ request.GET.run_id }}'),
entity_column: '',
entity_columns: entityCols.length ? entityCols : undefined,
feature_columns: featureCols.length ? featureCols : undefined,
algorithm: '{{ request.GET.algorithm }}',
min_cluster_size: parseInt('{{ request.GET.min_cluster_size }}'),
}),
});
const data = await resp.json();
const runId = '{{ request.GET.run_id }}';
const poll = setInterval(async () => {
const r = await fetch(`/runs/${runId}/status/`);
const s = await r.json();
document.getElementById('runStatus').textContent = s.status;
if (s.status === 'completed') { clearInterval(poll); window.location.href = `/runs/${runId}/`; }
if (s.status === 'failed') { clearInterval(poll); document.getElementById('runStatus').textContent = '失败: ' + (s.error_message || ''); }
}, 1000);
}
</script>
{% endif %}
{% endblock %}
templates/tianxuan/upload.html
+125
View File
@@ -0,0 +1,125 @@
{% extends 'base.html' %}
{% block title %}上传数据 - 天璇{% endblock %}
{% block content %}
<style>
#uploadCard { position: sticky; top: 1rem; z-index: 10; }
</style>
<div class="card" id="uploadCard">
<h2>上传 CSV 文件</h2>
<p style="color:#666;margin-bottom:1rem;">支持 .csv 文件，多文件批量上传，自动解压 .zip</p>
<form id="uploadForm" enctype="multipart/form-data">
{% csrf_token %}
<div style="border:2px dashed #ccc;border-radius:8px;padding:3rem;text-align:center;cursor:pointer;background:#fafafa;" id="dropZone">
<div style="font-size:2rem;margin-bottom:0.5rem;">📂</div>
<p>拖拽 CSV 文件到此处，或点击选择</p>
<input type="file" name="files" multiple accept=".csv,.zip" style="display:none;" id="fileInput">
</div>
<div id="submitBar" style="margin-top:1rem;display:none;">
<button type="submit" class="btn btn-primary" id="submitBtn">开始上传</button>
</div>
<div id="fileList" style="margin-top:0.5rem;"></div>
</form>
<div id="progressArea" style="display:none;margin-top:1rem;">
<p>上传中... <span id="progressText"></span></p>
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
<div id="progressBar" style="background:#4361ee;height:100%;width:0%;"></div>
</div>
</div>
</div>
<div class="card" style="margin-top:1.5rem;">
<h3>已上传的数据集</h3>
{% if runs %}
<table>
<thead><tr><th>ID</th><th>文件</th><th>行数</th><th>状态</th><th>操作</th></tr></thead>
<tbody>
{% for run in runs %}
<tr>
<td>#{{ run.id }}</td>
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
<td>{{ run.total_flows|default:"-" }}</td>
<td><span class="badge {% if run.status == 'ready' or run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-warning{% endif %}">{{ run.status }}</span></td>
<td>
{% if run.status == 'loading' or run.status == 'profiling' or run.status == 'aggregating' or run.status == 'clustering' or run.status == 'extracting' %}
<button class="btn btn-danger" style="background:#dc3545;color:#fff;padding:0.25rem 0.5rem;font-size:0.8rem;" onclick="deleteRun({{ run.id }}, true)">删除</button>
{% else %}
<button class="btn btn-danger" style="background:#dc3545;color:#fff;padding:0.25rem 0.5rem;font-size:0.8rem;" onclick="deleteRun({{ run.id }})">删除</button>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p style="color:#999;">暂无已上传的数据集</p>
{% endif %}
</div>
<script>
const dropZone = document.getElementById('dropZone');
const fileInput = document.getElementById('fileInput');
const fileList = document.getElementById('fileList');
const submitBar = document.getElementById('submitBar');
const submitBtn = document.getElementById('submitBtn');
let selectedFiles = [];
dropZone.addEventListener('click', () => fileInput.click());
dropZone.addEventListener('dragover', e => { e.preventDefault(); dropZone.style.borderColor = '#4361ee'; });
dropZone.addEventListener('dragleave', () => dropZone.style.borderColor = '#ccc');
dropZone.addEventListener('drop', e => { e.preventDefault(); handleFiles(e.dataTransfer.files); });
fileInput.addEventListener('change', () => handleFiles(fileInput.files));
function handleFiles(files) {
selectedFiles = [...files];
fileList.innerHTML = selectedFiles.map((f, i) =>
`<div style="padding:0.5rem;background:#f5f7fa;margin:0.25rem 0;border-radius:4px;">${i+1}. ${f.name} (${(f.size/1024).toFixed(0)} KB)</div>`
).join('');
submitBar.style.display = 'block';
}
document.getElementById('uploadForm').addEventListener('submit', async (e) => {
e.preventDefault();
const form = new FormData();
selectedFiles.forEach(f => form.append('files', f));
document.getElementById('progressArea').style.display = 'block';
submitBtn.disabled = true;
try {
const resp = await fetch('/upload/csv/', { method: 'POST', body: form,
headers: { 'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]').value } });
const data = await resp.json();
document.getElementById('progressText').textContent = '上传完成，后台处理中...';
// Poll for completion
const runId = data.run_id;
const poll = setInterval(async () => {
const r = await fetch(`/runs/${runId}/status/`);
const s = await r.json();
const pct = s.progress_pct || 0;
const msg = s.progress_msg || s.status;
document.getElementById('progressBar').style.width = pct + '%';
document.getElementById('progressText').textContent = `${msg} (${pct}%)`;
if (s.status === 'ready' || s.status === 'completed') { clearInterval(poll); window.location.href = `/runs/${runId}/`; }
if (s.status === 'failed') { clearInterval(poll); document.getElementById('progressText').textContent = '处理失败: ' + (s.error_message || '未知错误'); }
}, 1000);
} catch (err) { document.getElementById('progressText').textContent = '上传失败: ' + err.message; submitBtn.disabled = false; }
});
async function deleteRun(runId, processing) {
const msg = processing ? '后台处理中，删除将中断分析，确认删除？' : '确定要删除此数据集吗？此操作不可撤销。';
if (!confirm(msg)) return;
try {
const resp = await fetch(`/runs/${runId}/delete/`, {
method: 'POST',
headers: { 'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]').value },
});
const data = await resp.json();
if (data.deleted) {
location.reload();
} else {
alert('删除失败');
}
} catch (err) {
alert('删除失败: ' + err.message);
}
}
</script>
{% endblock %}
tests/__init__.py
View File
tests/test_data_loader.py
+367
View File
@@ -0,0 +1,367 @@
"""Tests for analysis.data_loader — BOM detection, schema validation, load/config."""
import csv
import os
import tempfile
from pathlib import Path
import pytest
import polars as pl
from analysis.data_loader import (
_file_count,
_merge_schema_entries,
_resolve_encoding,
detect_bom,
load_config,
load_csv_directory,
validate_schemas,
)
# ═══════════════════════════════════════════════════════════════════════
# BOM detection
# ═══════════════════════════════════════════════════════════════════════
class TestDetectBOM:
"""detect_bom() should return the correct encoding string for each BOM."""
def test_utf8_bom(self):
"""UTF-8 BOM (EF BB BF) → 'utf-8-sig'."""
path = _write_bytes(b'\xef\xbb\xbfcol1,col2\n1,2\n')
try:
assert detect_bom(path) == 'utf-8-sig'
finally:
os.unlink(path)
def test_utf16le_bom(self):
"""UTF-16 LE BOM (FF FE) → 'utf-16le'."""
path = _write_bytes(b'\xff\xfec\x00o\x00l\x001\x00,\x00c\x00o\x00l\x002\x00\n\x001\x00,\x002\x00\n')
try:
assert detect_bom(path) == 'utf-16le'
finally:
os.unlink(path)
def test_utf16be_bom(self):
"""UTF-16 BE BOM (FE FF) → 'utf-16be'."""
path = _write_bytes(b'\xfe\xff\x00c\x00o\x00l\x001\x00,\x00c\x00o\x00l\x002\x00\n\x001\x00,\x002\x00\n')
try:
assert detect_bom(path) == 'utf-16be'
finally:
os.unlink(path)
def test_no_bom(self):
"""Plain ASCII/UTF-8 without BOM → 'utf-8'."""
path = _write_bytes(b'col1,col2\n1,2\n')
try:
assert detect_bom(path) == 'utf-8'
finally:
os.unlink(path)
def test_empty_file(self):
"""Empty file → 'utf-8'."""
path = _write_bytes(b'')
try:
assert detect_bom(path) == 'utf-8'
finally:
os.unlink(path)
# ═══════════════════════════════════════════════════════════════════════
# Schema validation
# ═══════════════════════════════════════════════════════════════════════
class TestValidateSchemas:
"""validate_schemas() must raise ValueError when column sets differ."""
def test_matching_schemas(self):
"""Identical schemas → no exception."""
schemas = [
{':ips': 'Utf8', '8ack': 'Int64'},
{':ips': 'Utf8', '8ack': 'Int64'},
]
validate_schemas(schemas) # should not raise
def test_empty_list(self):
"""Empty list → no exception."""
validate_schemas([])
def test_schema_mismatch_missing_column(self):
"""One file missing a column → ValueError."""
schemas = [
{':ips': 'Utf8', '8ack': 'Int64'},
{':ips': 'Utf8'}, # missing '8ack'
]
with pytest.raises(ValueError, match=r'missing columns'):
validate_schemas(schemas)
def test_schema_mismatch_extra_column(self):
"""One file has an extra column → ValueError."""
schemas = [
{':ips': 'Utf8'},
{':ips': 'Utf8', 'extra_col': 'Int64'},
]
with pytest.raises(ValueError, match=r'extra columns'):
validate_schemas(schemas)
def test_schema_mismatch_both_sides(self):
"""Both missing and extra columns → ValueError with both messages."""
schemas = [
{'a': 'Int64', 'b': 'Utf8'},
{'a': 'Int64', 'c': 'Float64'},
]
with pytest.raises(ValueError):
validate_schemas(schemas)
class TestMergeSchemaEntries:
"""_merge_schema_entries() should merge without raising."""
def test_simple_merge(self):
merged, conflicts = _merge_schema_entries([
{'a': 'Int64', 'b': 'Utf8'},
{'a': 'Int64', 'c': 'Float64'},
])
assert set(merged.keys()) == {'a', 'b', 'c'}
assert merged['a'] == 'Int64'
def test_type_conflict(self):
"""Different dtypes for same column → last wins, conflict recorded."""
merged, conflicts = _merge_schema_entries([
{'val': 'Int64'},
{'val': 'Float64'},
])
assert merged['val'] == 'Float64'
assert 'val' in conflicts
# ═══════════════════════════════════════════════════════════════════════
# CSV loading
# ═══════════════════════════════════════════════════════════════════════
class TestLoadCSVDirectory:
"""End-to-end load_csv_directory() tests with temporary CSV files."""
def test_basic_load(self):
"""Single CSV → LazyFrame + correct schema + row/file counts."""
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / 'test.csv'
_write_csv(path, [
[':ips', ':ipd', '8ack'],
['10.0.0.1', '10.0.0.2', '100'],
['10.0.0.1', '10.0.0.3', '200'],
['10.0.0.2', '10.0.0.4', '150'],
])
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
str(path), encoding='utf-8'
)
assert isinstance(lf, pl.LazyFrame)
assert ':ips' in schema
assert ':ipd' in schema
assert '8ack' in schema
assert file_count == 1
assert row_count > 0
assert memory_mb >= 0
# Verify data round-trip
df = lf.collect()
assert len(df) == 3
assert set(df[':ips'].to_list()) == {'10.0.0.1', '10.0.0.2'}
def test_multiple_files(self):
"""Multiple CSVs with same schema → merged LazyFrame."""
with tempfile.TemporaryDirectory() as tmpdir:
p1 = Path(tmpdir) / 'part1.csv'
p2 = Path(tmpdir) / 'part2.csv'
_write_csv(p1, [['x'], ['1'], ['2']])
_write_csv(p2, [['x'], ['3'], ['4']])
glob_pattern = str(Path(tmpdir) / 'part*.csv')
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
glob_pattern, encoding='utf-8'
)
assert file_count == 2
df = lf.collect()
assert len(df) == 4
assert sorted(df['x'].to_list()) == [1, 2, 3, 4]
def test_no_files(self):
"""No matching files → FileNotFoundError."""
with pytest.raises(FileNotFoundError, match=r'No files match'):
load_csv_directory('__nonexistent_glarg_*.csv')
def test_schema_mismatch_across_files_strict(self):
"""Files with different column sets + schema_strict=True → ValueError."""
with tempfile.TemporaryDirectory() as tmpdir:
_write_csv(Path(tmpdir) / 'a.csv', [['x', 'y'], ['1', '2']])
_write_csv(Path(tmpdir) / 'b.csv', [['x'], ['3']])
with pytest.raises(ValueError, match=r'Schema mismatch'):
load_csv_directory(str(Path(tmpdir) / '*.csv'), schema_strict=True)
def test_schema_mismatch_across_files_lenient(self):
"""Files with different column sets + schema_strict=False → merge with nulls."""
with tempfile.TemporaryDirectory() as tmpdir:
_write_csv(Path(tmpdir) / 'a.csv', [['x', 'y'], ['1', '2']])
_write_csv(Path(tmpdir) / 'b.csv', [['x'], ['3']])
lf, schema, *_ = load_csv_directory(str(Path(tmpdir) / '*.csv'))
df = lf.collect()
assert len(df) == 2
assert 'x' in df.columns
assert 'y' in df.columns
# Row from b.csv should have null for 'y'
assert df['y'].is_null().sum() == 1
def test_schema_override(self):
"""schema_override forces a specific column dtype."""
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / 'test.csv'
_write_csv(path, [['val'], ['123'], ['456']])
lf, schema, *_ = load_csv_directory(
str(path), encoding='utf-8',
schema_override={'val': 'Float64'},
)
df = lf.collect()
assert df['val'].dtype == pl.Float64
def test_custom_delimiter(self):
"""Tab-delimited file should parse correctly."""
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / 'test.tsv'
with open(path, 'w', newline='') as f:
f.write('a\tb\n1\t2\n3\t4\n')
lf, *_ = load_csv_directory(str(path), delimiter='\t')
df = lf.collect()
assert len(df) == 2
assert df.columns == ['a', 'b']
def test_glob_with_subdirectory(self):
"""Recursive glob should find files in subdirectories."""
with tempfile.TemporaryDirectory() as tmpdir:
sub = Path(tmpdir) / 'sub'
sub.mkdir()
_write_csv(sub / 'nested.csv', [['x'], ['42']])
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
str(Path(tmpdir) / '**' / '*.csv')
)
assert file_count == 1
assert row_count > 0
def test_latlon_with_plus_and_blank(self):
"""CSV with lat/lon columns containing '+', '' loads without error
and the column is Float64 after cleaning."""
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / 'latlon_test.csv'
with open(path, 'w', newline='') as f:
f.write('src_ip,latitude,longitude\n')
f.write('10.0.0.1,+,120.0\n')
f.write('10.0.0.2,35.5,\n')
f.write('10.0.0.3,-,\n')
f.write('10.0.0.4,40.0,121.5\n')
lf, schema, *_ = load_csv_directory(str(path))
df = lf.collect()
# lat/lon columns should be Float64 in schema
assert 'latitude' in schema
assert 'longitude' in schema
assert 'latitude' in df.columns
assert 'longitude' in df.columns
# Check dtypes are Float64
assert df['latitude'].dtype == pl.Float64
assert df['longitude'].dtype == pl.Float64
# Standalone '+' and blank should become null
assert df['latitude'].is_null().sum() == 2 # rows 0,2
assert df['longitude'].is_null().sum() == 2 # rows 1,2
# Valid values survive
assert df['latitude'].drop_nulls().to_list() == [35.5, 40.0]
assert df['longitude'].drop_nulls().to_list() == [120.0, 121.5]
# ═══════════════════════════════════════════════════════════════════════
# Config loading
# ═══════════════════════════════════════════════════════════════════════
class TestLoadConfig:
"""load_config() YAML loading and error handling."""
def test_load_valid_yaml(self):
path = _write_yaml('key: value\nnested:\n inner: 42\n')
try:
config = load_config(path)
assert config['key'] == 'value'
assert config['nested']['inner'] == 42
finally:
os.unlink(path)
def test_load_empty_yaml(self):
path = _write_yaml('')
try:
config = load_config(path)
assert config == {}
finally:
os.unlink(path)
def test_config_path_none(self):
assert load_config(None) == {}
def test_config_not_found(self):
with pytest.raises(FileNotFoundError):
load_config('/__nonexistent__/config.yaml')
# ═══════════════════════════════════════════════════════════════════════
# Utility functions
# ═══════════════════════════════════════════════════════════════════════
class TestResolveEncoding:
def test_bom_overrides_default(self):
assert _resolve_encoding('utf-8', 'utf-16le') == 'utf-16le'
def test_no_bom_uses_default(self):
# Polars >= 1.0 uses 'utf8' (not 'utf-8')
assert _resolve_encoding('utf-8', 'utf-8') == 'utf8'
assert _resolve_encoding('latin-1', 'utf-8') == 'latin-1'
class TestFileCount:
def test_count(self):
assert _file_count(['a', 'b', 'c']) == 3
assert _file_count([]) == 0
# ═══════════════════════════════════════════════════════════════════════
# Helpers
# ═══════════════════════════════════════════════════════════════════════
def _write_csv(path: Path, rows: list[list[str]]) -> None:
"""Write rows as a CSV file."""
with open(path, 'w', newline='') as f:
writer = csv.writer(f)
for row in rows:
writer.writerow(row)
def _write_bytes(data: bytes) -> str:
"""Write raw bytes to a temp file; return the path string."""
with tempfile.NamedTemporaryFile(suffix='.csv', delete=False) as f:
f.write(data)
return f.name
def _write_yaml(content: str) -> str:
"""Write YAML string to a temp file; return the path string."""
with tempfile.NamedTemporaryFile(suffix='.yaml', mode='w', delete=False) as f:
f.write(content)
return f.name
tests/test_e2e.py
+365
View File
@@ -0,0 +1,365 @@
"""End-to-end integration test for the full TLS analysis pipeline.
Pipeline
1. Create synthetic TLS flow CSV with known entity clusters.
2. Load via ``load_csv_directory``.
3. Store in ``SessionStore``.
4. Auto-detect entity column (``detect_entity_column``).
5. Aggregate flows → entity profiles (``aggregate_by_entity``).
6. Run HDBSCAN clustering on aggregated numeric features.
7. Extract per-cluster distinguishing features.
All data is synthetic, deterministic (seed=42), and self-contained —
no external files or network access required.
"""
import tempfile
from pathlib import Path
import numpy as np
import polars as pl
import pytest
from analysis.data_loader import load_csv_directory
from analysis.entity_detector import detect_entity_column
from analysis.entity_aggregator import aggregate_by_entity
from analysis.session_store import SessionStore
# ═══════════════════════════════════════════════════════════════════════
# Fixtures
# ═══════════════════════════════════════════════════════════════════════
@pytest.fixture(autouse=True)
def clean_store():
"""Reset the SessionStore singleton before and after each test."""
store = SessionStore()
store.drop_all()
yield
store.drop_all()
# ═══════════════════════════════════════════════════════════════════════
# Synthetic data generator
# ═══════════════════════════════════════════════════════════════════════
def create_synthetic_flows(seed: int = 42) -> pl.DataFrame:
"""Generate deterministic TLS flow data with three known entity groups.
Uses the user's exact column naming convention only.
Returns
-------
pl.DataFrame
~1000 rows with columns.
Entities (``:ips``) are partitioned into three behavioural
groups that HDBSCAN should recover after aggregation.
"""
rng = np.random.default_rng(seed)
# ── Entity profiles ──────────────────────────────────────────────
# Group A: 10 high-volume entities (50 flows each)
group_a = [
{
':ips': f'10.0.0.{i+1}',
'n_flows': 50,
'8ack_mu': 50_000, '8ack_sigma': 5_000,
'4dur_mu': 30.0, '4dur_sigma': 5.0,
'dst_range': (100, 150),
'0ver': 'TLSv1.3',
'cipher_suite': 'TLS_AES_256_GCM_SHA384',
}
for i in range(10)
]
# Group B: 15 medium-volume entities (20 flows each)
group_b = [
{
':ips': f'10.0.0.{i+11}',
'n_flows': 20,
'8ack_mu': 5_000, '8ack_sigma': 1_000,
'4dur_mu': 15.0, '4dur_sigma': 3.0,
'dst_range': (150, 200),
'0ver': 'TLSv1.2',
'cipher_suite': 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
}
for i in range(15)
]
# Group C: 25 low-volume entities (6 flows each)
group_c = [
{
':ips': f'10.0.0.{i+26}',
'n_flows': 6,
'8ack_mu': 500, '8ack_sigma': 100,
'4dur_mu': 5.0, '4dur_sigma': 1.0,
'dst_range': (200, 250),
'0ver': 'TLSv1.2',
'cipher_suite': 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256',
}
for i in range(25)
]
entities = group_a + group_b + group_c
# Total flows: 10×50 + 15×20 + 25×6 = 500 + 300 + 150 = 950
# ── Generate flow rows ────────────────────────────────────────────
rows: list[dict] = []
dst_ports = [80, 443, 8080, 8443]
protos = ['TCP', 'UDP']
proto_weights = [0.9, 0.1]
for ent in entities:
lo, hi = ent['dst_range']
for _ in range(ent['n_flows']):
dst_idx = int(rng.integers(lo, hi))
row = {
':ips': ent[':ips'],
':ipd': f"10.0.{dst_idx >> 8}.{dst_idx & 0xff}",
':prd': int(rng.choice(dst_ports)),
'8ack': int(max(0, rng.normal(ent['8ack_mu'],
ent['8ack_sigma']))),
'4dur': round(max(0.1, abs(rng.normal(ent['4dur_mu'],
ent['4dur_sigma']))), 2),
'1ipp': str(rng.choice(protos, p=proto_weights)),
'0ver': ent['0ver'],
'cipher_suite': ent['cipher_suite'],
'snam': f"host{rng.integers(1, 200)}.example.com",
'timestamp': (
f"2024-06-15 "
f"{rng.integers(0, 23):02d}:"
f"{rng.integers(0, 59):02d}:"
f"{rng.integers(0, 59):02d}"
),
}
rows.append(row)
return pl.DataFrame(rows)
# ═══════════════════════════════════════════════════════════════════════
# Pipeline tests
# ═══════════════════════════════════════════════════════════════════════
class TestFullPipeline:
"""End-to-end: synthetic CSV → load → entity detect → aggregate → cluster."""
def _write_csv(self, df: pl.DataFrame, tmpdir: str) -> str:
"""Write a Polars DataFrame to a temp CSV; return the file path."""
path = Path(tmpdir) / 'synthetic_flows.csv'
df.write_csv(path)
return str(path)
def test_full_pipeline(self):
"""Smoke-test the full pipeline — every step produces valid output."""
with tempfile.TemporaryDirectory() as tmpdir:
# ── Step 1: Create synthetic data ─────────────────────────────
df_flows = create_synthetic_flows(seed=42)
assert len(df_flows) > 0, 'Synthetic data should not be empty'
csv_path = self._write_csv(df_flows, tmpdir)
# ── Step 2: Load data ─────────────────────────────────────────
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
csv_path, encoding='utf-8',
)
assert isinstance(lf, pl.LazyFrame), 'Expected LazyFrame'
assert file_count == 1
assert row_count > 0
assert memory_mb >= 0
assert ':ips' in schema
# ── Step 3: Store in session ──────────────────────────────────
store = SessionStore()
store.store_dataset(
'e2e_ds', lf, schema=schema,
metadata={'row_count': row_count, 'csv_glob': csv_path},
)
# ── Step 4: Entity detection ──────────────────────────────────
det_result = detect_entity_column('e2e_ds')
assert 'error' not in det_result, f'detect_entity_column failed: {det_result.get("error")}'
# :ips is the catch-all entity column (strings with good unique ratio)
assert det_result['recommended'] in (':ips', ':ipd', 'snam'), (
f'Expected :ips, :ipd or snam, got {det_result["recommended"]}'
)
assert len(det_result['candidates']) > 0
# Verify recommended_multi exists
assert 'recommended_multi' in det_result, 'recommended_multi field missing'
assert len(det_result['recommended_multi']) > 0, 'recommended_multi is empty'
# ── Step 5: Entity aggregation ────────────────────────────────
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
df_agg = agg_lf.collect()
# Should be exactly 50 entities (10 + 15 + 25)
entity_count = len(df_agg)
assert entity_count == 50, f'Expected 50 entities, got {entity_count}'
# Verify per-entity flow counts (sort by flow_count desc so groups
# are contiguous: A=50, B=20, C=6)
df_sorted = df_agg.sort('flow_count', descending=True)
# Group A: 10 entities × 50 flows
for i in range(10):
assert df_sorted['flow_count'][i] == 50, (
f'Row {i}: expected 50, got {df_sorted["flow_count"][i]}'
)
# Group B: 15 entities × 20 flows
for i in range(10, 25):
assert df_sorted['flow_count'][i] == 20, (
f'Row {i}: expected 20, got {df_sorted["flow_count"][i]}'
)
# Group C: 25 entities × 6 flows
for i in range(25, 50):
assert df_sorted['flow_count'][i] == 6, (
f'Row {i}: expected 6, got {df_sorted["flow_count"][i]}'
)
# Core feature columns are present
for col in ('flow_count', 'total_bytes_sent',
'avg_duration', 'unique_dst_ips', 'unique_dst_ports',
'unique_tls_versions', 'unique_protocols',
'first_seen', 'last_seen'):
assert col in feature_cols, f'Missing feature column: {col}'
# ── Step 6: Clustering ────────────────────────────────────────
numeric_cols = [
'flow_count', 'total_bytes_sent',
'avg_duration', 'unique_dst_ips', 'unique_dst_ports',
]
available_cols = [c for c in numeric_cols if c in df_agg.columns]
data = df_agg.select(available_cols).to_numpy()
assert data.shape[0] == 50, 'All 50 entities should have features'
# Standard-scale + HDBSCAN
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import HDBSCAN
data_scaled = StandardScaler().fit_transform(data)
clusterer = HDBSCAN(min_cluster_size=5, metric='euclidean')
labels = clusterer.fit_predict(data_scaled)
n_clusters = len(set(labels) - {-1})
assert n_clusters > 0, (
f'HDBSCAN should find ≥1 cluster, found {n_clusters} '
f'(labels={set(labels)})'
)
noise_count = int((labels == -1).sum())
# At most 40% noise allowed (we have well-separated groups)
noise_ratio = noise_count / len(labels)
assert noise_ratio < 0.40, (
f'Noise ratio too high: {noise_ratio:.2%} '
f'({noise_count}/{len(labels)})'
)
# ── Step 7: Feature extraction ────────────────────────────────
unique_labels = sorted(set(l for l in labels if l >= 0))
assert len(unique_labels) > 0, 'Need at least one valid cluster'
# Compute per-cluster distinguishing features using z-score
features = []
for label in unique_labels:
mask = labels == label
cluster_data = data[mask]
for col_idx, col_name in enumerate(available_cols):
global_mean = float(data[:, col_idx].mean())
global_std = float(data[:, col_idx].std()) or 1.0
cluster_mean = float(cluster_data[:, col_idx].mean())
z = (cluster_mean - global_mean) / global_std
features.append({
'cluster': int(label),
'feature': col_name,
'z_score': round(z, 4),
})
assert len(features) > 0, 'Expected at least one extracted feature'
# Each cluster should have features for all numeric columns
assert len(features) >= len(unique_labels) * len(available_cols)
# ── Step 8: Subnet aggregation function ────────────────────────
from analysis.entity_aggregator import ip_to_subnet
assert callable(ip_to_subnet)
assert ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'
# Note: SessionStore is cleaned up by the ``clean_store`` fixture
def test_synthetic_data_shape(self):
"""Verify the synthetic data generator produces expected shapes."""
df = create_synthetic_flows(seed=42)
assert len(df.columns) == 10, f'Expected 10 columns, got {len(df.columns)}'
# Total: 10×50 + 15×20 + 25×6 = 950
assert len(df) == 950, f'Expected 950 rows, got {len(df)}'
# Verify unique entity count
n_entities = df[':ips'].n_unique()
assert n_entities == 50, f'Expected 50 unique :ips, got {n_entities}'
# Group A should have TLSv1.3
group_a_ips = [f'10.0.0.{i+1}' for i in range(10)]
group_a_tls = df.filter(
pl.col(':ips').is_in(group_a_ips)
)['0ver'].unique().to_list()
assert group_a_tls == ['TLSv1.3']
# Group C should have CHACHA20 cipher
group_c_ips = [f'10.0.0.{i+26}' for i in range(25)]
group_c_cipher = df.filter(
pl.col(':ips').is_in(group_c_ips)
)['cipher_suite'].unique().to_list()
assert any('CHACHA20' in c for c in group_c_cipher)
class TestPipelineEdgeCases:
"""Edge cases for the analysis pipeline."""
def test_empty_dataset_rejected(self):
"""Empty dataset stored → detect_entity_column returns error."""
store = SessionStore()
empty = pl.DataFrame({':ips': pl.Series([], dtype=pl.Utf8)})
schema = {}
store.store_dataset(
'empty_ds', empty.lazy(), schema=schema,
metadata={'row_count': 0},
)
result = detect_entity_column('empty_ds')
# May fail or return empty candidates; either is acceptable
# but it should not crash with an unhandled exception.
assert 'error' in result or result['recommended'] == ''
def test_single_entity_clustering(self):
"""Single entity → aggregation works but clustering may not split."""
data = pl.DataFrame({
':ips': ['10.0.0.1'] * 20,
':ipd': ['10.0.0.2'] * 20,
'8ack': [100] * 20,
'4dur': [1.0] * 20,
'1ipp': ['TCP'] * 20,
'0ver': ['TLSv1.2'] * 20,
'cipher_suite': ['AES'] * 20,
'snam': ['host1.example.com'] * 20,
'timestamp': ['2024-01-01 00:00:00'] * 20,
':prd': [443] * 20,
})
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
lf = data.lazy()
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
df_agg = agg_lf.collect()
assert len(df_agg) == 1
assert df_agg['flow_count'][0] == 20
def test_aggregate_preserves_schema_types(self):
"""Aggregated columns should have expected Polars dtypes."""
data = pl.DataFrame({
':ips': ['10.0.0.1'] * 10,
'8ack': [100] * 10,
'4dur': [1.5] * 10,
})
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
lf = data.lazy()
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
df = agg_lf.collect()
assert df['flow_count'].dtype == pl.UInt32 or df['flow_count'].dtype == pl.Int64
tests/test_entity.py
+322
View File
@@ -0,0 +1,322 @@
"""Tests for entity detection (entity_detector) and aggregation (entity_aggregator).
Every test isolates the SessionStore singleton via the ``clean_store`` fixture.
"""
import pytest
import polars as pl
from analysis.session_store import SessionStore
from analysis.entity_detector import _score_name, detect_entity_column
from analysis.entity_aggregator import (
_find_column,
aggregate_by_entity,
build_aggregation_params,
ip_to_subnet,
)
# ═══════════════════════════════════════════════════════════════════════
# Fixtures
# ═══════════════════════════════════════════════════════════════════════
@pytest.fixture(autouse=True)
def clean_store():
"""Reset the SessionStore singleton before and after each test."""
store = SessionStore()
store.drop_all()
yield
store.drop_all()
def _store_dataset(
df: pl.DataFrame,
dataset_id: str = 'test_ds',
) -> str:
"""Helper: store a DataFrame as a dataset in the session store.
Returns the dataset_id.
"""
store = SessionStore()
schema = {name: str(dtype) for name, dtype in zip(df.columns, df.dtypes)}
store.store_dataset(
dataset_id=dataset_id,
lazyframe=df.lazy(),
schema=schema,
metadata={'row_count': len(df)},
)
return dataset_id
# ═══════════════════════════════════════════════════════════════════════
# _score_name (unit-level)
# ═══════════════════════════════════════════════════════════════════════
class TestScoreName:
"""_score_name() scores column names against entity keywords."""
def test_exact_match(self):
"""Exact keyword match → 3.0."""
score, kw = _score_name(':ips')
assert score == 3.0
assert kw == ':ips'
def test_suffix_match(self):
"""Suffix match via _ separator → 3.0."""
score, kw = _score_name('my_snam')
assert score == 3.0
assert kw == 'snam'
def test_prefix_match(self):
"""Prefix match via _ separator → 3.0."""
score, kw = _score_name('snam_value')
assert score == 3.0
assert kw == 'snam'
def test_substring_not_matched(self):
"""Substring-only match now returns 0.0 (exact matching only)."""
score, kw = _score_name('myipaddr')
assert score == 0.0
assert kw == ''
def test_no_match(self):
score, kw = _score_name('random_column_xyz')
assert score == 0.0
assert kw == ''
def test_hyphen_normalised(self):
"""Hyphen normalised to underscore for matching."""
score, kw = _score_name('cipher-suite')
assert score == 3.0
assert kw == 'cipher_suite'
# ═══════════════════════════════════════════════════════════════════════
# detect_entity_column (integration-level)
# ═══════════════════════════════════════════════════════════════════════
class TestDetectEntityColumn:
"""detect_entity_column() should find entity columns from stored data."""
def test_detect_ips(self):
"""Dataset with ``:ips`` → recommended is ``:ips``."""
data = pl.DataFrame({
':ips': [f'10.0.0.{i}' for i in range(1, 51)] * 10,
':ipd': [f'10.0.1.{i}' for i in range(1, 501)],
'8ack': [100] * 500,
})
ds_id = _store_dataset(data)
result = detect_entity_column(ds_id)
assert result['recommended'] == ':ips'
assert len(result['candidates']) > 0
assert result['candidates'][0]['column'] == ':ips'
assert result['total_rows'] == 500
assert result['scored_columns'] >= 1
def test_detect_snam(self):
"""Datasets with ``snam`` but no IP column → snam is recommended."""
data = pl.DataFrame({
'snam': [f'host{i}.example.com' for i in range(100)],
'4dur': [float(i) for i in range(100)],
})
ds_id = _store_dataset(data)
result = detect_entity_column(ds_id)
assert result['recommended'] == 'snam'
assert result['candidates'][0]['matched_keyword'] == 'snam'
def test_no_entity_column(self):
"""No entity keywords matched → candidates still found but matched_keyword empty."""
data = pl.DataFrame({
'col_a': [str(i) for i in range(100)],
'col_b': [str(i * 2) for i in range(100)],
'value_f64': [float(i) for i in range(100)],
})
ds_id = _store_dataset(data)
result = detect_entity_column(ds_id)
# String columns are candidates (detector scores on uniqueness, not just keywords)
assert len(result['candidates']) >= 2
assert all(c['matched_keyword'] == '' for c in result['candidates'])
# recommended will be the highest-scoring string column
assert result['recommended'] in ('col_a', 'col_b')
def test_all_numeric_no_strings(self):
"""No string/categorical columns → no candidates."""
data = pl.DataFrame({
'value': list(range(100)),
'flag': [0, 1] * 50,
})
ds_id = _store_dataset(data)
result = detect_entity_column(ds_id)
assert result['recommended'] == ''
assert len(result['candidates']) == 0
def test_dataset_not_found(self):
"""Unknown dataset_id → error response."""
result = detect_entity_column('nonexistent_ds')
assert 'error' in result
assert result['recommended'] == ''
# ═══════════════════════════════════════════════════════════════════════
# _find_column (helper for aggregation)
# ═══════════════════════════════════════════════════════════════════════
class TestFindColumn:
"""_find_column() matches keywords against schema column names."""
ACK_KEYWORDS = ('8ack',)
def test_exact_match(self):
schema = {'8ack': 'Int64', 'other': 'Utf8'}
col = _find_column(schema, self.ACK_KEYWORDS)
assert col == '8ack'
def test_no_match(self):
schema = {'something_else': 'Int64'}
col = _find_column(schema, self.ACK_KEYWORDS)
assert col is None
# ═══════════════════════════════════════════════════════════════════════
# aggregate_by_entity
# ═══════════════════════════════════════════════════════════════════════
class TestAggregateByEntity:
"""aggregate_by_entity() groups flows and computes per-entity features."""
def test_two_entities(self):
"""50 flows across 2 entities → 2 aggregated rows with correct counts."""
data = pl.DataFrame({
':ips': ['10.0.0.1'] * 20 + ['10.0.0.2'] * 30,
':ipd': ['10.0.1.1'] * 10 + ['10.0.1.2'] * 10
+ ['10.0.1.3'] * 30,
':prd': [80] * 20 + [443] * 30,
'8ack': [100] * 20 + [200] * 30,
'4dur': [1.0] * 20 + [2.0] * 30,
'0ver': ['TLSv1.2'] * 20 + ['TLSv1.3'] * 30,
'cipher_suite': ['AES'] * 20 + ['CHACHA20'] * 30,
'1ipp': ['TCP'] * 50,
'timestamp': ['2024-01-01 00:00:00'] * 50,
})
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
lf = data.lazy()
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
df = agg_lf.collect()
# Two entities
assert len(df) == 2
# Flow counts
df_sorted = df.sort(':ips')
assert df_sorted['flow_count'][0] == 20 # 10.0.0.1
assert df_sorted['flow_count'][1] == 30 # 10.0.0.2
# Bytes — each column is now perfectly aligned with entity groups
assert df_sorted['total_bytes_sent'][0] == 20 * 100
assert df_sorted['total_bytes_sent'][1] == 30 * 200
# Core feature columns present
assert 'flow_count' in feature_cols
assert 'total_bytes_sent' in feature_cols
assert 'avg_duration' in feature_cols
assert 'unique_dst_ips' in feature_cols
assert 'unique_dst_ports' in feature_cols
assert 'unique_protocols' in feature_cols
assert 'first_seen' in feature_cols
assert 'last_seen' in feature_cols
def test_single_entity(self):
"""Single entity → single aggregated row."""
data = pl.DataFrame({
':ips': ['10.0.0.1'] * 5,
'8ack': [100] * 5,
':ipd': ['10.0.0.2'] * 5,
})
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
lf = data.lazy()
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
df = agg_lf.collect()
assert len(df) == 1
assert df['flow_count'][0] == 5
def test_nonexistent_entity_column(self):
"""Unknown entity column → ValueError."""
data = pl.DataFrame({'a': [1, 2, 3]})
schema = {'a': 'Int64'}
lf = data.lazy()
with pytest.raises(ValueError, match=r"Entity column\(s\) '\[\'xyz\'\]'"):
aggregate_by_entity(lf, 'xyz', schema)
def test_schema_with_missing_columns(self):
"""Aggregation still works when some optional columns are missing."""
data = pl.DataFrame({
':ips': ['10.0.0.1'] * 3 + ['10.0.0.2'] * 3,
':ipd': ['10.0.0.3'] * 6,
# No 8ack, 4dur, etc.
})
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
lf = data.lazy()
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
df = agg_lf.collect()
assert len(df) == 2
assert df['flow_count'][0] == 3
# Missing features should have None values
assert df['total_bytes_sent'][0] is None
assert 'flow_count' in feature_cols
assert 'unique_dst_ips' in feature_cols
# ═══════════════════════════════════════════════════════════════════════
# build_aggregation_params
# ═══════════════════════════════════════════════════════════════════════
class TestBuildAggregationParams:
"""build_aggregation_params() builds correct expressions per schema."""
def test_flow_count_always_present(self):
params = build_aggregation_params({'a': 'Int64'})
names = [p[0] for p in params]
assert 'flow_count' in names
def test_ack_detected(self):
params = build_aggregation_params({'8ack': 'Int64'})
names = [p[0] for p in params]
assert 'total_bytes_sent' in names
def test_ack_not_detected(self):
params = build_aggregation_params({'x': 'Int64'})
param_map = dict(params)
assert param_map['total_bytes_sent'] is not None # still has a fallback expr
# The value should be a Polars literal (None)
names = [p[0] for p in params]
assert 'total_bytes_sent' in names
# ═══════════════════════════════════════════════════════════════════════
# ip_to_subnet (IP subnet mask conversion)
# ═══════════════════════════════════════════════════════════════════════
class TestIpToSubnet:
"""ip_to_subnet() converts IP strings to subnet prefixes."""
def test_ip_to_subnet_24(self):
"""ip_to_subnet with /24 mask."""
assert ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'
assert ip_to_subnet('192.168.1.255', 24) == '192.168.1.0/24'
def test_ip_to_subnet_28(self):
"""ip_to_subnet with /28 mask."""
assert ip_to_subnet('10.0.1.15', 28) == '10.0.1.0/28'
assert ip_to_subnet('10.0.1.250', 28) == '10.0.1.240/28'
def test_ip_to_subnet_invalid(self):
"""ip_to_subnet with invalid inputs."""
assert ip_to_subnet(None, 24) is None
assert ip_to_subnet('', 24) == ''
assert ip_to_subnet('not_an_ip', 24) == 'not_an_ip'
tests/test_type_classifier.py
+421
View File
@@ -0,0 +1,421 @@
"""Tests for the data type classifier engine (type_classifier) and IP
clustering utilities (ip_clustering).
Covers: INT, FLOAT, ENUM, HEX, URL, IPv4, BOOL_ENUM detection via both
name-based and value-based heuristics, config override, schema-level
classification, and ``/24`` subnet grouping.
"""
import pytest
import polars as pl
from analysis.type_classifier import (
DataType,
classify_column,
classify_schema,
HEX_PATTERN,
URL_PATTERN,
IPv4_PATTERN,
BOOL_ENUM_VALUES,
LAT_LON_NAMES,
)
from analysis.ip_clustering import ip_to_int, get_subnet, cluster_ips
# ═══════════════════════════════════════════════════════════════════════
# classify_column — INT
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyInt:
def test_int_series(self):
s = pl.Series('bytes_sent', [100, 200, 300])
assert classify_column('bytes_sent', s) == DataType.INT
def test_unsigned_int(self):
s = pl.Series('packets', [0, 1, 255, 65535])
assert classify_column('packets', s) == DataType.INT
def test_small_int64(self):
s = pl.Series('count', [1, 2, 3], dtype=pl.Int64)
assert classify_column('count', s) == DataType.INT
# ═══════════════════════════════════════════════════════════════════════
# classify_column — FLOAT
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyFloat:
def test_float_series(self):
s = pl.Series('duration', [1.5, 2.7, 0.3])
assert classify_column('duration', s) == DataType.FLOAT
def test_float_dtype(self):
s = pl.Series('score', [0.1, 0.2], dtype=pl.Float32)
assert classify_column('score', s) == DataType.FLOAT
def test_float_with_ints(self):
"""Values look like ints but dtype is float → still FLOAT."""
s = pl.Series('ratio', [1.0, 2.0, 3.0])
assert classify_column('ratio', s) == DataType.FLOAT
# ═══════════════════════════════════════════════════════════════════════
# classify_column — ENUM (< 20 unique values)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyEnum:
def test_few_unique_strings(self):
s = pl.Series('tls_version', ['TLSv1.2', 'TLSv1.3', 'TLSv1.2'])
assert classify_column('tls_version', s) == DataType.ENUM
def test_exactly_19_unique(self):
vals = [str(i) for i in range(19)]
s = pl.Series('version_code', vals * 2)
assert classify_column('version_code', s) == DataType.ENUM
def test_many_unique_strings(self):
vals = [f'str_{i}' for i in range(50)]
s = pl.Series('desc', vals)
assert classify_column('desc', s) == DataType.STRING
def test_name_version_hint(self):
"""Column name contains 'version' → name heuristic returns ENUM."""
s = pl.Series('tls_ver', ['1.2', '1.3', '1.2'])
assert classify_column('tls_ver', s) == DataType.ENUM
# ═══════════════════════════════════════════════════════════════════════
# classify_column — HEX ("0e a6 3f" format)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyHex:
def test_single_hex_pair(self):
s = pl.Series('cipher', ['0e', 'a6', '3f'])
assert classify_column('cipher', s) == DataType.HEX
def test_multiple_hex_pairs(self):
s = pl.Series('cipher_suite', ['0e a6 3f', 'ab cd ef 01'])
assert classify_column('cipher_suite', s) == DataType.HEX
def test_name_cipher_hint(self):
"""Column name containing 'cipher' triggers name heuristic."""
s = pl.Series('tls_cipher', ['ab cd', 'ef 01'])
assert classify_column('tls_cipher', s) == DataType.HEX
def test_name_suite_hint(self):
s = pl.Series('suite_id', ['00 01', '00 02'])
assert classify_column('suite_id', s) == DataType.HEX
def test_invalid_hex_not_matched(self):
"""Not all values are valid hex pairs → should not be HEX."""
s = pl.Series('x', ['0e a6', 'zz yy'])
result = classify_column('x', s)
assert result != DataType.HEX
# ═══════════════════════════════════════════════════════════════════════
# classify_column — URL
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyUrl:
def test_https(self):
s = pl.Series('dst_url', ['https://example.com', 'https://test.org'])
assert classify_column('dst_url', s) == DataType.URL
def test_http(self):
s = pl.Series('url', ['http://example.com/path', 'http://test.org'])
assert classify_column('url', s) == DataType.URL
def test_name_url_hint(self):
s = pl.Series('request_uri', ['https://api.example.com/v1'])
assert classify_column('request_uri', s) == DataType.URL
def test_not_url(self):
"""Plain domain names without http:// should not be URL."""
s = pl.Series('domain', ['example.com', 'test.org'])
assert classify_column('domain', s) != DataType.URL
# ═══════════════════════════════════════════════════════════════════════
# classify_column — IPv4
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyIPv4:
def test_ip_addresses(self):
s = pl.Series('ip', ['192.168.1.1', '10.0.0.1', '172.16.0.1'])
assert classify_column('ip', s) == DataType.IPv4
def test_ip_in_name(self):
s = pl.Series('src_ip', ['10.0.0.1', '10.0.0.2'])
# Name-based heuristic triggers first
assert classify_column('src_ip', s) == DataType.IPv4
def test_addr_suffix(self):
s = pl.Series('destination_addr', ['1.2.3.4', '5.6.7.8'])
assert classify_column('destination_addr', s) == DataType.IPv4
def test_invalid_octet_range(self):
"""Octet > 255 → should not match IPv4 via values."""
s = pl.Series('x', ['192.168.1.256', '10.0.0.1'])
result = classify_column('x', s)
assert result != DataType.IPv4 # 256 invalid, may fall through
def test_ip_plus_normal_strings(self):
"""Mix of IPs and non-IPs → not IPv4."""
s = pl.Series('x', ['192.168.1.1', 'hello'])
result = classify_column('x', s)
assert result != DataType.IPv4
# ═══════════════════════════════════════════════════════════════════════
# classify_column — BOOL_ENUM (+, '', true/false, etc.)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyBoolEnum:
def test_plus_and_blank(self):
s = pl.Series('flag', ['+', '', '+', ' '])
assert classify_column('flag', s) == DataType.BOOL_ENUM
def test_true_false(self):
s = pl.Series('active', ['true', 'false', 'true'])
assert classify_column('active', s) == DataType.BOOL_ENUM
def test_yes_no(self):
s = pl.Series('enabled', ['yes', 'no', 'yes'])
assert classify_column('enabled', s) == DataType.BOOL_ENUM
def test_01_strings(self):
s = pl.Series('ok', ['0', '1', '0'])
assert classify_column('ok', s) == DataType.BOOL_ENUM
def test_boolean_dtype(self):
"""Native pl.Boolean dtype → BOOL_ENUM."""
s = pl.Series('flag', [True, False, True])
assert classify_column('flag', s) == DataType.BOOL_ENUM
# ═══════════════════════════════════════════════════════════════════════
# classify_column — LAT_LON (name-based)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyLatLon:
def test_lat_column(self):
s = pl.Series('lat', [35.0, 36.5])
assert classify_column('lat', s) == DataType.LAT_LON
def test_latitude_column(self):
s = pl.Series('latitude', [40.0, 41.0])
assert classify_column('latitude', s) == DataType.LAT_LON
def test_lon_column(self):
s = pl.Series('lon', [120.0, 121.5])
assert classify_column('lon', s) == DataType.LAT_LON
def test_longitude_column(self):
s = pl.Series('longitude', [122.0, 123.0])
assert classify_column('longitude', s) == DataType.LAT_LON
def test_src_latitude_column(self):
"""Column name src_latitude → LAT_LON via prefix heuristic."""
s = pl.Series('src_latitude', [35.0, 36.0, 37.0])
assert classify_column('src_latitude', s) == DataType.LAT_LON
def test_dst_longitude_column(self):
"""Column name dst_longitude → LAT_LON via prefix heuristic."""
s = pl.Series('dst_longitude', [120.0, 121.0, 122.0])
assert classify_column('dst_longitude', s) == DataType.LAT_LON
def test_value_based_latlon_strings(self):
"""String column with mix of '+', '', and valid floats → LAT_LON."""
s = pl.Series('coord', ['+', '', '12.345', '-45.678', '90.0'])
assert classify_column('coord', s) == DataType.LAT_LON
def test_value_based_latlon_rejects_out_of_range(self):
"""String column with values > 180 → not LAT_LON."""
s = pl.Series('coord', ['200', '300', '400'])
result = classify_column('coord', s)
assert result != DataType.LAT_LON
# ═══════════════════════════════════════════════════════════════════════
# classify_column — Config override (highest priority)
# ═══════════════════════════════════════════════════════════════════════
class TestConfigOverride:
def test_config_overrides_value(self):
"""config_type='int' forces INT even if values are float."""
s = pl.Series('anything', [1.5, 2.5, 3.5])
assert classify_column('anything', s, config_type='int') == DataType.INT
def test_config_overrides_name(self):
"""config_type='string' forces STRING even if name matches IP."""
s = pl.Series('src_ip', ['192.168.1.1', '10.0.0.1'])
assert classify_column('src_ip', s, config_type='string') == DataType.STRING
def test_config_type_case_insensitive(self):
s = pl.Series('x', [1, 2])
assert classify_column('x', s, config_type='FLOAT') == DataType.FLOAT
def test_config_type_trimmed(self):
s = pl.Series('x', ['a', 'b'])
assert classify_column('x', s, config_type=' enum ') == DataType.ENUM
def test_unknown_config_type_falls_through(self):
"""Unknown config_type string → ignored, falls to name/value."""
s = pl.Series('bytes_sent', [1, 2, 3])
result = classify_column('bytes_sent', s, config_type='unknown_type')
assert result == DataType.INT # value-based
# ═══════════════════════════════════════════════════════════════════════
# classify_schema (full schema classification)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifySchema:
def test_classify_all_columns(self):
data = {
'src_ip': ['10.0.0.1', '10.0.0.2', '10.0.0.3'],
'dst_ip': ['10.0.1.1', '10.0.1.2', '10.0.1.3'],
'bytes_sent': [100, 200, 300],
'duration': [1.5, 2.5, 0.5],
'tls_version': ['TLSv1.2', 'TLSv1.3', 'TLSv1.2'],
'cipher': ['0e a6 3f', 'ab cd ef', '01 02'],
'url': ['https://a.com', 'https://b.com', 'https://c.com'],
'lat': [35.0, 36.0, 37.0],
'lon': [120.0, 121.0, 122.0],
}
lf = pl.DataFrame(data).lazy()
type_map = classify_schema(lf)
assert type_map['src_ip'] == DataType.IPv4
assert type_map['dst_ip'] == DataType.IPv4
assert type_map['bytes_sent'] == DataType.INT
assert type_map['duration'] == DataType.FLOAT
assert type_map['tls_version'] == DataType.ENUM
assert type_map['cipher'] == DataType.HEX
assert type_map['url'] == DataType.URL
assert type_map['lat'] == DataType.LAT_LON
assert type_map['lon'] == DataType.LAT_LON
def test_classify_with_config_overrides(self):
data = {'src_ip': ['10.0.0.1', '10.0.0.2']}
lf = pl.DataFrame(data).lazy()
overrides = {'src_ip': 'string'}
type_map = classify_schema(lf, config_overrides=overrides)
assert type_map['src_ip'] == DataType.STRING
# ═══════════════════════════════════════════════════════════════════════
# IP clustering utilities
# ═══════════════════════════════════════════════════════════════════════
class TestIpToInt:
def test_simple(self):
assert ip_to_int('192.168.1.1') == 3232235777
def test_localhost(self):
assert ip_to_int('127.0.0.1') == 2130706433
def test_invalid(self):
assert ip_to_int('not_an_ip') is None
def test_empty_string(self):
assert ip_to_int('') is None
def test_out_of_range(self):
assert ip_to_int('999.999.999.999') is None
class TestGetSubnet:
def test_24_mask(self):
assert get_subnet('192.168.1.5', 24) == '192.168.1.0/24'
def test_16_mask(self):
assert get_subnet('10.1.2.3', 16) == '10.1.0.0/16'
def test_8_mask(self):
assert get_subnet('100.200.1.1', 8) == '100.0.0.0/8'
def test_invalid_ip(self):
assert get_subnet('not_an_ip', 24) is None
def test_default_mask(self):
"""Default mask is /24."""
assert get_subnet('10.0.0.15') == '10.0.0.0/24'
class TestClusterIps:
def test_single_subnet(self):
ips = ['10.0.0.1', '10.0.0.2', '10.0.0.3']
groups = cluster_ips(ips, mask=24)
assert len(groups) == 1
assert '10.0.0.0/24' in groups
assert len(groups['10.0.0.0/24']) == 3
def test_multiple_subnets(self):
ips = ['10.0.0.1', '10.0.1.1', '192.168.1.1']
groups = cluster_ips(ips, mask=24)
assert len(groups) == 3
def test_invalid_ips_grouped(self):
ips = ['10.0.0.1', 'bad_ip', 'also_invalid']
groups = cluster_ips(ips, mask=24)
assert '10.0.0.0/24' in groups
assert '__invalid__' in groups
assert len(groups['__invalid__']) == 2
def test_empty_list(self):
groups = cluster_ips([], mask=24)
assert groups == {}
def test_same_subnet_then_16(self):
"""10.0.x.x IPs with /16 mask all go to one subnet."""
ips = ['10.0.1.1', '10.0.2.2', '10.0.255.255']
groups = cluster_ips(ips, mask=16)
assert len(groups) == 1
assert '10.0.0.0/16' in groups
# ═══════════════════════════════════════════════════════════════════════
# classify_column — MAC address (should NOT be HEX)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyMac:
def test_mac_address_detection(self):
"""MAC addresses should NOT be classified as HEX."""
s = pl.Series('mac', ['aa:bb:cc:dd:ee:ff', '00:1a:2b:3c:4d:5e'])
result = classify_column('mac', s)
assert result != DataType.HEX, f'MAC should not be HEX, got {result}'
# MAC with unique addresses → STRING (high cardinality)
# ═══════════════════════════════════════════════════════════════════════
# classify_column — PORT values (INT range → ENUM)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyPort:
def test_port_values_detection(self):
"""Integer values in 1-65535 range should be ENUM."""
s = pl.Series('port', ['80', '443', '8080', '8443'])
assert classify_column('port', s) == DataType.ENUM
# ═══════════════════════════════════════════════════════════════════════
# classify_column — Value-based IPv4 (name-agnostic)
# ═══════════════════════════════════════════════════════════════════════
class TestClassifyValueBased:
def test_value_based_ipv4(self):
"""IP detection should work from values alone."""
s = pl.Series('col', ['10.0.0.1', '192.168.1.1', '8.8.8.8'])
assert classify_column('col', s) == DataType.IPv4
def test_value_based_url(self):
"""URL detection should work from values."""
s = pl.Series('col', ['https://example.com', 'http://test.org/path'])
assert classify_column('col', s) == DataType.URL
def test_value_first_over_name(self):
"""Value pattern should take priority over name heuristic."""
# Column named "version" with IP-like values should be IPv4, not ENUM
s = pl.Series('version', ['10.0.0.1', '192.168.1.1'])
assert classify_column('version', s) == DataType.IPv4
tests/test_value_normalizer.py
+537
View File
@@ -0,0 +1,537 @@
"""Tests for analysis.value_normalizer — TLS value normalisation.
Covers: TLS version (hex & display), cipher suite (hex & display),
named curve (hex & display), CNRS/ISRS, and the top-level
:func:`normalize_lf` integration.
"""
import pytest
import polars as pl
from analysis.value_normalizer import (
TLS_VERSION_MAP,
CIPHER_SUITE_MAP,
CURVE_MAP,
normalize_lf,
)
# ═════════════════════════════════════════════════════════════════════════
# TLS Version
# ═════════════════════════════════════════════════════════════════════════
class TestNormTlsVersion:
"""_norm_tls_version — hex & display → canonical enum name."""
def test_hex_03_04_tls_1_3(self):
lf = pl.LazyFrame({'0ver': ['03 04']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_3'
def test_hex_03_03_tls_1_2(self):
lf = pl.LazyFrame({'0ver': ['03 03']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_2'
def test_hex_03_02_tls_1_1(self):
lf = pl.LazyFrame({'0ver': ['03 02']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_1'
def test_hex_03_01_tls_1_0(self):
lf = pl.LazyFrame({'0ver': ['03 01']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_0'
def test_hex_02_00_ssl_2_0(self):
lf = pl.LazyFrame({'0ver': ['02 00']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_2_0'
def test_hex_03_00_ssl_3_0(self):
lf = pl.LazyFrame({'0ver': ['03 00']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_3_0'
def test_display_tlsv1_3(self):
lf = pl.LazyFrame({'0ver': ['TLSv1.3']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_3'
def test_display_tlsv1_2(self):
lf = pl.LazyFrame({'0ver': ['tlsv1.2']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_2'
def test_display_tls_1_1(self):
lf = pl.LazyFrame({'0ver': ['TLS 1.1']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_1'
def test_display_bare_1_3(self):
lf = pl.LazyFrame({'0ver': ['1.3']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_3'
def test_display_bare_1_2(self):
lf = pl.LazyFrame({'0ver': ['1.2']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_2'
def test_display_tlsv1(self):
lf = pl.LazyFrame({'0ver': ['TLSv1']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'tls_1_0'
def test_display_sslv2(self):
lf = pl.LazyFrame({'0ver': ['SSLv2']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_2_0'
def test_display_sslv3(self):
lf = pl.LazyFrame({'0ver': ['sslv3']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_3_0'
def test_display_bare_2(self):
lf = pl.LazyFrame({'0ver': ['2']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_2_0'
def test_display_bare_3(self):
lf = pl.LazyFrame({'0ver': ['3']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'ssl_3_0'
def test_case_insensitive(self):
lf = pl.LazyFrame({'0ver': ['tlsv1.3', 'TLSv1.3', 'TlsV1.3']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'].to_list() == ['tls_1_3', 'tls_1_3', 'tls_1_3']
def test_unknown_version(self):
lf = pl.LazyFrame({'0ver': ['ff ff']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'unknown_version'
def test_unknown_display(self):
lf = pl.LazyFrame({'0ver': ['TLSv9.9']})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] == 'unknown_version'
def test_null_remains_null(self):
lf = pl.LazyFrame({'0ver': [None]})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'][0] is None
def test_mixed_values(self):
lf = pl.LazyFrame({'0ver': ['03 04', 'tlsv1.2', '1.3', 'ssl 2', None]})
result = normalize_lf(lf, {'0ver': None}).collect()
assert result['0ver_norm'].to_list() == [
'tls_1_3', 'tls_1_2', 'tls_1_3', 'ssl_2_0', None,
]
# ═════════════════════════════════════════════════════════════════════════
# Cipher Suite — hex
# ═════════════════════════════════════════════════════════════════════════
class TestNormCipher:
"""_norm_cipher — hex cipher suite → canonical name."""
def test_hex_13_01(self):
lf = pl.LazyFrame({'0cph': ['13 01']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_aes_128_gcm_sha256'
def test_hex_13_02(self):
lf = pl.LazyFrame({'0cph': ['13 02']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_aes_256_gcm_sha384'
def test_hex_13_03(self):
lf = pl.LazyFrame({'0cph': ['13 03']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_chacha20_poly1305_sha256'
def test_hex_c0_2b(self):
lf = pl.LazyFrame({'0cph': ['c0 2b']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
def test_hex_00_35(self):
lf = pl.LazyFrame({'0cph': ['00 35']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_rsa_aes_128_cbc_sha'
def test_hex_00_2f(self):
"""00 2f also maps to tls_rsa_aes_128_cbc_sha (same canonical)."""
lf = pl.LazyFrame({'0cph': ['00 2f']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_rsa_aes_128_cbc_sha'
def test_unknown_hex(self):
lf = pl.LazyFrame({'0cph': ['ab cd']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'unknown_cipher_0xab cd'
def test_null_remains_null(self):
lf = pl.LazyFrame({'0cph': [None]})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] is None
def test_case_insensitive_hex(self):
lf = pl.LazyFrame({'0cph': ['C0 2B']})
result = normalize_lf(lf, {'0cph': None}).collect()
assert result['0cph_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
# ═════════════════════════════════════════════════════════════════════════
# Cipher Suite — display name
# ═════════════════════════════════════════════════════════════════════════
class TestNormCipherDisplay:
"""_norm_cipher_display — IANA/display name → canonical name."""
def test_iana_full_name(self):
lf = pl.LazyFrame({'cipher-suite': ['TLS_AES_128_GCM_SHA256']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
def test_iana_full_name_lowercase(self):
lf = pl.LazyFrame({'cipher-suite': ['tls_aes_256_gcm_sha384']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_aes_256_gcm_sha384'
def test_display_aes128gcm(self):
lf = pl.LazyFrame({'cipher-suite': ['AES 128 GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
def test_display_aes256gcm(self):
lf = pl.LazyFrame({'cipher-suite': ['AES 256 GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_aes_256_gcm_sha384'
def test_display_chacha20(self):
lf = pl.LazyFrame({'cipher-suite': ['CHACHA20-POLY1305']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_chacha20_poly1305_sha256'
def test_display_ecdhe_ecdsa_aes128gcm(self):
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-ECDSA-AES128-GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
def test_display_ecdhe_rsa_aes256gcm(self):
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-RSA-AES256-GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_rsa_aes_256_gcm_sha384'
def test_display_rsa_aes128gcm(self):
lf = pl.LazyFrame({'cipher-suite': ['RSA AES 128 GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_rsa_aes_128_gcm_sha256'
def test_display_hyphen_variants(self):
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-RSA-AES128-GCM']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_rsa_aes_128_gcm_sha256'
def test_unknown_display(self):
lf = pl.LazyFrame({'cipher-suite': ['CUSTOM_CIPHER_999']})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] == 'customcipher999_unknown'
def test_null_remains_null(self):
lf = pl.LazyFrame({'cipher-suite': [None]})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
assert result['cipher_suite_norm'][0] is None
def test_mixed_display_values(self):
lf = pl.LazyFrame({
'cipher-suite': [
'TLS_AES_128_GCM_SHA256',
'AES 256 GCM',
'ECDHE-RSA-AES128-GCM',
'bogus',
None,
],
})
result = normalize_lf(lf, {'cipher-suite': None}).collect()
expected = [
'tls_aes_128_gcm_sha256',
'tls_aes_256_gcm_sha384',
'tls_ecdhe_rsa_aes_128_gcm_sha256',
'bogus_unknown',
None,
]
assert result['cipher_suite_norm'].to_list() == expected
# ═════════════════════════════════════════════════════════════════════════
# Named Curve — hex
# ═════════════════════════════════════════════════════════════════════════
class TestNormCurve:
"""_norm_curve — hex curve → canonical name."""
def test_hex_00_1d_x25519(self):
lf = pl.LazyFrame({'0crv': ['00 1d']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'x25519'
def test_hex_00_17_secp256r1(self):
lf = pl.LazyFrame({'0crv': ['00 17']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'secp256r1'
def test_hex_00_18_secp384r1(self):
lf = pl.LazyFrame({'0crv': ['00 18']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'secp384r1'
def test_hex_00_19_secp521r1(self):
lf = pl.LazyFrame({'0crv': ['00 19']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'secp521r1'
def test_hex_00_1e_x448(self):
lf = pl.LazyFrame({'0crv': ['00 1e']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'x448'
def test_unknown_hex_curve(self):
lf = pl.LazyFrame({'0crv': ['ff ff']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'unknown_curve_0xff ff'
def test_null_remains_null(self):
lf = pl.LazyFrame({'0crv': [None]})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] is None
def test_case_insensitive_hex_curve(self):
lf = pl.LazyFrame({'0crv': ['00 1D']})
result = normalize_lf(lf, {'0crv': None}).collect()
assert result['0crv_norm'][0] == 'x25519'
# ═════════════════════════════════════════════════════════════════════════
# Named Curve — display name
# ═════════════════════════════════════════════════════════════════════════
class TestNormCurveDisplay:
"""_norm_curve_display — display name → canonical name."""
def test_x25519(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['X25519']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
def test_curve25519(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['Curve25519']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
def test_secp256r1(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['secp256r1']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
def test_prime256v1(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['prime256v1']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
def test_p256(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P256']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
def test_p_256(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P-256']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
def test_p_256_underscore(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P_256']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
def test_secp384r1(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['secp384r1']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
def test_p384(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P384']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
def test_p_384(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['p-384']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
def test_secp521r1(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['Secp521r1']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp521r1'
def test_p521(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P521']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'secp521r1'
def test_x448(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['x448']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'x448'
def test_curve448(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['Curve448']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'x448'
def test_unknown_curve_display(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['brainpoolP256r1']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] == 'unknown_curve_brainpoolp256r1'
def test_null_remains_null(self):
lf = pl.LazyFrame({'ecdhe-named-curve': [None]})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'][0] is None
def test_case_insensitive_p256(self):
lf = pl.LazyFrame({'ecdhe-named-curve': ['P-256', 'p_256', 'P256']})
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
assert result['ecdhe_named_curve_norm'].to_list() == [
'secp256r1', 'secp256r1', 'secp256r1',
]
# ═════════════════════════════════════════════════════════════════════════
# CNRS / ISRS
# ═════════════════════════════════════════════════════════════════════════
class TestNormCnrs:
"""_norm_cnrs — '+' → 'yes', blank/null → 'no'."""
def test_plus_yes(self):
lf = pl.LazyFrame({'cnrs': ['+']})
result = normalize_lf(lf, {'cnrs': None}).collect()
assert result['cnrs_norm'][0] == 'yes'
def test_blank_no(self):
lf = pl.LazyFrame({'cnrs': ['']})
result = normalize_lf(lf, {'cnrs': None}).collect()
assert result['cnrs_norm'][0] == 'no'
def test_null_no(self):
lf = pl.LazyFrame({'cnrs': [None]})
result = normalize_lf(lf, {'cnrs': None}).collect()
assert result['cnrs_norm'][0] == 'no'
def test_whitespace_plus(self):
lf = pl.LazyFrame({'cnrs': [' + ']})
result = normalize_lf(lf, {'cnrs': None}).collect()
assert result['cnrs_norm'][0] == 'yes'
def test_anything_else_no(self):
lf = pl.LazyFrame({'cnrs': ['-', 'true', 'maybe', '0', '1']})
result = normalize_lf(lf, {'cnrs': None}).collect()
assert result['cnrs_norm'].to_list() == ['no', 'no', 'no', 'no', 'no']
def test_isrs_plus_yes(self):
lf = pl.LazyFrame({'isrs': ['+']})
result = normalize_lf(lf, {'isrs': None}).collect()
assert result['isrs_norm'][0] == 'yes'
def test_isrs_blank_no(self):
lf = pl.LazyFrame({'isrs': ['']})
result = normalize_lf(lf, {'isrs': None}).collect()
assert result['isrs_norm'][0] == 'no'
# ═════════════════════════════════════════════════════════════════════════
# Integration — normalize_lf
# ═════════════════════════════════════════════════════════════════════════
class TestNormalizeLf:
"""Top-level normalize_lf — column routing & no-op cases."""
def test_all_columns_at_once(self):
lf = pl.LazyFrame({
'0ver': ['03 04'],
'0cph': ['13 01'],
'cipher-suite': ['TLS_AES_128_GCM_SHA256'],
'0crv': ['00 1d'],
'ecdhe-named-curve': ['X25519'],
'cnrs': ['+'],
'isrs': [''],
})
type_map = {
'0ver': None,
'0cph': None,
'cipher-suite': None,
'0crv': None,
'ecdhe-named-curve': None,
'cnrs': None,
'isrs': None,
}
result = normalize_lf(lf, type_map).collect()
assert result['0ver_norm'][0] == 'tls_1_3'
assert result['0cph_norm'][0] == 'tls_aes_128_gcm_sha256'
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
assert result['0crv_norm'][0] == 'x25519'
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
assert result['cnrs_norm'][0] == 'yes'
assert result['isrs_norm'][0] == 'no'
def test_no_matching_columns(self):
"""Empty type_map → no _norm columns added."""
lf = pl.LazyFrame({'0ver': ['03 04']})
result = normalize_lf(lf, {}).collect()
assert '0ver_norm' not in result.columns
assert result['0ver'][0] == '03 04'
def test_subset_of_columns(self):
"""Only columns present in type_map get normalised."""
lf = pl.LazyFrame({
'0ver': ['03 04'],
'0cph': ['13 01'],
'cnrs': ['+'],
})
result = normalize_lf(lf, {'0ver': None, 'cnrs': None}).collect()
assert '0ver_norm' in result.columns
assert 'cnrs_norm' in result.columns
assert '0cph_norm' not in result.columns
def test_preserves_original_columns(self):
"""Original columns are kept unchanged."""
lf = pl.LazyFrame({
'0ver': ['03 04'],
'src_ip': ['10.0.0.1'],
})
result = normalize_lf(lf, {'0ver': None}).collect()
assert '0ver' in result.columns
assert 'src_ip' in result.columns
assert result['src_ip'][0] == '10.0.0.1'
def test_duplicate_normalized_column_exists(self):
"""If a _norm column already exists, Polars appends suffixed."""
lf = pl.LazyFrame({
'0ver': ['03 04'],
'0ver_norm': ['old'],
})
result = normalize_lf(lf, {'0ver': None}).collect()
# Polars will create 0ver_norm with new value
assert result['0ver_norm'][0] == 'tls_1_3'
def test_lazyness_preserved(self):
"""normalize_lf returns a LazyFrame, not materialised."""
lf = pl.LazyFrame({'0ver': ['03 04']})
result = normalize_lf(lf, {'0ver': None})
assert isinstance(result, pl.LazyFrame)
tianxuan/__init__.py
View File
tianxuan/llm_orchestrator.py
+85
View File
@@ -0,0 +1,85 @@
"""LLM orchestrator optimized for 32B model via MCP tool calling.
Short prompts, aggressive truncation, step-by-step guidance."""
import json, logging, time, urllib.request, urllib.error, asyncio
from typing import Optional
from analysis.tool_registry import handle_call
logger = logging.getLogger('tianxuan.llm')
TOOLS = [
{"type": "function", "function": {"name": "profile_data",
"description": "Examine dataset columns and statistics",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}}},
{"type": "function", "function": {"name": "build_entity_profiles",
"description": "Group flows by entity for user behavior analysis",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "auto_detect": {"type": "boolean"}}, "required": ["dataset_id"]}}},
{"type": "function", "function": {"name": "run_clustering",
"description": "Cluster entities into behavior groups",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_columns": {"type": "array", "items": {"type": "string"}}, "algorithm": {"type": "string", "enum": ["hdbscan", "kmeans"]}}, "required": ["dataset_id", "cluster_columns"]}}},
{"type": "function", "function": {"name": "extract_features",
"description": "Identify distinguishing features per cluster",
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_result_id": {"type": "string"}, "method": {"type": "string", "enum": ["zscore", "anova"]}}, "required": ["dataset_id", "cluster_result_id"]}}},
]
class LLMConfig:
def __init__(self, base_url="", api_key="", model="deepseek-v4-flash", max_tokens=2048, temperature=0.1):
self.base_url = base_url.rstrip('/')
self.api_key = api_key
self.model = model
self.max_tokens = max_tokens
self.temperature = temperature
def call_llm(messages, config):
payload = json.dumps({"model": config.model, "messages": messages, "tools": TOOLS,
"tool_choice": "auto", "max_tokens": config.max_tokens, "temperature": config.temperature}).encode()
headers = {"Content-Type": "application/json"}
if config.api_key: headers["Authorization"] = f"Bearer {config.api_key}"
req = urllib.request.Request(f"{config.base_url}/chat/completions", data=payload, headers=headers, method='POST')
logger.info(f'[LLM] model={config.model}')
start = time.monotonic()
with urllib.request.urlopen(req, timeout=90) as resp:
d = json.loads(resp.read().decode())
logger.info(f'[LLM] latency_ms={int((time.monotonic()-start)*1000)}')
return d
def run_llm_pipeline(dataset_id, entity_col="", config=None, callback=None):
if config is None:
from config import get_config; cfg = get_config()
config = LLMConfig(cfg.llm.base_url, cfg.llm.api_key, cfg.llm.model)
msgs = [{"role": "system", "content": "Analyze dataset step by step using the available tools."},
{"role": "user", "content": f"Profile dataset {dataset_id}, then build entities, cluster, and extract features."}]
guides = ["", "Good. Now build entity profiles.", "Good. Now run clustering.", "Good. Now extract features.", "Good. Summarize results."]
gi = 0
for step in range(15):
if step > 0 and gi < len(guides) and guides[gi]:
msgs.append({"role": "user", "content": guides[gi]})
gi += 1
if callback:
callback(step, "thinking", None)
resp = call_llm(msgs, config)
if not resp:
if callback:
callback(step, "error", "LLM call failed")
return {"error": "LLM call failed"}
msg = resp["choices"][0]["message"]
if "tool_calls" not in msg or not msg["tool_calls"]:
if callback:
callback(step, "complete", msg.get("content", ""))
return {"result": msg.get("content","") or "Complete", "steps": step+1}
msgs.append(msg)
for tc in msg["tool_calls"]:
f = tc.get("function",{}); name = f.get("name","")
try: args = json.loads(f.get("arguments","{}"))
except: args = {}
logger.info(f'[TOOL] {name}')
if callback:
callback(step, name, None)
try: result = asyncio.run(handle_call(name, args))
except Exception as e: result = {"error": str(e)}
if callback:
callback(step, name, result)
msgs.append({"role": "tool", "tool_call_id": tc.get("id",""),
"content": json.dumps(result, default=str, ensure_ascii=False)[:1200]})
if callback:
callback(15, "max_steps", None)
return {"error": "Max steps", "steps": 15}

Some files were not shown because too many files have changed in this diff Show More