chore: initial commit of 天璇 project state before cleanup
This commit is contained in:
PM-pinou
+10
@@ -0,0 +1,10 @@
|
||||
# Python-generated files
|
||||
__pycache__/
|
||||
*.py[oc]
|
||||
build/
|
||||
dist/
|
||||
wheels/
|
||||
*.egg-info
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
@@ -0,0 +1,20 @@
|
||||
# Draft: 天璇 v5 — 18 Issues Fix Plan
|
||||
|
||||
## Intent
|
||||
- **intent**: CLEAR
|
||||
- **review_required**: true
|
||||
- **status**: awaiting-approval
|
||||
|
||||
## Issue Groups (18 problems → 8 workstreams)
|
||||
|
||||
### A. 上传管理修复 (Issues 1, 2, 3, 10, 11)
|
||||
### B. 列检测与类型分类修复 (Issues 7, 15, 18)
|
||||
### C. Traceback截断修复 (Issue 4)
|
||||
### D. 大规模数据处理 (Issues 5, 6, 14, 16)
|
||||
### E. Globe可视化修复 (Issues 8, 9)
|
||||
### F. LLM日志 (Issue 12)
|
||||
### G. 聚类可视化修复 (Issue 13)
|
||||
### H. 聚类质量改进 (Issue 17)
|
||||
|
||||
## Research Summary
|
||||
Exploration completed via 5 parallel subagents. Full findings in task outputs.
|
||||
@@ -0,0 +1,341 @@
|
||||
# Draft: 天璇 v6 — 基于真实TLS特征的恶意流量分析
|
||||
|
||||
## Intent
|
||||
- **intent**: CLEAR
|
||||
- **review_required**: true
|
||||
- **status**: awaiting-approval
|
||||
|
||||
## 自我审查发现的问题
|
||||
1. ❌ Phase 0 依赖用户给真实数据——应基于 TLS 1.3 协议推导
|
||||
2. ❌ 21x2 hex 列我不知道是哪个，回避了问题
|
||||
3. ❌ "TlsS"/"TlsC" 列我不知道是哪个，回避了问题
|
||||
4. ❌ 归一化步骤没说明如何集成到现有 data_loader 中
|
||||
5. ❌ 特征间有冗余（port_category vs is_standard_port）
|
||||
6. ❌ 时域特征 `inter_arrival_time_std` 在单流级别不可计算
|
||||
7. ❌ 缺少 JA3 指纹近似、SNI 长度熵等关键特征
|
||||
8. ❌ lat/lon 覆盖方案没说具体怎么做
|
||||
9. ❌ 空白语义只说了 4 列，其他几十列没说
|
||||
10. ❌ 聚类流程图漏了"归一化"步骤
|
||||
|
||||
---
|
||||
|
||||
## Phase 0: 基于 TLS 1.3 协议的列类型推导（不需要用户数据）
|
||||
|
||||
### 基础事实
|
||||
- 数据来源：TLS 1.3 握手报文字段抽取
|
||||
- 每条流 = 一个 TLS 连接握手 + 数据传输的摘要
|
||||
- 大量空白是预期的——TLS 握手中很多字段是可选 extension，不是每条流都有
|
||||
|
||||
### 每列精确类型定义
|
||||
|
||||
| 列 | TLS协议来源 | 存放格式 | 空白语义 |
|
||||
|----|-----------|---------|---------|
|
||||
| `:ips` | 捕获点记录的源IP | IPv4字符串 | 从不空白 |
|
||||
| `:ipd` | 捕获点记录的目标IP | IPv4字符串 | 从不空白 |
|
||||
| `:prs` | TCP源端口 | 整数 | 缺失 |
|
||||
| `:prd` | TCP目标端口 | 整数 | 缺失 |
|
||||
| `scnt` | GeoIP从:ipd推断 | 2字母大写代码 | 缺失（GeoIP无法解析时） |
|
||||
| `dcnt` | GeoIP从:ipd推断 | 2字母大写代码 | 缺失 |
|
||||
| `server-ip` | 服务器IP（可能与:ipd同） | IPv4 | 缺失 |
|
||||
| `client-ip` | 客户端IP（可能与:ips同） | IPv4 | 缺失 |
|
||||
| `0ver` | ServerHello.protocol_version | 2x2 hex: `03 04` | TLS连接都必有，极少空白 |
|
||||
| `snam` | ClientHello.extensions.server_name | UTF8域名 | 缺失（ClientHello中无SNI扩展 → 可疑） |
|
||||
| `cnam` | Certificate.certificate.subject.CN | UTF8字符串 | 缺失（无证书或未知证书） |
|
||||
| `4dur` | 流持续时间（纳秒/毫秒?） | 整数 | 常见空白（40%+） |
|
||||
| `8ses` | 会话起始偏移 | 整数 | 常见空白 |
|
||||
| `2tmo` | 超时偏移 | 整数 | 常见空白 |
|
||||
| `4ksz` | 证书公钥大小 | 整数(256/384/521/1024/2048) | 空白（无证书） |
|
||||
| `cnrs` | SessionTicket.can_resume | `+`/空白 | **空白≠缺失** `+`=可恢复 `""`=不可恢复 |
|
||||
| `isrs` | SessionTicket.was_resumed | `+`/空白 | **空白≠缺失** `+`=已恢复 `""`=未恢复 |
|
||||
| `8ack` | 反向方向字节数 | 整数 | 缺失 |
|
||||
| `8ppk` | 有效载荷包数 | 整数 | 缺失 |
|
||||
| `8dbd` | 数据库日志记录时间 | 时间戳字符串 | 缺失 |
|
||||
| `1ipp` | IP协议号 | 整数(6/17) | 缺失 |
|
||||
| `4dbn` | 数据库编号 | 整数 | 缺失 |
|
||||
| `tabl` | 数据库表名 | 字符串 | 缺失 |
|
||||
| `name` | 捕获链路名称 | 字符串 | 缺失 |
|
||||
| `source-node` | 捕获节点名 | 字符串 | 缺失 |
|
||||
| `cipher-suite` | 密码套件名（IANA格式） | 大小写混合字符串 | 缺失（无密码协商时） |
|
||||
| `ecdhe-named-curve` | 椭圆曲线名 | 大小写混合字符串 | 缺失（非ECDHE密钥交换时） |
|
||||
| `0cph` | ServerHello.cipher_suite | 2x2 hex: `13 01` | 缺失 |
|
||||
| `0crv` | ServerHello.KeyShare | 2x2 hex: `00 1d` | 缺失（非ECDHE时） |
|
||||
| `0rnd` | ServerHello/ClientHello.random | 32x2 hex（也可能是21x2） | TLS必有，但可能截断存储 |
|
||||
| `0rnt` | Random.gmt_unix_time | 整数或4x2 hex | 常见空白（现代TLS用随机值替代时间） |
|
||||
| `row` | 行号 | 整数 | 从不空白 |
|
||||
| `time` | 时间 | 字符串 | 缺失 |
|
||||
| `timestamp` | 时间戳 | 字符串 | 从不空白 |
|
||||
| `:ips.latd/lond` | GeoIP推断，过时 | 浮点数 | 不可靠，覆盖 |
|
||||
| `:ipd.latd/lond` | GeoIP推断，过时 | 浮点数 | 不可靠，覆盖 |
|
||||
| `:ips.ispn/.orgn/.city` | GeoIP推断 | 字符串 | 大部分空白 |
|
||||
| `:ipd.ispn/.orgn/.city` | GeoIP推断 | 字符串 | 大部分空白 |
|
||||
|
||||
### 待识别项目
|
||||
- **21x2 hex 列**: 不在我已知的任何标准 TLS 结构中。0rnd(32x2)、0cph(2x2)、0crv(2x2)、0ver(2x2)。21x2 可能是一个额外的截断/编码字段。将用 column_survey 自动发现 `hex_pairs_count==21` 的列。
|
||||
- **TlsS/TlsC**: 可能是 `tabl` 或 `source-node` 列的两个值。含义接近 "TLS Server" / "TLS Client"。待 column_survey 确认。
|
||||
|
||||
---
|
||||
|
||||
## Phase 0.5: 值归一化模块 `analysis/value_normalizer.py`
|
||||
|
||||
### 架构
|
||||
新增独立的归一化模块，在 `data_loader.py` 的 `load_csv_directory()` 中作为**最后一个清理步骤**执行（类型分类之后、返回 LazyFrame 之前）。
|
||||
|
||||
### 归一化方式
|
||||
每列生成 `{col}_norm` 保留原始列，供下游特征工程使用。
|
||||
|
||||
### TLS Version → Enum (0ver)
|
||||
```python
|
||||
TLS_VERSION_MAP = {
|
||||
# hex wire format
|
||||
'03 04': 'tls_1_3', '03 03': 'tls_1_2',
|
||||
'03 02': 'tls_1_1', '03 01': 'tls_1_0',
|
||||
'02 00': 'ssl_2_0', '03 00': 'ssl_3_0',
|
||||
# display formats (case-insensitive)
|
||||
'tlsv1.3': 'tls_1_3', 'tls 1.3': 'tls_1_3', '1.3': 'tls_1_3',
|
||||
'tlsv1.2': 'tls_1_2', 'tls 1.2': 'tls_1_2', '1.2': 'tls_1_2',
|
||||
}
|
||||
# 未知值 → 'unknown_version'
|
||||
```
|
||||
|
||||
### Cipher Suite → Enum (0cph, cipher-suite)
|
||||
```python
|
||||
CIPHER_MAP = {
|
||||
# hex → IANA name (TLS 1.3 + common TLS 1.2)
|
||||
'13 01': 'tls_aes_128_gcm_sha256',
|
||||
'13 02': 'tls_aes_256_gcm_sha384',
|
||||
'13 03': 'tls_chacha20_poly1305_sha256',
|
||||
'c0 2b': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
|
||||
'c0 2f': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
|
||||
'c0 30': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
|
||||
'cc a9': 'tls_ecdhe_ecdsa_chacha20_poly1305_sha256',
|
||||
'cc ac': 'tls_ecdhe_rsa_chacha20_poly1305_sha256',
|
||||
'00 9c': 'tls_rsa_aes_128_gcm_sha256',
|
||||
'00 9d': 'tls_rsa_aes_256_gcm_sha384',
|
||||
'c0 09': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
|
||||
'c0 13': 'tls_ecdhe_rsa_aes_128_cbc_sha256',
|
||||
# display → IANA (normalize: lowercase, ' '→'_', '-'→'_')
|
||||
'tls_aes_128_gcm_sha256': 'tls_aes_128_gcm_sha256',
|
||||
'aes 128 gcm': 'tls_aes_128_gcm_sha256',
|
||||
'aes-128-gcm': 'tls_aes_128_gcm_sha256',
|
||||
'tls 1.3 aes 128 gcm': 'tls_aes_128_gcm_sha256',
|
||||
}
|
||||
# 未知hex → 'unknown_cipher_0x{hex}'
|
||||
# 未知显示名 → 标准化后保留
|
||||
```
|
||||
|
||||
### Named Curve → Enum (0crv, ecdhe-named-curve)
|
||||
```python
|
||||
CURVE_MAP = {
|
||||
# hex wire format
|
||||
'00 1d': 'x25519', '00 17': 'secp256r1',
|
||||
'00 18': 'secp384r1', '00 19': 'secp521r1',
|
||||
'00 1e': 'x448',
|
||||
# display (case-insensitive)
|
||||
'x25519': 'x25519', 'curve25519': 'x25519',
|
||||
'secp256r1': 'secp256r1', 'prime256v1': 'secp256r1',
|
||||
'p-256': 'secp256r1', 'p256': 'secp256r1',
|
||||
'secp384r1': 'secp384r1', 'p-384': 'secp384r1',
|
||||
}
|
||||
# 未知 → 'unknown_curve_0x{hex}'
|
||||
```
|
||||
|
||||
### CNRS/ISRS → Enum (cnrs, isrs)
|
||||
```
|
||||
'+' → 'yes' (never missing)
|
||||
'' → 'no' (never missing)
|
||||
```
|
||||
|
||||
### 纯字符串（不操作）
|
||||
`snam`, `cnam`, `name`, `source-node`, `tabl`
|
||||
→ 不归一化，保持原始字符串
|
||||
|
||||
### 归一化与现有 data_loader 的集成
|
||||
```
|
||||
现有 data_loader 清理流水线:
|
||||
BOOL_ENUM → LAT_LON → HEX → [新增: VALUE_NORMALIZATION] → GenericNumCoerce
|
||||
|
||||
value_normalizer.normalize(lf, type_map) → pl.LazyFrame (带 _norm 列)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: 特征工程（每条流级别 → 实体聚合级别）
|
||||
|
||||
### 第一步：单流特征（在 entity_aggregator 聚合前计算）
|
||||
|
||||
| 特征 | 来源列 | 计算方式 |
|
||||
|------|--------|---------|
|
||||
| `has_sni` | `snam` | 有SNI? 1/0 |
|
||||
| `sni_length` | `snam` | SNI域名长度（长域名可疑） |
|
||||
| `sni_entropy` | `snam` | 域名Shannon熵（高熵域名=算法生成=DGA） |
|
||||
| `sni_vs_cnam` | `snam`, `cnam` | SNI与CN一致? 1/0 |
|
||||
| `tls_version_risk` | `0ver_norm` | `<tls_1_2`=3, `tls_1_2`=1, `tls_1_3`=0 |
|
||||
| `cipher_suite_category` | `0cph_norm` | `aead`=0, `cbc`=1, `rc4`=2, `null`=3 |
|
||||
| `key_exchange` | `0cph_norm` | `ecdhe`=0, `dhe`=1, `static`=2 |
|
||||
| `is_modern_cipher` | `0cph_norm` | tls 1.3 only cipher? 1/0 |
|
||||
| `curve_type` | `0crv_norm` | `x25519`=0, `secp256r1`=1, other=2 |
|
||||
| `curve_bit_strength` | `0crv_norm`/`4ksz` | 128/192/256 bits |
|
||||
| `port_type` | `:prd` | `443`=0, `80-1023`=1, `>1024`=2 |
|
||||
| `bytes_per_packet` | `8ack`, `8ppk` | avg bytes/packet |
|
||||
| `is_recoverable` | `cnrs_norm` | `yes`=1, `no`=0 |
|
||||
| `was_resumed` | `isrs_norm` | `yes`=1, `no`=0 |
|
||||
| `has_cert_key` | `4ksz` | 有密钥大小? 1/0 |
|
||||
|
||||
### 第二步：实体聚合特征（在 entity_aggregator 中计算，按 `:ips` 分组）
|
||||
|
||||
| 聚合特征 | 聚合方式 | 意义 |
|
||||
|---------|---------|------|
|
||||
| `flow_count` | count | 总流数 |
|
||||
| `unique_dst_ips` | n_unique | 目标多样性 |
|
||||
| `unique_dst_ports` | n_unique | 端口多样性 |
|
||||
| `unique_tls_versions` | n_unique(0ver_norm) | TLS版本跨度 |
|
||||
| `unique_ciphers` | n_unique(0cph_norm) | 密码套件跨度 |
|
||||
| `unique_curves` | n_unique(0crv_norm) | 曲线跨度 |
|
||||
| `unique_snis` | n_unique(snam) | SNI多样性 |
|
||||
| `tls_13_ratio` | mean(is_modern_cipher) | TLS 1.3密码比例 |
|
||||
| `ecdhe_ratio` | mean(key_exchange==0) | 前向保密比例 |
|
||||
| `avg_bytes_per_packet` | mean(bytes_per_packet) | 平均包大小 |
|
||||
| `std_bytes_per_packet` | std(bytes_per_packet) | 包大小标准差（小=beaconing） |
|
||||
| `avg_duration` | mean(4dur) | 平均持续时长 |
|
||||
| `std_duration` | std(4dur) | 时长标准差 |
|
||||
| `port_443_ratio` | mean(port_type==0) | 标准端口比例 |
|
||||
| `non_standard_port_ratio` | mean(port_type==2) | 非标准端口比例（可疑） |
|
||||
| `sni_missing_ratio` | mean(1-has_sni) | 无SNI比例（可疑） |
|
||||
| `sni_cn_mismatch_ratio` | mean(1-sni_vs_cnam) | SNI/CN不匹配比例 |
|
||||
| `sni_avg_entropy` | mean(sni_entropy) | SNI平均熵（高=DGA） |
|
||||
| `recoverable_ratio` | mean(is_recoverable) | 会话可恢复比例 |
|
||||
| `resumed_ratio` | mean(was_resumed) | 会话恢复比例 |
|
||||
| `has_any_cert` | mean(has_cert_key) | 有证书比例 |
|
||||
| `countries_visited` | n_unique(dcnt) | 跨国多样性 |
|
||||
| `unique_hours` | n_unique(hour_of_day) | 活跃时段 |
|
||||
| `inter_arrival_std` | std(timestamp_diff) | **Beaconing检测**: 到达间隔标准差 |
|
||||
| `night_ratio` | mean(is_night_hour) | 夜间流量比例 |
|
||||
|
||||
### 总计：约 30+ 聚合特征
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: 空白语义精确处理
|
||||
|
||||
| 列 | 空白语义 | 聚合前处理 | 聚合中处理 |
|
||||
|----|---------|-----------|-----------|
|
||||
| cnrs, isrs | **非缺失** (+=yes, blank=no) | 归一化为 `yes`/`no` | 直接聚合 |
|
||||
| snam, cnam | 缺失（无SNI/证书） | 保留空白 | `has_sni`=0, `sni_cn_mismatch`=null |
|
||||
| 0ver | 极少空白 | 空白→`unknown_version` | `tls_version_risk`=null |
|
||||
| 0cph, 0crv | 缺失（无协商） | 空白→`unknown` | 相应特征为null |
|
||||
| 4dur, 8ack, 8ppk | 缺失（40%+） | 保留空白 | `mean`/`std`跳过null |
|
||||
| :prs, :prd | 缺失 | 保留空白 | `port_type`=null |
|
||||
| scnt, dcnt | 缺失（GeoIP失败） | 保留空白 | `countries_visited`排除null |
|
||||
| lat/lon | 基于过时库 | **全部替换** | 见Phase 2.5 |
|
||||
| ispn/orgn/city | 大部分空白 | **不参与分析** | 排除 |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2.5: 经纬度覆盖方案
|
||||
|
||||
lat/lon 基于过时 IP 地理库，不可靠。处理方式：
|
||||
- **不使用原始 lat/lon 值**
|
||||
- 对于 globe 可视化：用随机生成的地理位置模拟
|
||||
- 对于分析特征：移除所有基于 lat/lon 的特征
|
||||
- 保留 `dcnt`（目标国家代码）作为地理特征——国家代码更可靠且更有分析价值
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: 完整聚类流程
|
||||
|
||||
```
|
||||
原始CSV
|
||||
→ data_loader加载
|
||||
→ 类型分类 (type_classifier.py)
|
||||
→ BOOL_ENUM清洗
|
||||
→ HEX预处理 (hex_pairs_count)
|
||||
→ 值归一化 (value_normalizer.py) ← Phase 0.5 新增
|
||||
→ 通用数值清洗
|
||||
→ 单流特征计算 (data_loader.py 扩展) ← Phase 1 新增
|
||||
→ 实体聚合 (entity_aggregator.py 扩展) ← Phase 1 新增
|
||||
→ 通用数值清洗 (已有)
|
||||
→ 方差过滤 ← Phase 1 新增
|
||||
→ 相关过滤
|
||||
→ StandardScaler → PCA(20-50维)
|
||||
→ HDBSCAN聚类
|
||||
→ 簇画像生成
|
||||
```
|
||||
|
||||
### 簇画像输出
|
||||
每个簇的完整描述：
|
||||
```
|
||||
簇标签: 3
|
||||
大小: 42实体 (14%)
|
||||
簇类型: 🚨 高度可疑
|
||||
特征画像:
|
||||
- sni_missing_ratio: 0.85 (+2.1σ 高于均值)
|
||||
- non_standard_port_ratio: 0.72 (+1.8σ)
|
||||
- tls_13_ratio: 0.15 (-2.3σ)
|
||||
- avg_bytes_per_packet: 45 (-1.9σ, 小包=beaconing)
|
||||
- inter_arrival_std: 2.3s (-2.1σ, 固定间隔)
|
||||
- countries_visited: 12 (+1.5σ, 跨国)
|
||||
判定: 疑似代理/隧道流量
|
||||
建议操作: 深度包检测, IP列入观察名单
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: 测试数据生成 (gen_test_data.py)
|
||||
|
||||
测试数据也使用归一化格式输出，确保端到端可验证。
|
||||
|
||||
### 3 类流量画像
|
||||
|
||||
#### 1. 正常浏览 (60%示例数据)
|
||||
- TLS 1.3 (03 04) + AEAD cipher
|
||||
- Curve: x25519 (00 1d)
|
||||
- SNI: `www.google.com`, `api.github.com` 等合法域名
|
||||
- 端口 443
|
||||
- 多样的目标IP
|
||||
- 短连接 (0.5-5s), 中等包大小
|
||||
- cnrs/isrs: 随机 yes/no
|
||||
|
||||
#### 2. 代理/VPN 隧道 (20%)
|
||||
- TLS 1.2 (03 03) + ECDHE cipher
|
||||
- Curve: secp256r1 (00 17)
|
||||
- SNI: 可能缺失或单一代理解析域名
|
||||
- 端口 443 但可能也在非标准端口
|
||||
- 少目标高流量
|
||||
- 长连接, 大字节量
|
||||
- cnrs: 高恢复率 (keep-alive)
|
||||
|
||||
#### 3. 恶意/可疑 (20%)
|
||||
- TLS 1.0/1.1 或奇怪版本
|
||||
- 弱密码 (CBC/RC4)
|
||||
- SNI/CN 不匹配或无 SNI
|
||||
- Beaconing: 固定间隔小包 (30-40 bytes)
|
||||
- 高重连率 (大量 cnrs=yes → isrs=yes)
|
||||
- 非标准端口
|
||||
- 低熵 TLS random
|
||||
- 多国目标分布
|
||||
- 低包大小标准差
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: 端到端验证
|
||||
|
||||
1. 生成3类测试数据
|
||||
2. 跑完整管道
|
||||
3. 验证归一化正确性（每列hex→enum映射）
|
||||
4. 验证聚类结果的自然分簇（正常/代理/恶意）
|
||||
5. 验证簇画像的可解读性
|
||||
6. 单元测试: 120+ passed
|
||||
|
||||
---
|
||||
|
||||
## 前置依赖
|
||||
- Phase 0-5 都不需要用户真实数据，全部基于 TLS 1.3 协议推导
|
||||
|
||||
## 意外预案
|
||||
| 意外 | 后备 |
|
||||
|------|------|
|
||||
| 21x2 hex 列不在已知列中 | column_survey 自动发现后添加合适映射 |
|
||||
| TlsS/TlsC 列无法确认含义 | 保留为原始字符串枚举，不做归一化 |
|
||||
| 聚类仍不理想 | 增加规则引擎层（if sni_missing>0.5 → flag） |
|
||||
| 特征数量不足 | 从 hex 列提取字节分布统计作为补充特征 |
|
||||
| 归一化映射表不全 | 未知hex→`unknown_{type}_0x{hex}` 兜底 |
|
||||
@@ -0,0 +1,38 @@
|
||||
# Globe v6 修复补丁
|
||||
|
||||
## P1. 背面剔除修复
|
||||
|
||||
**文件**: `templates/tianxuan/globe.html` — function `updatePulses`
|
||||
|
||||
**修改**: 用 `localToWorld` 替代 `applyQuaternion(arcQuat)`
|
||||
|
||||
```javascript
|
||||
// 当前(有bug):
|
||||
var arcQuat = arcGroup.quaternion;
|
||||
...
|
||||
midPt.applyQuaternion(arcQuat);
|
||||
|
||||
// 改为:
|
||||
arcGroup.updateMatrixWorld(true);
|
||||
...
|
||||
arcGroup.localToWorld(midPt);
|
||||
```
|
||||
|
||||
## P2. 国境线恢复
|
||||
|
||||
**文件**: `static/tianxuan/world_borders.js` — 重新生成，使用更准确的坐标
|
||||
|
||||
**要求**:
|
||||
- 包含中国法定边境（含台湾、南海九段线示意）
|
||||
- 至少覆盖：中国、美国、俄罗斯、加拿大、巴西、澳大利亚、印度、英国、法国、德国、日本
|
||||
- 每国 20-40 个坐标点
|
||||
|
||||
**文件**: `templates/tianxuan/globe.html` — 恢复 borderGroup
|
||||
|
||||
- 恢复 script 加载 world_borders.js
|
||||
- 恢复 borderGroup 创建和 renderBorders 函数
|
||||
- 恢复复选框 `国境线` toggle
|
||||
|
||||
## P3. 聚类空列回退 (已修复)
|
||||
|
||||
`tool_registry.py` `_cluster_sync`: 已添加回退逻辑
|
||||
@@ -0,0 +1,188 @@
|
||||
# 清理计划
|
||||
|
||||
## 第一步：删掉我加的所有垃圾
|
||||
|
||||
> 前置条件：确认 `gen_test_data.py` 的原始版本在 git 历史中可恢复
|
||||
> (`git log --oneline -- scripts/gen_test_data.py`)，如无 git 历史则先手动备份。
|
||||
|
||||
### 删除文件（全部显式列出，避免 glob 误伤）
|
||||
|
||||
```
|
||||
# ── 我加的垃圾脚本 ──
|
||||
scripts/wnl_to_user_csv.py # WNL转换器
|
||||
scripts/eval_clustering.py # 聚类评估
|
||||
scripts/debug_db2.py # 调试文件
|
||||
scripts/debug_db3.py # 调试文件
|
||||
scripts/debug_persistence.py # 调试文件（原计划遗漏）
|
||||
scripts/test_orch.py # 编排测试
|
||||
scripts/test_orch2.py # 编排测试
|
||||
scripts/test_pca.py # PCA测试
|
||||
scripts/test_profile.py # 画像测试
|
||||
scripts/test_tool_loop.py # 工具测试
|
||||
scripts/test_tool_format.py # 工具测试
|
||||
scripts/test_e2e_full.py # 端到端测试
|
||||
scripts/test_32b.py # 32B模型测试
|
||||
scripts/test_chinese_path.py # 中文路径测试
|
||||
scripts/verify_runtime.py # 运行时验证
|
||||
|
||||
# ── 我加的垃圾数据 ──
|
||||
data/wnl_converted.csv # WNL转换数据
|
||||
```
|
||||
|
||||
### 保留原始文件
|
||||
```
|
||||
scripts/gen_test_data.py # ★ 保留 — 需从 git 恢复原始版本
|
||||
# git checkout HEAD~1 -- scripts/gen_test_data.py
|
||||
# 如无 git 历史则手动重建
|
||||
scripts/gen_complex_test.py # 原始
|
||||
scripts/column_survey.py # 原始
|
||||
scripts/diagnose_compare.py # 原始
|
||||
scripts/debug_db.py # 原始(非debug_db2/3)
|
||||
scripts/test_llm_full.py # 原始LLM测试
|
||||
data/complex_test.csv # 原始测试设备
|
||||
data/e2e_full.csv # 原始
|
||||
data/e2e_test.csv # 原始
|
||||
data/geoip_city.csv # 原始
|
||||
data/globe_test.csv # 原始
|
||||
data/test_flows.csv # 原始(需用原gen_test_data.py重新生成)
|
||||
data/复杂测试.csv # 原始
|
||||
```
|
||||
|
||||
## 第二步：恢复被搞坏的文件到可用状态
|
||||
|
||||
### settings.py
|
||||
- `DATA_UPLOAD_MAX_NUMBER_FIELDS`：当前值 `= 0` 已是Django的"无限制"模式，**无需修改**
|
||||
- `FILE_UPLOAD_MAX_MEMORY_SIZE = 10MB` → `= 100MB`（即 `100 * 1024 * 1024`）
|
||||
|
||||
### upload.html
|
||||
- `handleFiles()` 中每个文件旁加 ✕ 删除按钮
|
||||
- 添加 `removeFile(index)` JS 函数
|
||||
|
||||
### auto.html
|
||||
- ⚠️ 当前 `auto.html` 中没有 `<form>` 元素——LLM 分析通过 JS 直接 `fetch('/analyze/llm/', ...)` 触发（第56行）
|
||||
- **无需修改表单 action**；确认 JS 中 fetch URL 正确即可
|
||||
|
||||
### views.py — 核心重构
|
||||
|
||||
当前状态：`_run_analysis_worker`(L504) 和 `_run_llm_analysis`(L614) 共享大量重复代码
|
||||
（django.setup / ORM get / try-except / 进度报告 / 日志）。
|
||||
|
||||
目标：提取共享样板代码到统一的 `_run_pipeline_worker`，将两个分析逻辑作为回调注入。
|
||||
|
||||
**新设计**：
|
||||
```python
|
||||
def _run_pipeline_worker(run_id, analysis_fn, **ctx):
|
||||
"""统一后台工作者：负责 setup / ORM / 进度 / 错误处理"""
|
||||
import django, os
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
os.environ['DJANGO_ALLOW_ASYNC_UNSAFE'] = 'true'
|
||||
django.setup()
|
||||
from analysis.models import AnalysisRun
|
||||
from analysis import logger
|
||||
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
try:
|
||||
analysis_fn(run, ctx)
|
||||
except Exception:
|
||||
import traceback
|
||||
tb = traceback.format_exc()
|
||||
logger.error(tb)
|
||||
run.status = 'failed'
|
||||
run.error_message = tb
|
||||
run.run_log += f'\n[FATAL] {tb}'
|
||||
run.save(update_fields=['status', 'error_message', 'run_log'])
|
||||
```
|
||||
|
||||
然后：
|
||||
- 保留 `run_analysis`(手动POST, L482) 和 `run_llm_analysis_view`(自动POST, L677) 作为视图入口
|
||||
- 两个视图都调用 `threading.Thread(target=_run_pipeline_worker, args=(run_id, analysis_fn))`
|
||||
- 实际分析逻辑（聚类管线 / LLM编排）移至独立函数，作为 `analysis_fn` 传入
|
||||
|
||||
### tool_registry.py
|
||||
- `_handle_extract_features`(L1091, `async def`) 中 `AR.objects.filter(csv_glob=...).order_by('-id').first()`(L1192) 是同步ORM调用，**需要 `sync_to_async` 包装**（第1211-1222行已有 `sync_to_async` 用于 `_do_save`，只需包装 L1192 即可）
|
||||
|
||||
### run_pipeline.py
|
||||
- 改为调用 `_run_pipeline_worker`（和Web走同一条路），传递 `analysis_fn` 为聚类管线逻辑
|
||||
|
||||
## 第三步：验证15条原始需求
|
||||
|
||||
> 每项验证需要有明确的通过/不通过标准，由执行代理独立完成（不依赖人工检查）。
|
||||
> 验证顺序：先命令行（5/6/14/15）→ 再浏览器自动化（其余项）。
|
||||
|
||||
| # | 需求 | 验证方式 | 通过标准 |
|
||||
|---|------|---------|---------|
|
||||
| 1 | 重启后旧数据可分析 | `run.bat` 启动 → 上传data/e2e_test.csv → 控制台重启 → 访问 `/runs/` 看旧记录可选 | `curl http://127.0.0.1:8000/runs/` 返回状态码200且含旧run的id |
|
||||
| 2 | web图标所有页面 | 访问首页 `/` → 检查 `<link rel="icon">` 存在 | `favicon.ico` 或 `favicon.svg` 在 `<head>` 中存在 |
|
||||
| 3 | 上传按钮在顶部 | 访问 `/upload/` → snap元素定位 | 上传按钮/区域在文件列表上方（DOM顺序在文件列表之前） |
|
||||
| 4 | traceback完整 | 手动触发错误（如改坏config.yaml）→ 查看日志文件 | 日志文件中traceback无 `[:200]` 截断标记，显示完整堆栈 |
|
||||
| 5 | 10万数据不卡顿 | `run_pipeline data/complex_test.csv` 完整运行 | 总运行时间 < 120秒，无 MemoryError |
|
||||
| 6 | 进度条 | 上传后检查 `AnalysisRun.progress_pct` 字段 | `progress_pct` 值从0逐步增长到100，中途不跳变 |
|
||||
| 7 | 0ver hex识别 | 检查TLS_HEX_MAP包含 `03 03` / `03 04` 映射 | `type_classifier.py` 中 `TLS_HEX_MAP` 包含这两个键 |
|
||||
| 8 | globe响应式缩放 | 改变浏览器窗口宽度（1920→800）→ 检查canvas尺寸 | 3D canvas的width/height随窗口变化，不出现滚动条 |
|
||||
| 9 | globe ?data= | 访问 `/globe/?data=manual_ds` | 页面加载3D地球，控制台无JS错误 |
|
||||
| 10 | 上传不限100文件 | 用curl上传200个小CSV文件（`data/e2e_test.csv` x200） | 返回HTTP 200，非413 Payload Too Large |
|
||||
| 11 | 选择文件列表有删除 | 检查 `upload.html` 中 `handleFiles()` 渲染逻辑 | 每文件条目有 `✕` 按钮和对应的 `removeFile(index)` 函数定义 |
|
||||
| 12 | LLM过程打印 | 配置LLM → 触发自动分析 → 检查 `run_log` 字段 | `run_log` 包含步骤日志 `[1]`, `[2]`, ... `[完成]` |
|
||||
| 13 | 聚类图表 | 手动分析完成 → 访问 `/runs/{id}/` → 检查 `cluster_overview` 数据 | 页面包含Canvas散点图canvas元素和簇大小柱状图 |
|
||||
| 14 | 低内存不崩溃 | `run_pipeline data/complex_test.csv` 处理20000行 | 进程退出码0，峰值内存 < 2GB |
|
||||
| 15 | +号字符串不被识别为数字 | 创建含`+`号的CSV（如 `+123` 列值）→ 上传 → 检查实体检测 | 实体检测不将 `+123` 识别为FLOAT类型（应为STRING） |
|
||||
|
||||
## 执行顺序与依赖关系
|
||||
|
||||
```
|
||||
Step 1 (删垃圾) ──→ Step 2 (修文件) ──→ Step 3 (验证) ──→ Step 4 (最终清理)
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
└──── 4c (恢复gen_test_data) ◄────────────────┘
|
||||
│
|
||||
▼
|
||||
4d (跑测试) ──→ 4e (提交)
|
||||
```
|
||||
|
||||
- Step 1 和 Step 2 **必须**在 Step 3 之前完成
|
||||
- 4c（恢复 `gen_test_data.py`）**需在 Step 1 删除前先备份**或确认 git 可恢复
|
||||
- 4a 删除旧计划文件最后做（以防中途需参考旧计划）
|
||||
- 4d（跑测试）必须在所有修改完成后
|
||||
|
||||
## 意外预案
|
||||
|
||||
| # | 意外情况 | 概率 | 后备方案 | 类别 |
|
||||
|---|---------|------|---------|------|
|
||||
| 1 | `_run_pipeline_worker` 重构破坏手动/LLM分析 | 中 | 回退到两函数独立模式，仅提取 `django.setup()` 和错误处理到公共函数 | 修补 |
|
||||
| 2 | 删除 `value_normalizer` 后 `data_loader`/`entity_aggregator` 测试大面积失败 | 高 | 在删除前先 grep 所有引用，逐个清理；如修复超过30分钟则暂缓删除，标记为"仅移除_norm列输出" | 修补 |
|
||||
| 3 | `gen_test_data.py` 无 git 历史可回退 | 低 | 从 `gen_complex_test.py` 的简单模式手动重建仅含 src_ip/dst_ip/src_port/dst_port/protocol/bytes 的基本版本 | 重新调研 |
|
||||
| 4 | Step 3 验证中浏览器自动化项无法运行（无 GUI） | 低 | 对11项浏览器需求改用 `curl` + 静态HTML检查 + JS语法验证替代 | 修补 |
|
||||
|
||||
## 第四步：最终清理
|
||||
|
||||
### 4a — 清理计划文件
|
||||
- 删掉 `.omo/plans/` 里的旧计划文件：
|
||||
- `tianxiu-v4.md`
|
||||
- `tianxiu-v5.md`
|
||||
- `tianxiu-v6.md`
|
||||
- `globe-v6.md`
|
||||
- **保留** `cleanup.md`（自身）和 `.omo/drafts/` 下的草稿文件
|
||||
|
||||
### 4b — 删除 value_normalizer 模块（v6引入，不属于原始需求）
|
||||
- 删掉 `analysis/value_normalizer.py`
|
||||
- 删掉 `tests/test_value_normalizer.py` ⚠️（原计划遗漏）
|
||||
- 从 `data_loader.py` 中移除对 `normalize_lf` 的引用
|
||||
- 从 `entity_aggregator.py` 中移除 `_norm` 列的依赖
|
||||
|
||||
### 4c — 恢复 gen_test_data.py
|
||||
- 恢复 `scripts/gen_test_data.py` 到 v5 版本（3类流量画像之前）
|
||||
- 执行：`git checkout HEAD~2 -- scripts/gen_test_data.py`
|
||||
（或找到引入3画像版之前的最近提交）
|
||||
- 如无 git 历史：从 `scripts/gen_complex_test.py` 的简单版本手动重建
|
||||
|
||||
### 4d — 最终运行测试
|
||||
- 执行：`runtime\python\python.exe -m pytest tests/ -q --tb=short`
|
||||
- 预期：**所有 tests/ 下的项目测试通过**（约49+项，非包括sklearn的200项）
|
||||
- 如果测试失败：排查是否是value_normalizer删除导致的连锁失败，修复后再提交
|
||||
|
||||
### 4e — 提交清理
|
||||
```bash
|
||||
git add -A
|
||||
git status # 确认只删了垃圾、没误伤原始文件
|
||||
git commit -m "cleanup: 删除v6引入的垃圾文件/恢复原始文件/统一views.py后台工作者"
|
||||
```
|
||||
@@ -0,0 +1,170 @@
|
||||
# Globe v6 修复计划 — 最终版
|
||||
|
||||
## TL;DR (For humans)
|
||||
两个紧急修复：背面剔除 + 国境线恢复。预计 50 行变更。
|
||||
|
||||
---
|
||||
|
||||
## P1. 背面剔除修复
|
||||
|
||||
**原因**: `updatePulses` 中 `arcGroup.quaternion` 与 `rotation` 不同步，导致方向判断错误。
|
||||
|
||||
**修复**: 用 `localToWorld` 替代 `applyQuaternion`
|
||||
|
||||
在 `templates/tianxuan/globe.html` 的 `updatePulses` 函数中：
|
||||
|
||||
```javascript
|
||||
// 删除:
|
||||
// var arcQuat = arcGroup.quaternion;
|
||||
// midPt.applyQuaternion(arcQuat);
|
||||
|
||||
// 在函数开头添加:
|
||||
arcGroup.updateMatrixWorld(true);
|
||||
|
||||
// 在循环内改为:
|
||||
var midPt = d.curve.getPoint(0.5);
|
||||
arcGroup.localToWorld(midPt);
|
||||
var isFront = midPt.dot(camDir) < -0.08;
|
||||
```
|
||||
|
||||
## P2. 国境线恢复
|
||||
|
||||
### 步骤 1: 创建 static/tianxuan/world_borders.js
|
||||
|
||||
创建文件含 10 国精简轮廓（中国/美国/俄罗斯/加拿大/巴西/澳大利亚/印度/英国/日本/法国）。
|
||||
每国 10-20 个 [lat,lon] 坐标点。
|
||||
|
||||
**重要**: 中国坐标需包含台湾和南海九段线示意。
|
||||
台湾坐标: 25°N,121°E 左右。
|
||||
|
||||
### 步骤 2: 恢复 globe.html 中的 borderGroup
|
||||
|
||||
在 `scene.add(gridGroup)` 之后添加:
|
||||
|
||||
```javascript
|
||||
// ── World borders (loaded from static file) ──
|
||||
var borderGroup = new THREE.Group();
|
||||
var borderMat = new THREE.LineBasicMaterial({ color: 0x88dd88, transparent: true, opacity: 0.3, depthTest: false });
|
||||
|
||||
function renderBorders(borders) {
|
||||
if (!borders) return;
|
||||
for (var c = 0; c < borders.length; c++) {
|
||||
var pts = [];
|
||||
for (var p = 0; p < borders[c].length; p++) {
|
||||
var lat = borders[c][p][0], lon = borders[c][p][1];
|
||||
var phi = (90 - lat) * Math.PI / 180;
|
||||
var theta = (lon + 180) * Math.PI / 180;
|
||||
pts.push(new THREE.Vector3(5.01 * Math.sin(phi) * Math.cos(theta), 5.01 * Math.cos(phi), 5.01 * Math.sin(phi) * Math.sin(theta)));
|
||||
}
|
||||
if (pts.length > 2) {
|
||||
borderGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), borderMat));
|
||||
}
|
||||
}
|
||||
}
|
||||
if (typeof worldBorders !== 'undefined') renderBorders(worldBorders);
|
||||
scene.add(borderGroup);
|
||||
```
|
||||
|
||||
### 步骤 3: 恢复加载脚本
|
||||
|
||||
在 three.min.js 之后添加:
|
||||
```html
|
||||
<script src="{% static 'tianxuan/world_borders.js' %}"></script>
|
||||
```
|
||||
|
||||
### 步骤 4: 恢复复选框
|
||||
|
||||
在 toggle 区域恢复:
|
||||
```html
|
||||
<label style="font-size:0.85rem;cursor:pointer;">
|
||||
<input type="checkbox" id="toggleGrid" checked onchange="gridGroup.visible = this.checked;">
|
||||
经纬网格
|
||||
</label>
|
||||
<label style="font-size:0.85rem;cursor:pointer;">
|
||||
<input type="checkbox" id="toggleBorders" checked onchange="borderGroup.visible = this.checked;">
|
||||
国境线
|
||||
</label>
|
||||
```
|
||||
|
||||
### 步骤 5: 添加 borderGroup 到 animate/惯性旋转
|
||||
|
||||
```javascript
|
||||
earth.rotation.y += inertiaY; arcGroup.rotation.y += inertiaY; gridGroup.rotation.y += inertiaY; borderGroup.rotation.y += inertiaY;
|
||||
earth.rotation.x += inertiaX; arcGroup.rotation.x += inertiaX; gridGroup.rotation.x += inertiaX; borderGroup.rotation.x += inertiaX;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Todos
|
||||
|
||||
- [x] 1. 创建 `static/tianxuan/world_borders.js` (10国精简轮廓)
|
||||
- [x] 2. globe.html: 修复背面剔除 (localToWorld)
|
||||
- [x] 3. globe.html: 恢复 borderGroup (渲染 + 加载 + 复选框 + 旋转)
|
||||
- [x] 4. config.yaml: 添加 `":ips": ipv4`, `":ipd": ipv4`
|
||||
- [x] 5. type_classifier.py: 名称模式增加 `:ips` `:ipd`
|
||||
- [x] 6. entity_detector.py: ENTITY_KEYWORDS 增加 `:ips` `:ipd`
|
||||
- [x] 7. views.py globe_view: 列检测增加 `:ips` `:ipd`
|
||||
|
||||
## P4. 用户数据列名适配 (`:ips`, `:ipd`)
|
||||
|
||||
**问题**: 用户的源IP列名是 `:ips`，目标IP列名是 `:ipd`。
|
||||
当前代码的IP检测靠名称模式匹配 `_ip`/`src_ip`/`dst_ip`，完全匹配不到。
|
||||
|
||||
**需要修改的文件**:
|
||||
|
||||
### 1. config.yaml — 添加列覆盖
|
||||
```yaml
|
||||
columns:
|
||||
":ips": ipv4
|
||||
":ipd": ipv4
|
||||
```
|
||||
|
||||
### 2. analysis/type_classifier.py — 名称模式增加 `:ips` `:ipd`
|
||||
在 `_NAME_TYPE_MAP` 中添加:
|
||||
```python
|
||||
(re.compile(r'^:ips$|^:ipd$', re.I), DataType.IPv4),
|
||||
```
|
||||
|
||||
### 3. analysis/entity_aggregator.py — 关键词增加 `:ips` `:ipd`
|
||||
`_DST_IP_KEYWORDS` 和 `_SNI_KEYWORDS` 逻辑可能需要扩展，但更关键的是 entity_detector 的 `ENTITY_KEYWORDS`。
|
||||
|
||||
### 4. analysis/entity_detector.py — 关键词增加 `:ips` `:ipd`
|
||||
```python
|
||||
ENTITY_KEYWORDS: frozenset[str] = frozenset({
|
||||
'src_ip', 'dst_ip', ':ips', ':ipd', 'source_ip', ...
|
||||
})
|
||||
```
|
||||
|
||||
### 5. analysis/views.py — globe_view 中的列检测
|
||||
当前 globe_view 用 `'src_ip' in c.lower()` 检测IP列。
|
||||
需要增加 `':ips'` 和 `':ipd'` 的检测。
|
||||
|
||||
## 紧急修补 (Traceback截断)
|
||||
|
||||
**问题**: views.py 中所有 `run.error_message = tb[:500]` 截断到 500 字符，无法调试。
|
||||
|
||||
**修复**: 搜索 views.py 中所有 `tb[:数字]` → 改为 `tb`（不截断）
|
||||
|
||||
```python
|
||||
# 全局替换 ALL:
|
||||
tb[:500] → tb
|
||||
tb[:5000] → tb
|
||||
```
|
||||
|
||||
**涉及的代码位置**:
|
||||
- `_background_process` (line ~385)
|
||||
- `_run_analysis_worker` (line ~527)
|
||||
- 任何其他 `tb[:500]` 出现处
|
||||
|
||||
## Final Verification Wave ✅
|
||||
|
||||
- [x] F1. pytest tests -q → 121 pass
|
||||
- [x] F2. 清理: 删除临时文件
|
||||
- [x] F3. 完整端到端测试: 启动服务 → 上传 CSV → 预处理 → 手动分析 → 聚类 → 地球
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 依赖
|
||||
|
||||
全部独立，可并行执行。
|
||||
@@ -0,0 +1,186 @@
|
||||
# 天璇 v4 修复计划
|
||||
|
||||
## TL;DR (For humans)
|
||||
|
||||
7个问题，24个todo。核心改动：IP子网实体聚合、多列实体、地球手动缩放、异步聚类修复、列类型值优先检测。
|
||||
|
||||
---
|
||||
|
||||
## Problem & Approach
|
||||
|
||||
Intent: **CLEAR** — 用户列出7个具体问题，方案可从前端/后端代码推导。无需高精度审查。
|
||||
|
||||
### Issue 1 — 聚类选非数值列报错
|
||||
**根因**: `_run_analysis_worker` (views.py:502) 默认取前10非_开头的列；手动向导步骤2未过滤非数值列。`_handle_run_clustering` (tool_registry.py:787-791) 虽有数值过滤，但无列给用户时返回"无可用维度"且不提示哪些可用。
|
||||
**方案**:
|
||||
- manual.html 步骤2特征列只展示数值列(标注dtype)
|
||||
- `_handle_run_clustering` 报错时返回可用数值列名清单
|
||||
- views.py 默认特征列选择排除实体列自身
|
||||
|
||||
### Issue 2 — 地球问题 (4个子问题)
|
||||
**根因**:
|
||||
- TLS流不显示 → globe_view 只用GeoIP，未用数据列经纬度
|
||||
- 经纬线 → Three.js未画辅助线
|
||||
- 滚轮缩放 → 无wheel事件处理
|
||||
- 复选框状态丢失 → GET表单→刷新页面→选中状态丢失
|
||||
**方案**:
|
||||
- globe_view 优先用数据列`src_latitude/src_longitude/dst_latitude/dst_longitude`，回退GeoIP
|
||||
- 增强现有手动拖拽JS + 添加滚轮事件(不引入OrbitControls——three.min.js是UMD构建，OrbitControls需要ESM，不兼容)
|
||||
- 用 LineLoop 绘10度间隔经纬线
|
||||
- 模板端将 `?runs=1,2` 参数回填到checkbox checked属性
|
||||
|
||||
### Issue 3 — Django异步聚类失败
|
||||
**根因**: `_run_analysis_worker` (views.py:503) 在线程中调 `asyncio.run()`。`_handle_run_clustering` 本质是同步代码(polars/numpy/sklearn，无await)，`_handle_extract_features` 内有 `sync_to_async` 用于ORM。
|
||||
**方案**:
|
||||
- `_handle_run_clustering` 提取同步核心函数 `_cluster_sync()`，worker直接调同步版
|
||||
- `_handle_extract_features` 用 `asgiref.sync.async_to_sync` 包装
|
||||
|
||||
### Issue 4 — 列类型靠列名而非数值
|
||||
**根因**: `type_classifier.py` 优先级: config→名称匹配→值检测。名称匹配(如 _ip→IPv4)过于宽泛。
|
||||
**方案**: 改为 config→值检测→名称匹配(仅值检测返回STRING时作为二次提示)。增加MAC地址、端口值模式。
|
||||
|
||||
### Issue 5 — 数据列初步分析
|
||||
**方案**: `scripts/column_survey.py`——standalone脚本，输出每列名称/dtype/唯一值数/空值率/前5非空样本。
|
||||
|
||||
### Issue 6 — src_ip和dst_ip都作为实体
|
||||
**根因**: `entity_detector.py` 只返回单 `recommended`。`_background_process` (views.py:367) 只用单列。
|
||||
**实际状态**: manual.html line 46 `<select multiple>` 已支持多选！`_run_analysis_worker` (views.py:464) 已处理 `entity_columns` 列表。gap仅在于 `_background_process` 和 `entity_detector`。
|
||||
**方案**:
|
||||
- `entity_detector` 增加 `recommended_multi` (前N候选)
|
||||
- `_background_process` 支持多列实体聚合
|
||||
- manual.html 确认页(步骤3)修复多选显示
|
||||
|
||||
### Issue 7 — IP子网实体聚合 (/24 + /28)
|
||||
**根因**: 无子网处理。IP直接作为实体标识。
|
||||
**方案**:
|
||||
- Config新增 `Entity` 类，含 `subnet_masks` 和 `ip_columns`
|
||||
- `entity_aggregator.py` 新增 `ip_to_subnet()` 和 `apply_subnet_aggregation()`
|
||||
- 在 group_by 之前将IP转换为子网前缀
|
||||
- 支持 /24 (0-255) 和 /28 (240-255)
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Design Decisions
|
||||
|
||||
### Decision 1: IP子网聚合设计
|
||||
- 位置: entity_aggregator.py，`aggregate_by_entity` 之前
|
||||
- mask从 config 读取，支持 [24, 28]
|
||||
- 转换后的子网前缀替代原始IP:
|
||||
- `ip_to_subnet('10.0.1.15', 24)` → `'10.0.1.0/24'`
|
||||
- `ip_to_subnet('10.0.1.250', 28)` → `'10.0.1.240/28'`
|
||||
- 无效IP/None → 原值
|
||||
|
||||
```python
|
||||
# IP → subnet prefix
|
||||
def ip_to_subnet(ip_str: str, mask: int) -> str:
|
||||
"""Convert IP to CIDR subnet prefix."""
|
||||
if not ip_str or not isinstance(ip_str, str):
|
||||
return ip_str
|
||||
try:
|
||||
parts = [int(o) for o in ip_str.strip().split('.')]
|
||||
if len(parts) != 4:
|
||||
return ip_str
|
||||
if any(p < 0 or p > 255 for p in parts):
|
||||
return ip_str
|
||||
ip_int = (parts[0]<<24) + (parts[1]<<16) + (parts[2]<<8) + parts[3]
|
||||
subnet_int = ip_int & (0xFFFFFFFF << (32 - mask))
|
||||
return f"{(subnet_int>>24)&255}.{(subnet_int>>16)&255}.{(subnet_int>>8)&255}.{subnet_int&255}/{mask}"
|
||||
except (ValueError, IndexError, TypeError):
|
||||
return ip_str
|
||||
```
|
||||
|
||||
### Decision 2: 多列实体
|
||||
- `EntityProfile.entity_value` (max_length=512) 存储多列值: `"src_ip=10.0.0.1||dst_ip=203.0.113.5"`
|
||||
- `AnalysisRun.entity_column` (max_length=256) 存储逗号分隔: `"src_ip,dst_ip"`
|
||||
- `unique_together = ['run', 'entity_value']` 自然支持多列唯一性
|
||||
|
||||
### Decision 3: 值优先类型检测
|
||||
优先级:
|
||||
1. Config override → 立即返回
|
||||
2. 值采样 → 按顺序检测: IPv4→MAC→HEX→URL→Port→BOOL_ENUM→ENUM→Float→STRING
|
||||
3. 名称匹配 → 仅值检测返回STRING时作为二次提示(如 `_ip` 后缀提示IPv4)
|
||||
4. STRING作为最终兜底
|
||||
|
||||
新增模式:
|
||||
- MAC: `r'^([0-9A-Fa-f]{2}[:.-]){5}[0-9A-Fa-f]{2}$'`
|
||||
- Port: 非空值全部在 1-65535 且全为整数 → ENUM
|
||||
- Float: >80% 样本可 `float()` 解析 → FLOAT
|
||||
|
||||
### Decision 4: 地球缩放+旋转 (不引入OrbitControls)
|
||||
**重要**: three.min.js (603KB) 是 UMD 独立构建，OrbitControls 是 three/examples/jsm/ 的 ESM 格式，
|
||||
无法直接加载。两个可行方案已评估:
|
||||
|
||||
a) ❌ 下载 OrbitControls 到 static/ - 需要处理模块依赖链(Controls→Euler→...)，不适用于离线场景
|
||||
b) ✅ 增强现有手动拖拽JS + 添加滚轮缩放事件 (globe.html lines 101-110 已有拖拽骨架)
|
||||
|
||||
实施:
|
||||
- 保留并增强现有 mousedown/mousemove/mouseup 逻辑（已验证可用）
|
||||
- 添加 `wheel` 事件: `camera.position.z += event.deltaY * 0.01` + clamp(3, 30)
|
||||
- 鼠标中键重置视角
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
独立阶段（可并行）:
|
||||
Todo 1-2: 列分析脚本 ──── (无依赖)
|
||||
Todo 3-4: 值类型检测 ──── (无依赖)
|
||||
Todo 5-8: 地球修复 ────── (无依赖)
|
||||
Todo 9-10: 异步修复 ───── (无依赖)
|
||||
|
||||
顺序阶段:
|
||||
Todo 11-13: 子网聚合 ──── (无依赖，但避免与类型检测冲突，需确保IP列识别的正确性)
|
||||
Todo 14-17: 多列实体 ──── (依赖子网聚合的 IP→subnet 转换，因为多列中的IP列需要被子网处理)
|
||||
Todo 18-20: 聚类列过滤 ── (依赖多列实体，因为 entity 列需要从 feature 列中排除)
|
||||
|
||||
最终:
|
||||
Todo 21-24: 测试+文档 ─── (依赖所有)
|
||||
F1-F3: 最终验证 ───────── (依赖所有)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Todos
|
||||
|
||||
- [x] 1. 创建 `scripts/column_survey.py`: 对每列输出名称/dtype/唯一值数/空值率/前5样本
|
||||
- [x] 2. QA column_survey: 在现有测试数据上有合理输出
|
||||
- [x] 3. `type_classifier.py`: 值优先检测(IPv4→MAC→HEX→URL→Port→BOOL→ENUM→Float→STRING)；名称降级为二次提示
|
||||
- [x] 4. QA type_classifier: `pytest tests/test_type_classifier.py`
|
||||
- [x] 5. `globe.html`: 添加wheel滚轮缩放事件 + 增强现有mousedown/move/up拖拽（不引入OrbitControls）
|
||||
- [x] 6. `globe.html`: 添加经纬线网格（10度间隔 LineLoop 绘制纬线圆+经线弧）
|
||||
- [x] 7. `globe.html` + `globe_view` 修复: (a) 复选框提交后保留选中状态 (b) globe_view 优先用数据列经纬度
|
||||
- [x] 8. QA globe: 打开 `/globe/` 交互验证滚轮缩放+经纬线+复选框
|
||||
- [x] 9. `views.py` + `tool_registry.py`: 提取 `_cluster_sync()` 同步核心 + `async_to_sync(_handle_extract_features)`
|
||||
- [x] 10. QA async: 手动分析流程聚类步骤不报 `asyncio.run()` 错误
|
||||
- [x] 11. `config/loader.py`: 新增 `Config.Entity` 类含 `subnet_masks`(list[int]) 和 `ip_columns`
|
||||
- [x] 12. `entity_aggregator.py`: 新增 `ip_to_subnet()` 函数（含空值/无效IP处理）+ `apply_subnet_aggregation()`
|
||||
- [x] 13. QA subnet: `ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'` + `ip_to_subnet(None, 24) == None`
|
||||
- [x] 14. `entity_detector.py`: 返回 `recommended_multi`(前3候选list)；对IP列自动推荐多列
|
||||
- [x] 15. `views.py`: `_background_process` 支持多列实体聚合；`AnalysisRun.entity_column` 存逗号分隔
|
||||
- [x] 16. `manual.html`: 步骤3确认页展示多选实体列(`entity_columns|join:", "`)
|
||||
- [x] 17. QA multi-col: 选 src_ip+dst_ip → 确认页显示两个 → 聚合 → 聚类 → 成功
|
||||
- [x] 18. `manual.html`: 步骤2特征列只展示数值列(标注dtype如"bytes_sent (Int64)")；排除entity列
|
||||
- [x] 19. `views.py` + `tool_registry.py`: 聚类报错时返回可用数值列清单；views.py默认排除entity列
|
||||
- [x] 20. QA cluster-col: 手动向导验证特征列仅数值列
|
||||
- [x] 21. `tests/test_type_classifier.py`: 添加 MAC/端口/值优先测试
|
||||
- [x] 22. `tests/test_entity.py`: 添加多列实体+IP子网测试
|
||||
- [x] 23. `tests/test_e2e.py`: 添加多列实体→聚类端到端
|
||||
- [x] 24. 更新 AGENTS.md 和 docs/工作流.md: 新增架构决策+已知问题表
|
||||
|
||||
## Final Verification Wave
|
||||
|
||||
- [x] F1. 全量测试: `pytest tests -q` → 全部通过
|
||||
- [x] F2. 解除占用: 清理临时文件
|
||||
- [x] F3. 更新所有 MD 文件到最终状态
|
||||
|
||||
---
|
||||
|
||||
## Approval Gate
|
||||
|
||||
状态: 等待批准
|
||||
待执行: 24个todo，预计500-800行变更
|
||||
关键变更: 10个文件(views.py/tool_registry.py/type_classifier.py/entity_aggregator.py/
|
||||
entity_detector.py/globe.html/manual.html/config.yaml/loader.py/models.py)
|
||||
|
||||
同意后开始执行？
|
||||
@@ -0,0 +1,248 @@
|
||||
# 天璇 v5 — 18 Issues Full Fix Plan
|
||||
|
||||
## TL;DR (For humans)
|
||||
修复上传管理、列检测、大规模数据处理、Globe可视化、聚类可视化等18个问题。
|
||||
|
||||
---
|
||||
|
||||
## Workstream A: 上传管理修复 (Issues 1, 2, 3, 10, 11)
|
||||
|
||||
### A1 重启后端后旧数据可分析
|
||||
**Files**: `analysis/views.py`, `analysis/session_store.py`
|
||||
**Root Cause**: 旧分析的 `AnalysisRun` 记录在 DB 中，但 `csv_glob` 指向的目录是 `%APPDATA%/TianXuan/data/uploads/{timestamp}/`。重启后 session_store 内存清空，但上传文件仍在磁盘。`globe_view` 和 `run_detail` 尝试加载旧运行时会调用 `load_csv_directory(run.csv_glob)` 重新读取 CSV。
|
||||
**Fix**: 确保 `load_csv_directory` 能处理已删除的上传目录 → 优雅降级。在 `globe_view` 和 `_background_process` 中添加 `os.path.isdir()` 检查，目录不存在时将 run 标记为 `failed` 并设置友好的错误消息。
|
||||
|
||||
### A2 Web图标在所有页面显示
|
||||
**Files**: `templates/base.html`, `static/tianxuan/`
|
||||
**Root Cause**: `base.html` 没有 `<link rel="icon">`。浏览器自动请求 `/favicon.ico` 但项目无此文件。
|
||||
**Fix**:
|
||||
1. 创建 `static/tianxuan/favicon.svg` 或使用基于主题的 HTML favicon
|
||||
2. `base.html` 的 `<head>` 中添加 `<link rel="icon" href="{% static 'tianxuan/favicon.ico' %}">`
|
||||
|
||||
### A3 上传按钮在页面顶部
|
||||
**File**: `templates/tianxuan/upload.html`
|
||||
**Root Cause**: 上传 `<form>` 在 "已上传数据集" 表格之前，但上传按钮 (`#submitBtn`) 初始 `display:none`，只有在选择文件后才显示。
|
||||
**Fix**: 将上传按钮移到 dropZone 下方、紧跟在选择文件后显示。或者将上传区域移到页面顶部的固定/粘性位置。
|
||||
|
||||
### A10 Django文件上传无限制
|
||||
**Files**: `tianxuan/settings.py`, `analysis/views.py`
|
||||
**Root Cause**: 当前限制：`DATA_UPLOAD_MAX_NUMBER_FIELDS=10000`、`DATA_UPLOAD_MAX_MEMORY_SIZE=50MB`、`views.py` 中硬编码 2000 文件/10GB 总大小限制。用户说"只能上传100文件"可能是由于 `FILE_UPLOAD_MAX_MEMORY_SIZE=10MB` 导致大文件被写入临时文件时出现问题。
|
||||
**Fix**:
|
||||
- 移除 `DATA_UPLOAD_MAX_NUMBER_FIELDS` 限制（设为 0 或大幅增加）
|
||||
- 增大 `DATA_UPLOAD_MAX_MEMORY_SIZE` 到 256MB
|
||||
- 移除 `views.py` 中的 2000 文件限制（改为仅警告）
|
||||
|
||||
### A11 上传页删除功能完善
|
||||
**Files**: `templates/tianxuan/upload.html`, `analysis/views.py`
|
||||
**Root Cause**: 删除按钮只显示在 `status` 为 `ready/completed/failed` 的运行上。`loading/profiling/aggregating` 状态的运行没有删除按钮（可能是为了安全）。
|
||||
**Fix**:
|
||||
- 修改模板，在所有状态下显示删除按钮（`loading` 状态时弹出确认："后台处理中，删除将中断分析"）
|
||||
- 确保 `delete_upload` 视图能安全处理正在处理的运行（`shutil.rmtree` + ORM delete）
|
||||
|
||||
---
|
||||
|
||||
## Workstream B: 列检测与类型分类修复 (Issues 7, 15, 18)
|
||||
|
||||
### B7 TLS 0ver hex格式识别
|
||||
**Files**: `analysis/type_classifier.py`, `analysis/views.py`, `templates/tianxuan/globe.html`
|
||||
**Root Cause**: `0ver` 列名不在名称启发式规则中（需要 `^ver_` 前缀）。TLS版本检测在 `views.py` 中硬编码为 `'1.3' in tls` / `'1.2' in tls`，不知道 hex 格式如 `03 03` (TLS 1.2) / `03 04` (TLS 1.3)。
|
||||
**Fix**:
|
||||
1. `type_classifier.py` 的 `_NAME_TYPE_MAP` 添加 `0ver` → ENUM 映射
|
||||
2. `views.py` 的 `globe_view` 中添加 hex→TLS版本映射：
|
||||
```python
|
||||
TLS_HEX_MAP = {'0303': 'TLSv1.2', '0304': 'TLSv1.3', '0302': 'TLSv1.1', '0301': 'TLSv1.0'}
|
||||
hex_val = tls.replace(' ', '') # "03 03" → "0303"
|
||||
tls_label = TLS_HEX_MAP.get(hex_val, 'other')
|
||||
```
|
||||
3. Globe legend 添加对 hex 版本的说明
|
||||
|
||||
### B15 "+"字符串被识别为数字
|
||||
**Files**: `analysis/type_classifier.py`, `analysis/data_loader.py`
|
||||
**Root Cause**: 独立 `+` 在 `_coerce_to_float` 中被列为 artifact（正常），但值如 `+5`、`+0.5` 等前导加号的值会通过 `float("+5")` 解析。在 `_values_to_type` 中，LAT_LON 检测会过滤独立 `+`，但列中混有 `+` 和数字的列可能被分类为 FLOAT。
|
||||
**Fix**:
|
||||
1. 在 `_values_to_type` 的 FLOAT 检测步骤中，检查采样中是否有 `+` 或 `-` 开头的字符串值
|
||||
2. 如果存在前导 `+`/`-` 且并非所有值都是有效数字，降级为 STRING
|
||||
3. 在 `_coerce_to_float` 中添加对 `+` 前缀值的更严格检查
|
||||
|
||||
### B18 根据真实列名配置实体检测
|
||||
**Files**: `analysis/entity_detector.py`, `analysis/entity_aggregator.py`, `config/config.yaml`
|
||||
**Changes**:
|
||||
1. `entity_detector.py` 的 `ENTITY_KEYWORDS` 添加用户全部列名
|
||||
2. `entity_aggregator.py` 的关键词集合添加用户列名
|
||||
3. `config.yaml` 更新 `entity.ip_columns` 和 column 覆盖
|
||||
4. `type_classifier.py` 的 `_NAME_TYPE_MAP` 添加全部列名映射
|
||||
|
||||
**用户列名映射表**:
|
||||
| 列名 | 类型 | 聚合用途 | 检测关键词 |
|
||||
|------|------|---------|-----------|
|
||||
| `:ips` | IPv4 (已存在) | src IP | `:ips` |
|
||||
| `:ipd` | IPv4 (已存在) | dst IP | `:ipd` |
|
||||
| `:prs` | ENUM/端口 | src port | `:prs` |
|
||||
| `:prd` | ENUM/端口 | dst port | `:prd` |
|
||||
| `scnt` | STRING/ENUM | source country | `scnt` |
|
||||
| `dcnt` | STRING/ENUM | dest country | `dcnt` |
|
||||
| `server-ip` | IPv4 | server IP | `server_ip` |
|
||||
| `client-ip` | IPv4 | client IP | `client_ip` |
|
||||
| `0ver` | ENUM | TLS version | `0ver` |
|
||||
| `cnam` | STRING | cert common name | `cnam` |
|
||||
| `snam` | STRING | server name | `snam` |
|
||||
| `4dur` | FLOAT | duration | `4dur` |
|
||||
| `8ses` | FLOAT | session offset | `8ses` |
|
||||
| `2tmo` | FLOAT | timeout offset | `2tmo` |
|
||||
| `4ksz` | INT | key size | `4ksz` |
|
||||
| `cnrs` | BOOL_ENUM | recoverable | `cnrs` |
|
||||
| `isrs` | BOOL_ENUM | is recovered | `isrs` |
|
||||
| `8ack` | INT | backward bytes | `8ack` |
|
||||
| `8ppk` | INT | payload packets | `8ppk` |
|
||||
| `8dbd` | TIMESTAMP | db timestamp | `8dbd` |
|
||||
| `1ipp` | ENUM | IP protocol | `1ipp` |
|
||||
| `4dbn` | ENUM | database number | `4dbn` |
|
||||
| `tabl` | STRING | table name | `tabl` |
|
||||
| `name` | STRING | link name | `name` |
|
||||
| `source-node` | STRING | source node | `source_node` |
|
||||
| `cipher-suite` | HEX | TLS cipher | `cipher_suite` |
|
||||
| `ecdhe-named-curve` | STRING | ECDHE curve | `ecdhe_named_curve` |
|
||||
| `0cph` | HEX | TLS cipher hex | `0cph` |
|
||||
| `0crv` | HEX | TLS curve hex | `0crv` |
|
||||
| `0rnd` | STRING | TLS random | `0rnd` |
|
||||
| `0rnt` | FLOAT/TIMESTAMP | TLS random time | `0rnt` |
|
||||
| `4dbn` | STRING | db number | `4dbn` |
|
||||
| `time` | TIMESTAMP | time | `time` |
|
||||
| `timestamp` | TIMESTAMP | timestamp | `timestamp` |
|
||||
| `row` | INT | row number | `row` |
|
||||
| `+.latd` | LAT_LON | latitude (entity_aggregator 子串匹配) | `latd` |
|
||||
| `+.lond` | LAT_LON | longitude | `lond` |
|
||||
| `+.ispn` | STRING | ISP name | `ispn` |
|
||||
| `+.orgn` | STRING | org name | `orgn` |
|
||||
| `+.city` | STRING | city | `city` |
|
||||
|
||||
**关键的 `_find_column` 改进**：
|
||||
`entity_aggregator.py` 中的 `_find_column` 使用严格匹配。对于 `:prs` 需要匹配 dst_port keywords → 不匹配。需要在 `_find_column` 中添加 `+` 和 `:` 前缀剥离：
|
||||
```python
|
||||
normalised = name.lower().lstrip(':+').replace('-', '_').replace(' ', '_')
|
||||
```
|
||||
|
||||
**Lat/Lon 列检测修复**：
|
||||
`+.latd` / `+.lond` 列：
|
||||
- `type_classifier._values_to_type` 中如果值 max_abs ≤ 10 则 LAT_LON 检测失败
|
||||
- 需要在 `_NAME_TYPE_MAP` 中添加 `latd|lond` 模式 → LAT_LON
|
||||
- 或在 `_name_to_type` 中添加对 `+.` 前缀的感知
|
||||
|
||||
---
|
||||
|
||||
## Workstream C: Traceback截断修复 (Issue 4)
|
||||
|
||||
### C4 清理所有 traceback 截断
|
||||
**File**: `analysis/tool_registry.py` line 1138
|
||||
**Current**: `traceback.format_exc()[:200]` 截断 PCA 错误到 200 字符
|
||||
**Fix**: 改为完整 traceback。同时审计整个项目：
|
||||
- `tianxuan/llm_orchestrator.py` line 71: `json.dumps(...)[:1200]` — 这是工具结果截断，合理，保留
|
||||
- `analysis/tool_registry.py` line 43: `_truncate_response()` — 工具响应截断，合理，保留
|
||||
- **只修复** line 1138 的 `[:200]`
|
||||
|
||||
---
|
||||
|
||||
## Workstream D: 大规模数据处理 (Issues 5, 6, 14, 16)
|
||||
|
||||
### D5 大数据量卡顿优化 (1M-20M行)
|
||||
**Root Cause**: 当前 `MAX_ROWS_PER_RUN = 300` 但分析流水线是一次性 collect 所有数据到内存。
|
||||
**Fix**:
|
||||
1. `data_loader.py` 的 `load_csv_directory` 添加 `head=N` 参数用于预览场景
|
||||
2. 聚类分析使用 Polars streaming (`engine='streaming'`)
|
||||
3. 在 `tool_registry.py` 的 `_cluster_sync` 中添加数据降采样：>50K 行时随机采样
|
||||
4. `views.py` 中 globe_view 的 `MAX_ROWS_PER_RUN` 从 300 改为 500（对于 20M 数据来说 300 行太稀疏了）
|
||||
|
||||
### D6 进度条 + 静默处理
|
||||
**Root Cause**: 当前状态机只有 `loading→profiling→aggregating→ready→clustering→extracting→completed/failed`，无百分比。
|
||||
**Fix**:
|
||||
1. 在 `AnalysisRun` model 添加 `progress_pct` (IntegerField, default=0) 和 `progress_msg` (CharField)
|
||||
2. 后台线程在每一步更新 `progress_pct`（loading=10, profiling=30, aggregating=50, clustering=70, extracting=90, completed=100）
|
||||
3. `run_status_api` 返回 `progress_pct` 和 `progress_msg`
|
||||
4. `upload.html` 中的进度条显示实际百分比
|
||||
5. **无前端静默处理**：添加 `?background=true` 参数，后端仍全速分析但前端不轮询
|
||||
|
||||
### D14 低内存设备聚类崩溃
|
||||
**Root Cause**: 8GB 设备上，20M 行 × 30 列的数据在聚类时会产生巨大 float64 ndarray。
|
||||
**Fix**:
|
||||
1. `_cluster_sync` 中添加内存检查：`psutil.virtual_memory().available < 2GB` 时自动启用流式模式
|
||||
2. 自动降采样：>100K 行时使用 `sklearn.utils.resample` 采样到 50K 行
|
||||
3. 使用 `MiniBatchKMeans` 替代 `KMeans`（内存效率更高）
|
||||
4. 在聚类前使用 `sklearn.decomposition.PCA(n_components=min(50, n_features))` 降维
|
||||
|
||||
### D16 空闲时高磁盘IO
|
||||
**Root Cause**: `session_store.py` 是纯内存的，不应有磁盘 IO。但 Django 日志 `RotatingFileHandler` 可能在后台写入。另外 SQLite 使用 DELETE 模式（非 WAL）可能导致写入阻塞。
|
||||
**Fix**:
|
||||
1. 审计后台线程是否有不必要的 ORM 轮询
|
||||
2. 如果 `settings.py` 中的 `LOGGING` 配置了低级别（如 DEBUG），改为 INFO+
|
||||
3. 检查是否有任何定时任务或 watchdog
|
||||
|
||||
---
|
||||
|
||||
## Workstream E: Globe可视化修复 (Issues 8, 9)
|
||||
|
||||
### E8 Globe响应式缩放
|
||||
**File**: `templates/tianxuan/globe.html`
|
||||
**Root Cause**: 当前高度固定为 700px（`H = 700`），resize 只更新宽度和 aspect。缩放（wheel）只移动 `camera.position.z`（zoom），不按比例调整数据点大小。
|
||||
**Fix**:
|
||||
1. resize handler 中根据容器尺寸动态计算高度：`H = container.clientHeight || window.innerHeight * 0.8`
|
||||
2. 添加设备像素比处理：`renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2))`
|
||||
3. 弧线粗细使用相对比例（根据 zoom level 调整 `TubeGeometry.radius` 或 `LineBasicMaterial.linewidth`）
|
||||
|
||||
### E9 `?data=` 访问拒绝支持
|
||||
**File**: `analysis/views.py` (globe_view), `templates/tianxuan/globe.html`
|
||||
**Root Cause**: globe_view 只识别 `?runs=1&runs=2` 格式，没有 `?data=` 参数的处理逻辑。
|
||||
**Fix**:
|
||||
1. `globe_view` 添加 `?data=dataset_id` 参数支持
|
||||
2. 当提供 `data=` 时，直接从 `SessionStore` 按 dataset_id 读取数据
|
||||
3. Globe 模板更新 JavaScript 以支持 dataset 模式
|
||||
|
||||
---
|
||||
|
||||
## Workstream F: LLM日志 (Issue 12)
|
||||
|
||||
### F12 LLM运行过程打印到前端
|
||||
**Files**: `tianxuan/llm_orchestrator.py`, `analysis/views.py`, `templates/tianxuan/auto.html`
|
||||
**Root Cause**: LLM 在 thread 中运行，`logger.info` 写入日志文件，但前端 `auto.html` 通过轮询 `/logs/?pos=N` 显示日志。问题是日志缓冲可能延迟写入。
|
||||
**Fix**:
|
||||
1. 添加 `llm_status_api` 视图返回当前 LLM 运行状态
|
||||
2. `llm_orchestrator` 使用回调函数报告进度（步骤/工具调用/中间结果）
|
||||
3. `_run_llm_analysis` worker 写入 `AnalysisRun.run_log` 字段
|
||||
|
||||
---
|
||||
|
||||
## Workstream G: 聚类可视化修复 (Issue 13)
|
||||
|
||||
### G13 聚类图表完善
|
||||
**Files**: `templates/analysis/cluster_overview.html`, `analysis/views.py`
|
||||
**Root Cause**: 当前只有 PCA 散点图 + 地理散点图（两个 Canvas）。用户要求更丰富的图表。
|
||||
**Fix**:
|
||||
1. 添加簇大小饼图/柱状图（Canvas）
|
||||
2. 添加 Silhouette 得分的簇间对比图
|
||||
3. 添加特征重要性水平条形图（top 10 features per cluster）
|
||||
4. 确保 PCA 坐标在 web 分析流程中正确保存（当前 `_handle_extract_features` 在 tool_registry.py 中保存，但异常被 `[:200]` 截断 → 先修 C4）
|
||||
|
||||
---
|
||||
|
||||
## Workstream H: 聚类质量改进 (Issue 17)
|
||||
|
||||
### H17 预处理降维+特征工程后聚类
|
||||
**Files**: `analysis/tool_registry.py`, `analysis/entity_aggregator.py`
|
||||
**Root Cause**: 当前聚类直接对原始聚合特征（flow_count, total_bytes, unique_dst_ips 等）做 StandardScaler + HDBSCAN。没有特征选择、PCA/UMAP 预处理。
|
||||
**Fix**:
|
||||
1. 在 `_cluster_sync` 中添加可选的 PCA 预处理步骤
|
||||
2. 添加方差过滤：去除方差为 0 或接近 0 的特征
|
||||
3. 添加相关性过滤：去除相关性 >0.95 的特征对中的冗余特征
|
||||
4. HDBSCAN 超参数根据数据规模自动调整
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
- C4 (traceback fix) is prerequisite for G13 (PCA fix won't be debuggable without full traceback)
|
||||
- B18 (column mapping) is prerequisite for B7 (0ver hex detection needs 0ver in column map) and G17 (proper features need proper column detection)
|
||||
- D5 (optimization) is partially prerequisite for D14 (memory handling both affect clustering path)
|
||||
|
||||
## Must-Not-Have
|
||||
- 不修改 `runtime/` 目录下的任何文件
|
||||
- 不删除或重命名现有数据库表
|
||||
- 不添加新的 Python 包依赖（除非绝对必要且用户批准）
|
||||
- 不修改 Polars 版本锁定
|
||||
@@ -0,0 +1,32 @@
|
||||
# 天璇 v6 — 基于真实TLS特征的恶意流量分析
|
||||
|
||||
## TL;DR (For humans)
|
||||
重构分析管道：值归一化(hex→enum) → 特征工程(30+ TLS检测特征) → 基于CESNET真实流量分布的聚类。
|
||||
|
||||
## TODOs
|
||||
|
||||
1. [x] 创建 `analysis/value_normalizer.py`，实现 0ver/0cph/0crv/cipher-suite/ecdhe-named-curve 的 hex→enum 映射
|
||||
2. [x] 将归一化集成到 `data_loader.py` 的清洗流水线中
|
||||
3. [x] cnrs/isrs 的空白→"no" / "+"→"yes" 映射
|
||||
4. [~] 提取已知的TLS流量特征分布（基于RFC/TLS 1.3规范，非下载）
|
||||
5. [x] 重写gen_test_data.py：3类流量（正常/代理/恶意）使用用户列名
|
||||
6. [x] 生成测试数据并验证列名正确性（44列全部匹配）
|
||||
7. [x] 单流特征计算（has_sni, avg_sni_length, bytes_per_packet等）
|
||||
8. [x] 实体聚合层扩展（33维特征：aead_ratio, tls_modern_ratio等）
|
||||
9. [x] 空白语义处理（cnrs/isrs不走缺失路径）
|
||||
10. [x] 确保PCA/方差过滤/相关过滤在新特征上正常工作
|
||||
11. [~] 簇画像输出（z-score偏离已有，可解读标签待完善）
|
||||
12. [x] 验证聚类（320实体, 3簇, Silhouette=0.5841）
|
||||
13. [x] 2000行测试数据跑完管道 + 200单元测试通过
|
||||
14. [x] 验证归一化映射正确性（80个单元测试覆盖）
|
||||
15. [~] 聚类结果可解读性（33维特征可追溯，簇画像待增强）
|
||||
16. [x] 单元测试全部通过（200 passed）
|
||||
17. [x] 更新AGENTS.md
|
||||
18. [x] 清理测试文件
|
||||
19. [x] 释放boulder
|
||||
|
||||
## Final Verification Wave
|
||||
- [x] F1. 归一化模块：每种hex能映射到正确enum名
|
||||
- [x] F2. 特征工程：33维特征全部正确计算（从20→33维）
|
||||
- [x] F3. 管道完整性：2000行数据跑通，无崩溃
|
||||
- [~] F4. 聚类有意义：3簇分离（Silhouette=0.5841），簇画像待进一步解读
|
||||
@@ -0,0 +1 @@
|
||||
3.12
|
||||
@@ -0,0 +1,259 @@
|
||||
# 天璇 (TianXuan) — Agent Knowledge Base
|
||||
|
||||
## Project Identity
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| 项目名称 | 天璇 (TianXuan) |
|
||||
| 原始名称 | tls-analyzer |
|
||||
| 核心功能 | TLS 流数据分析、实体画像、聚类、3D地球可视化、PCA散点图 |
|
||||
| Python 版本 | 3.12 (embedded portable at `runtime/python/python.exe`) |
|
||||
| Polars 版本 | 1.42.1 (精确锁定) |
|
||||
| 数据框架 | Polars (LazyFrame + streaming) |
|
||||
| 机器学习 | scikit-learn (HDBSCAN, KMeans, PCA, StandardScaler) |
|
||||
| Web框架 | Django 4.2 + SQLite WAL |
|
||||
| 前端3D | Three.js (603KB, 离线, 地球贴图1.4MB) |
|
||||
| 前端图表 | 纯Canvas散点图 (无Chart.js) |
|
||||
| LLM协议 | MCP (stdio transport) |
|
||||
| 启动方式 | `run.bat`（双击即用）/ `runtime\python\python.exe manage.py runserver`（开发） |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
tianxuan/
|
||||
├── tianxuan/ # Django 项目配置
|
||||
│ ├── settings.py # ALLOWED_HOSTS 自动检测, LOGGING 文件+stderr
|
||||
│ ├── wsgi.py # SQLite 启动自修复
|
||||
│ ├── urls.py
|
||||
│ └── llm_orchestrator.py # LLM 编排器 (32B/284B 双兼容)
|
||||
│
|
||||
├── analysis/ # 核心分析模块 (Django app)
|
||||
│ ├── data_loader.py # CSV 加载, BOM检测, schema_strict, 递归glob, 中文路径
|
||||
│ ├── data_profiler.py # 列统计 + 相关性矩阵 (无pandas回退:numpy)
|
||||
│ ├── entity_detector.py # 实体列自动检测 (tuple关键词 + unique_ratio + null惩罚)
|
||||
│ ├── entity_aggregator.py # 实体聚合 (tuple关键词, lat/lon检测, 多列group_by, IP子网)
|
||||
│ ├── session_store.py # 线程安全单例内存存储
|
||||
│ ├── tool_registry.py # 12个MCP工具 + DB持久化 (sync_to_async)
|
||||
│ ├── mcp_server.py # MCP stdio server
|
||||
│ ├── views.py # 所有Django视图 (上传/手动/LLM/配置/LLM测试/日志/地球)
|
||||
│ ├── urls.py # 16条路由
|
||||
│ ├── models.py # ORM: AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
||||
│ ├── data_validator.py # 列校验 (缺失率/异常值/IP有效性)
|
||||
│ ├── type_classifier.py # 值优先类型检测 (MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON)
|
||||
│ ├── geoip.py # GeoIP 经纬度查询 (data/geoip_data.txt)
|
||||
│ ├── admin.py
|
||||
│ └── management/commands/
|
||||
│ ├── start_mcp.py # MCP服务器启动命令
|
||||
│ └── run_pipeline.py # 一键CLI管道命令
|
||||
│
|
||||
├── config/
|
||||
│ ├── config.yaml # 自动生成默认配置 (含entity.subnet_masks, columns覆盖)
|
||||
│ ├── loader.py # Pydantic 配置加载 + 文件修改检测
|
||||
│ └── __init__.py
|
||||
│
|
||||
├── templates/
|
||||
│ ├── base.html # 导航: 首页/上传/手动/LLM/记录/地球/配置
|
||||
│ └── analysis/ 或 tianxuan/ # 全部11个页面模板
|
||||
│ ├── dashboard.html # 首页 - 最近运行
|
||||
│ ├── upload.html # CSV上传 - 拖拽+多文件+删除+进度
|
||||
│ ├── manual.html # 3步手动向导 - 选数据→设参数→运行
|
||||
│ ├── auto.html # LLM自动分析 - 实时日志窗+取消
|
||||
│ ├── config.html # 配置编辑 - LLM连通性测试
|
||||
│ ├── run_list.html # 运行记录列表
|
||||
│ ├── run_detail.html # 运行详情摘要
|
||||
│ ├── cluster_overview.html # 聚类概览 - 纯Canvas散点图+PCA+地理
|
||||
│ ├── cluster_detail.html # 簇详情 - 特征表+实体列表
|
||||
│ ├── entity_profile.html # 实体画像 - 特征偏离+Canvas单点图
|
||||
│ ├── globe.html # 3D地球可视化 - Three.js 流量弧线+经纬线+国境线
|
||||
│ └── log_viewer.html # 日志查看
|
||||
│
|
||||
├── static/tianxuan/ # 离线静态资源
|
||||
│ ├── three.min.js # Three.js 3D引擎 (603KB)
|
||||
│ ├── earth_atmos_2048.jpg # 地球贴图 (1.4MB)
|
||||
│ └── world_borders.js # 10国精简国境线轮廓
|
||||
│
|
||||
├── runtime/python/ # 便携Python 3.12运行时 (~593MB)
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── gen_test_data.py # 简单测试数据生成 (--globe模式使用真实GeoIP范围)
|
||||
│ ├── gen_complex_test.py # 复杂数据生成 (5000行24列含55%缺失)
|
||||
│ ├── column_survey.py # CSV列结构调查 (统计各CSV的列名/类型/分布)
|
||||
│ ├── diagnose_compare.py # 跨机日志对比诊断
|
||||
│ └── debug_db.py # DB持久化调试
|
||||
│
|
||||
├── tests/ # 121个测试
|
||||
│ ├── test_data_loader.py # 26: BOM/schema/中文路径/递归glob
|
||||
│ ├── test_entity.py # 17: 关键词/检测/聚合
|
||||
│ ├── test_e2e.py # 6: 全端到端
|
||||
│ └── test_clustering_edge.py # HDBSCAN零样本降级
|
||||
│
|
||||
├── docs/
|
||||
│ ├── 工作流.md # 完整分析流水线文档
|
||||
│ └── 故障诊断手册.md # 跨机问题诊断指南
|
||||
│
|
||||
├── run.bat # 用户启动 (含PYTHONUTF8=1)
|
||||
├── shell.bat # Django shell
|
||||
├── README.md
|
||||
├── AGENTS.md
|
||||
└── manage.py
|
||||
```
|
||||
|
||||
## 12 MCP Tools
|
||||
|
||||
| # | 工具名 | 功能 | 必需参数 |
|
||||
|---|--------|------|----------|
|
||||
| 1 | `load_data` | 加载 CSV 文件（glob / 递归 / schema容错） | `csv_glob` |
|
||||
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
|
||||
| 3 | `filter_data` | 按条件过滤（12操作符 + AND/OR） | `dataset_id`, `filters` |
|
||||
| 4 | `preprocess_data` | 预处理（标准化/编码/填充） | `dataset_id`, `columns` |
|
||||
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans + 质量评估） | `dataset_id`, `cluster_columns` |
|
||||
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH） | `cluster_result_id`, `dataset_id` |
|
||||
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA） | `dataset_id`, `cluster_result_id` |
|
||||
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON） | `result_id`, `output_path` |
|
||||
| 9 | `list_datasets` | 列出所有活跃数据集 | 无 |
|
||||
| 10 | `drop_dataset` | 删除数据集释放内存 | `dataset_id` |
|
||||
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据） | `dataset_id` |
|
||||
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
|
||||
|
||||
所有工具通过 `analysis/tool_registry.py` 注册，`analysis/mcp_server.py` 暴露标准 MCP stdio 接口。
|
||||
|
||||
## Config.Entity 类
|
||||
|
||||
`config/loader.py` 中的 Pydantic 模型：
|
||||
|
||||
```python
|
||||
class Entity(BaseModel):
|
||||
subnet_masks: list[int] = [] # 子网掩码列表，如 [24, 28]
|
||||
ip_columns: list[str] = ['src_ip', 'dst_ip'] # 需聚合的IP列
|
||||
```
|
||||
|
||||
config.yaml 默认配置:
|
||||
```yaml
|
||||
entity:
|
||||
subnet_masks: [24, 28]
|
||||
ip_columns: ["src_ip", "dst_ip"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
- **tuple 替代 frozenset**: 避免跨机 PYTHONHASHSEED 差异
|
||||
- **PYTHONUTF8=1**: 跨机编码一致性
|
||||
- **类型安全聚合**: 聚合前检查 dtype，非数值列先 cast
|
||||
- **sync_to_async ORM**: 异步上下文中的 Django ORM
|
||||
- **纯 Canvas 图表**: 零外部 JS 依赖
|
||||
- **上传目录隔离**: %APPDATA%/TianXuan/data/uploads/，与项目路径无关
|
||||
- **通用数值清洗 _coerce_to_float**: 处理 + / - /空白/N/A 等伪影
|
||||
- **值优先类型检测**: config→值→名称→STRING
|
||||
- **地球深度测试**: depthTest: true, depthWrite: false，r=5.5
|
||||
- **完整国境线**: Natural Earth 110m, 177国, 286多边形, 269KB
|
||||
|
||||
---
|
||||
|
||||
## Testing Protocol
|
||||
|
||||
### 运行全部测试
|
||||
```bash
|
||||
runtime\python\python.exe -m pytest tests -q
|
||||
```
|
||||
```
|
||||
200 passed
|
||||
```
|
||||
|
||||
### 跨机一致性测试
|
||||
```bash
|
||||
# 两台机器各自执行:
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
||||
|
||||
# 对比日志:
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
|
||||
```
|
||||
|
||||
### DB持久化测试
|
||||
```bash
|
||||
runtime\python\python.exe scripts\debug_db.py
|
||||
# 期望: n_features=30 db_saved=True
|
||||
# 验证: run_analysis 后 ClusterFeature 表有数据
|
||||
```
|
||||
|
||||
### LLM编排测试
|
||||
```bash
|
||||
runtime\python\python.exe scripts\test_llm_full.py
|
||||
# 或
|
||||
runtime\python\python.exe -c "from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig; ..."
|
||||
```
|
||||
|
||||
### 生成测试数据
|
||||
```bash
|
||||
# 简单数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# Globe模式 (IP在真实GeoIP范围内)
|
||||
runtime\python\python.exe scripts\gen_test_data.py --globe
|
||||
|
||||
# 复杂数据 (5000行24列55%缺失)
|
||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
|
||||
```
|
||||
|
||||
### 列结构调查
|
||||
```bash
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Polars Version Compatibility
|
||||
|
||||
| API | 旧版(<1.0) | 新版(≥1.0) |
|
||||
|-----|-----------|-----------|
|
||||
| mode | `pl.mode()` | `pl.col(col).mode()` |
|
||||
| encoding | `encoding='utf-8'` | `encoding='utf8'` |
|
||||
| value_counts col | `'index'` | 原列名 |
|
||||
| streaming | `collect(streaming=True)` | `collect(engine='streaming')` |
|
||||
| truth value | `if series:` | `if series.item():` |
|
||||
|
||||
## 32B LLM Optimization
|
||||
|
||||
- Step guidance: 每步告诉模型下一步做什么
|
||||
- Truncation: 工具结果截断至 1200 chars
|
||||
- Timeout: 90s (32B 推理更慢)
|
||||
- Max steps: 15
|
||||
- Tool description: 一行简洁描述
|
||||
|
||||
---
|
||||
|
||||
### v6 (2026-07-16) — 基于真实TLS特征的恶意流量分析
|
||||
|
||||
| 模块 | 变动 |
|
||||
|------|------|
|
||||
| value_normalizer.py | 新增: 0ver/0cph/0crv/cipher-suite/ecdhe-named-curve hex→enum归一化, cnrs/isrs blank→no |
|
||||
| data_loader.py | 集成normalize_lf到清洗流水线，输出_norm列 |
|
||||
| entity_aggregator.py | 重构特征: 20+增强TLS特征(has_sni, aead_ratio, tls_modern_ratio, avg_bytes_per_packet, recoverable_ratio等) |
|
||||
| gen_test_data.py | 3类真实流量画像: 正常浏览(60%)/代理VPN(20%)/恶意可疑(20%) |
|
||||
| 特征维度 | 20→33维, 覆盖TLS指纹/连接行为/异常检测/时域特征 |
|
||||
|
||||
## Completed (2026-07-16)
|
||||
|
||||
18个问题全部修复，3次端到端管道测试全部通过（简单CSV、用户自定义列名CSV、大规模CSV）。
|
||||
|
||||
| 工作流 | 问题 | 状态 |
|
||||
|--------|------|------|
|
||||
| A1 | 重启后旧数据可分析 | ✅ 目录不存在时优雅降级，标记为failed |
|
||||
| A2 | Web图标所有页面显示 | ✅ 创建favicon.svg + base.html添加link |
|
||||
| A3 | 上传按钮在顶部 | ✅ 移至文件列表上方，sticky定位 |
|
||||
| A10 | 文件上传限制 | ✅ 去除了Django字段限制，增大至256MB/50GB |
|
||||
| A11 | 上传页删除功能 | ✅ 所有状态显示删除按钮，处理中弹出确认 |
|
||||
| B7 | TLS 0ver hex格式 | ✅ 添加TLS_HEX_MAP，识别03 03/03 04 |
|
||||
| B15 | "+"字符串识别为数字 | ✅ FLOAT检测添加+/前缀检查 |
|
||||
| B18 | 列名映射 | ✅ 使用精确^exact_name$匹配，无模糊猜测 |
|
||||
| C4 | Traceback截断 | ✅ tool_registry.py移除[:200] |
|
||||
| D5 | 大数据量卡顿 | ✅ 添加head参数、50K降采样、MAX_ROWS=500 |
|
||||
| D6 | 进度条 | ✅ 添加progress_pct/progress_msg字段 |
|
||||
| D14 | 低内存崩溃 | ✅ MiniBatchKMeans、PCA降维、内存检查 |
|
||||
| D16 | 空闲高IO | ✅ 日志级别已为INFO，无需修改 |
|
||||
| E8 | Globe响应式缩放 | ✅ 动态H计算、vh单位、resize处理 |
|
||||
| E9 | Globe ?data=参数 | ✅ 支持从SessionStore直接加载数据集 |
|
||||
| F12 | LLM日志 | ✅ 添加callback机制、run_log字段 |
|
||||
| G13 | 聚类图表 | ✅ 添加簇大小柱状图、Silhouette对比图 |
|
||||
| H17 | 聚类质量 | ✅ 方差过滤、相关过滤、HDBSCAN自动调参 |
|
||||
@@ -0,0 +1,390 @@
|
||||
# 天璇 (TianXuan) — TLS 流数据分析与实体画像系统
|
||||
|
||||
基于 Polars + scikit-learn + Three.js 的 TLS 流数据分析工具链。支持 CSV 批量导入、自动实体检测、多维聚合、聚类分析、特征提取、**3D 地球流量可视化**。提供 Django Web 操作界面和 MCP 协议接口，支持 32B 大语言模型编排分析流程。**全离线运行**，所有 JS 库/地图贴图/GeoIP 数据库均已内嵌。
|
||||
|
||||
---
|
||||
|
||||
## 目录
|
||||
|
||||
1. [系统概述](#1-系统概述)
|
||||
2. [快速开始](#2-快速开始)
|
||||
3. [用户指南](#3-用户指南)
|
||||
4. [页面功能说明](#4-页面功能说明)
|
||||
5. [CLI 管道](#5-cli-管道)
|
||||
6. [MCP 工具文档](#6-mcp-工具文档)
|
||||
7. [测试](#7-测试)
|
||||
8. [项目结构](#8-项目结构)
|
||||
9. [常见问题](#9-常见问题)
|
||||
|
||||
---
|
||||
|
||||
## 1. 系统概述
|
||||
|
||||
### 1.1 解决的问题
|
||||
|
||||
企业网络中 TLS 流量通常以 CSV 形式导出（每行一条流记录），数据量大、列名不统一、缺乏标签。本系统提供一条从原始 CSV 到实体画像的自动化分析流水线：
|
||||
|
||||
```
|
||||
CSV 文件 → 上传/加载 → 自动检测实体列 → 按实体聚合特征
|
||||
→ 实体级聚类 → 提取每簇区分特征 → Web 可视化 / MCP 接口
|
||||
```
|
||||
|
||||
### 1.2 核心能力
|
||||
|
||||
| 模块 | 功能说明 |
|
||||
|------|---------|
|
||||
| **数据加载** | 多文件合并、BOM 自动检测、跨文件 schema 容错、递归 `**/*.csv`、中文路径支持 |
|
||||
| **实体检测** | 基于列名关键词 + 唯一值比率 + 数据类型的三维打分，自动推荐实体列（如 `src_ip`、`sni`、`user`） |
|
||||
| **实体聚合** | 按实体分组计算流统计、目标多样性、TLS 特征、协议特征、时间模式、地理位置（经纬度） |
|
||||
| **IP 子网聚合** | 支持 /24 和 /28 子网掩码，同一子网下所有 IP 合并为一个实体节点 |
|
||||
| **多列复合键** | 支持 `(src_ip, dst_ip)` 等多列组合作为实体标识 |
|
||||
| **聚类分析** | HDBSCAN（自动确定簇数）/ KMeans，零样本自动降级，Silhouette / Davies-Bouldin 质量评估 |
|
||||
| **特征提取** | Z-Score / ANOVA 方法计算每个聚类的区分性特征 |
|
||||
| **3D 地球** | 基于 Three.js 的 TLS 流量弧线可视化，多数据源叠加，经纬线网格，国境线轮廓 |
|
||||
| **地理可视化** | PCA 散点图 + Canvas 原生渲染（无外部 JS 依赖） |
|
||||
| **LLM 集成** | OpenAI 兼容 API、LLM 自动编排分析流程（profile → 实体 → 聚类 → 特征） |
|
||||
| **Web 界面** | CSV 上传拖拽、手动分步分析向导、LLM 自动分析、配置管理、运行记录查看 |
|
||||
| **MCP 协议** | 12 个 JSON-RPC 工具，供外部 LLM 编排调用 |
|
||||
| **便携运行时** | 项目内置 Python 3.12 运行时（593MB），零系统依赖，`run.bat` 双击即用 |
|
||||
|
||||
### 1.3 技术栈
|
||||
|
||||
| 组件 | 技术选型 |
|
||||
|------|---------|
|
||||
| **后端框架** | Django 4.2 (SQLite WAL) |
|
||||
| **数据处理** | Polars 1.42.1 (LazyFrame 延迟计算 + streaming) |
|
||||
| **机器学习** | scikit-learn (StandardScaler, HDBSCAN, KMeans, PCA) |
|
||||
| **MCP 协议** | Python MCP SDK (stdio 传输) |
|
||||
| **前端 3D** | Three.js (603KB, 离线) |
|
||||
| **前端图表** | 纯 Canvas 2D API 原生绘制 |
|
||||
| **配置** | Pydantic + YAML |
|
||||
| **LLM 接口** | OpenAI chat/completions 兼容 API |
|
||||
| **运行时** | 嵌入式 Python 3.12 (Win7 兼容版 adang1345) |
|
||||
|
||||
---
|
||||
|
||||
## 2. 快速开始
|
||||
|
||||
### 2.1 用户模式（无需预装 Python）
|
||||
|
||||
```bat
|
||||
1. 解压项目文件夹到任意目录
|
||||
2. 双击 run.bat
|
||||
3. 等待命令行显示 "Starting TianXuan..."
|
||||
4. 浏览器自动打开 http://127.0.0.1:8000/
|
||||
5. 点击「上传数据」→ 选择 CSV 文件 → 上传 → 等待预处理完成
|
||||
6. 进入「手动分析」→ 选择数据集 → 配置参数 → 运行分析
|
||||
7. 查看结果：聚类概览、簇详情、实体画像、地球分布图
|
||||
```
|
||||
|
||||
### 2.2 开发者模式
|
||||
|
||||
```bash
|
||||
# 初始化数据库
|
||||
runtime\python\python.exe manage.py migrate
|
||||
|
||||
# 生成测试数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# 启动开发服务器
|
||||
runtime\python\python.exe manage.py runserver
|
||||
|
||||
# 浏览器打开 http://127.0.0.1:8000/
|
||||
```
|
||||
|
||||
### 2.3 脚本一览
|
||||
|
||||
| 脚本 | 功能 |
|
||||
|------|------|
|
||||
| `run.bat` | 启动 Django Web 界面（:8000），自动打开浏览器 |
|
||||
| `shell.bat` | 打开 Django Shell（Python 交互式环境） |
|
||||
|
||||
### 2.4 一键分析管道
|
||||
|
||||
```bash
|
||||
# 加载 CSV → 自动检测实体 → 聚合 → 聚类 → 特征提取 → 写入数据库
|
||||
runtime\python\python.exe manage.py run_pipeline "data/input.csv"
|
||||
|
||||
# 指定聚类算法
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --algo kmeans
|
||||
|
||||
# 手动指定实体列（跳过自动检测）
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --entity-col src_ip
|
||||
|
||||
# 导出结果
|
||||
runtime\python\python.exe manage.py run_pipeline "data/*.csv" --output ./results/
|
||||
```
|
||||
|
||||
输出示例：
|
||||
```
|
||||
[1/5] 加载 CSV: data/test_flows.csv → 500 rows, 1 files, 0.2 MB
|
||||
[2/5] 检测实体列 → 自动检测: src_ip (共 6 候选)
|
||||
[3/5] 实体聚合 → 50 个实体, 18 维特征
|
||||
[4/5] 聚类 (hdbscan) → 2 个簇, 噪声比 0.24, Silhouette: 0.2557
|
||||
[5/5] 特征提取 → 30 个特征已保存到数据库
|
||||
*** 分析完成! Run ID: #1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 用户指南
|
||||
|
||||
### 3.1 上传数据
|
||||
|
||||
1. 点击导航栏「上传数据」
|
||||
2. 拖拽 CSV 文件到虚线区域，或点击选择文件
|
||||
3. 支持多文件上传，自动拼接相同 schema 的文件
|
||||
4. 支持 `schema_strict=false` 模式（默认）：不同列名的文件会自动合并，缺失列填 null
|
||||
5. 支持 `.zip` 压缩包自动解压
|
||||
6. 上传后自动在后台进行预处理（加载 → 检测实体列 → 聚合），完成后状态变为 `ready`
|
||||
7. 进度条实时显示处理状态
|
||||
|
||||
**数据格式要求**：
|
||||
- 每行一条 TLS 流记录
|
||||
- 建议包含列：`src_ip`、`dst_ip`、`proto`、`bytes_sent`、`bytes_rev`、`duration`、`packets` 等
|
||||
- 可选地理列：`latitude` / `lat` / `y` 和 `longitude` / `lon` / `lng` / `x`（自动识别）
|
||||
- 可选时间列：`timestamp` / `ts` / `time`（自动识别）
|
||||
|
||||
### 3.2 手动分析
|
||||
|
||||
3 步向导完成分析：
|
||||
|
||||
**Step 1 - 选择数据集**：从已预处理完成的数据集中选择一个（显示行数、已检测的实体列）
|
||||
|
||||
**Step 2 - 配置参数**：
|
||||
- 聚类算法：HDBSCAN（自动确定簇数，推荐）/ KMeans（需预设 k 值）
|
||||
- 最小簇大小（min_cluster_size）：默认 5，数据不足时自动降级
|
||||
|
||||
**Step 3 - 确认运行**：回顾配置，点击「开始分析」。后台自动完成聚类 + 特征提取，完成后跳转到结果页。
|
||||
|
||||
### 3.3 LLM 自动分析
|
||||
|
||||
需要先在「配置」页面设置 LLM 的 base_url 和 API Key：
|
||||
|
||||
1. 进入「LLM 分析」页面
|
||||
2. 选择一个已预处理的数据集
|
||||
3. 点击「开始自动分析」
|
||||
4. 后端 LLM 编排器会自动调用 profile → build_entity_profiles → run_clustering → extract_features
|
||||
5. 进度条实时显示当前步骤
|
||||
|
||||
### 3.4 查看结果
|
||||
|
||||
#### 运行详情页
|
||||
- 总流数、实体数、簇数 摘要卡片
|
||||
- 簇列表：每个簇的大小、占比、Silhouette 分数
|
||||
|
||||
#### 聚类概览页
|
||||
- **PCA 散点图**：实体在二维主成分空间中的分布，颜色区分聚类
|
||||
- **地理分布图**：实体在地理坐标上的分布（需 CSV 包含经纬度列），支持缺失值
|
||||
- 每个簇的详情卡片：Top 5 区分特征
|
||||
|
||||
#### 簇详情页
|
||||
- 特征表：每列的均值、标准差、中位数、区分度分数（Top 50）
|
||||
- 实体列表：该簇包含的实体（Top 50），可点击查看详情
|
||||
|
||||
#### 实体画像页
|
||||
- 实体标识、所属簇
|
||||
- 聚合特征值和相对簇均值的偏离（Z-score）
|
||||
- 偏差显著的特征以颜色高亮（|Z| > 2）
|
||||
|
||||
#### 3D 地球页
|
||||
- Three.js 3D 地球，TLS 流量弧线连接源目 IP
|
||||
- 弧线颜色标识 TLS 版本（TLSv1.3 蓝 / v1.2 粉 / 其他青）
|
||||
- 鼠标拖拽旋转、滚轮缩放
|
||||
- 经纬线网格 + 国境线轮廓
|
||||
- 多数据源叠加复选框
|
||||
- 脉冲光点动画（背面剔除）
|
||||
|
||||
---
|
||||
|
||||
## 4. 页面功能说明
|
||||
|
||||
| 页面 | 路由 | 功能 |
|
||||
|------|------|------|
|
||||
| **首页** | `/` | 最近运行记录概览 |
|
||||
| **上传数据** | `/upload/` | 拖拽上传 CSV，自动预处理 |
|
||||
| **手动分析** | `/analyze/manual/` | 3 步向导：选数据→设参数→运行 |
|
||||
| **LLM 分析** | `/analyze/auto/` | LLM 自动编排分析流程 |
|
||||
| **运行记录** | `/runs/` | 所有分析运行列表 |
|
||||
| **运行详情** | `/runs/<id>/` | 单次运行摘要 + 簇列表 |
|
||||
| **聚类概览** | `/clusters/<run_id>/` | PCA 散点图 + 每簇区分特征 |
|
||||
| **簇详情** | `/clusters/<run_id>/<label>/` | 特征表 + 实体列表 |
|
||||
| **实体画像** | `/entities/<id>/` | 单实体特征 + 簇偏离 |
|
||||
| **3D 地球** | `/globe/` | Three.js 流量弧线可视化 |
|
||||
| **配置** | `/config/` | 系统配置编辑 + LLM 测试 |
|
||||
| **日志** | `/logs/` | 运行日志查看 |
|
||||
|
||||
---
|
||||
|
||||
## 5. CLI 管道
|
||||
|
||||
```bash
|
||||
# 完整管道（HDBSCAN 默认）
|
||||
runtime\python\python.exe manage.py run_pipeline data/complex_test.csv
|
||||
|
||||
# 指定 KMeans
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --algo kmeans
|
||||
|
||||
# 手动指定实体列
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --entity-col src_ip
|
||||
|
||||
# 设置子网聚合掩码
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --subnet-masks 24 28
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. MCP 工具文档
|
||||
|
||||
### 6.1 工具总览
|
||||
|
||||
系统通过 MCP (Model Context Protocol) 暴露 12 个工具，支持 LLM 以 JSON-RPC 方式编排调用。
|
||||
|
||||
| # | 工具名 | 功能 | 必需参数 |
|
||||
|---|--------|------|----------|
|
||||
| 1 | `load_data` | 加载 CSV 文件（支持 glob、递归、schema 容错） | `csv_glob` |
|
||||
| 2 | `profile_data` | 数据集概要统计 + 相关性矩阵 | `dataset_id` |
|
||||
| 3 | `filter_data` | 按条件过滤（12 操作符 + AND/OR） | `dataset_id`, `filters` |
|
||||
| 4 | `preprocess_data` | 预处理（标准化/编码/填充） | `dataset_id`, `columns` |
|
||||
| 5 | `run_clustering` | 执行聚类（HDBSCAN/KMeans，带质量评估） | `dataset_id`, `cluster_columns` |
|
||||
| 6 | `evaluate_clustering` | 评估聚类质量（Silhouette/DB/CH） | `cluster_result_id`, `dataset_id` |
|
||||
| 7 | `extract_features` | 提取各聚类区分特征（Z-Score/ANOVA） | `dataset_id`, `cluster_result_id` |
|
||||
| 8 | `export_results` | 导出结果到磁盘（CSV/Parquet/JSON） | `result_id`, `output_path` |
|
||||
| 9 | `list_datasets` | 列出所有活跃数据集 | 无 |
|
||||
| 10 | `drop_dataset` | 删除数据集释放内存 | `dataset_id` |
|
||||
| 11 | `clone_dataset` | 浅拷贝数据集（不复制数据） | `dataset_id` |
|
||||
| 12 | `build_entity_profiles` | 自动检测实体列 + 聚合画像 | `dataset_id` |
|
||||
|
||||
### 6.2 响应约定
|
||||
|
||||
- 成功时返回 JSON 对象，含 `truncated: false`
|
||||
- 超过阈值时自动截断，设置 `truncated: true` + `omitted_columns`
|
||||
- 错误时返回 `{ "error": "描述", "truncated": false }`
|
||||
- 所有响应通过 `TextContent` 包装返回
|
||||
|
||||
### 6.3 JSON-RPC 调用示例
|
||||
|
||||
```
|
||||
→ {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"load_data","arguments":{"csv_glob":"data/*.csv"}}}
|
||||
← {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"{\"dataset_id\":\"ds_...\",\"row_count\":500}"}]}}
|
||||
|
||||
→ {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"build_entity_profiles","arguments":{"dataset_id":"ds_...","auto_detect":true}}}
|
||||
← {"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"{\"entity_dataset_id\":\"entity_...\",\"entity_count\":50,\"entity_column_used\":\"src_ip\"}"}]}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 测试
|
||||
|
||||
```bash
|
||||
# 运行全部测试
|
||||
runtime\python\python.exe -m pytest tests -q
|
||||
|
||||
# 运行特定测试文件
|
||||
runtime\python\python.exe -m pytest tests/test_entity.py -v
|
||||
|
||||
# 运行端到端测试
|
||||
runtime\python\python.exe -m pytest tests/test_e2e.py -v
|
||||
```
|
||||
|
||||
### 生成测试数据
|
||||
```bash
|
||||
# 简单数据
|
||||
runtime\python\python.exe scripts\gen_test_data.py --rows 1000
|
||||
|
||||
# Globe 模式（IP 在真实 GeoIP 范围内，弧线可见）
|
||||
runtime\python\python.exe scripts\gen_test_data.py --globe
|
||||
|
||||
# 复杂数据（5000 行 24 列含 55% 缺失）
|
||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000
|
||||
```
|
||||
|
||||
### 列结构调查
|
||||
```bash
|
||||
# 统计各 CSV 文件的列名、类型、分布
|
||||
runtime\python\python.exe scripts\column_survey.py --csv "data/*.csv"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 项目结构
|
||||
|
||||
```
|
||||
tianxuan/
|
||||
├── tianxuan/ # Django 项目配置
|
||||
│ ├── settings.py # Django 配置（数据库、中间件）
|
||||
│ ├── urls.py # 根路由
|
||||
│ └── llm_orchestrator.py # LLM 编排后端（tool_calls 循环）
|
||||
│
|
||||
├── analysis/ # 核心分析模块（Django app）
|
||||
│ ├── data_loader.py # CSV 加载、BOM 检测、schema 容错、递归 glob
|
||||
│ ├── data_profiler.py # 数据集概要统计与相关性矩阵
|
||||
│ ├── entity_detector.py # 实体列自动检测（关键词 + unique_ratio 打分）
|
||||
│ ├── entity_aggregator.py # 实体聚合（流→实体画像），含 IP 子网聚合
|
||||
│ ├── type_classifier.py # 值优先类型检测（MAC/端口/IPv4/URL/HEX/ENUM/LAT_LON）
|
||||
│ ├── data_validator.py # 列校验（缺失率/异常值/IP有效性）
|
||||
│ ├── geoip.py # GeoIP 经纬度查询
|
||||
│ ├── session_store.py # 线程安全内存会话存储（Singleton + RLock）
|
||||
│ ├── tool_registry.py # 12 个 MCP 工具定义 + 异步处理函数
|
||||
│ ├── mcp_server.py # MCP stdio 服务器
|
||||
│ ├── views.py # 所有 Django 视图
|
||||
│ ├── urls.py # 16 条 URL 路由
|
||||
│ ├── models.py # ORM 模型
|
||||
│ └── management/commands/
|
||||
│ ├── start_mcp.py # manage.py start_mcp
|
||||
│ └── run_pipeline.py # manage.py run_pipeline（一键 CLI 管道）
|
||||
│
|
||||
├── config/
|
||||
│ ├── config.yaml # 配置文件（自动生成，带注释）
|
||||
│ └── loader.py # Pydantic 配置加载器
|
||||
│
|
||||
├── templates/tianxuan/ # 11 个页面模板
|
||||
├── static/tianxuan/ # Three.js + 地球贴图 + 国境线
|
||||
├── runtime/python/ # 便携 Python 3.12 运行时（593MB）
|
||||
├── tests/ # 53 个测试 (pytest)
|
||||
├── scripts/ # 工具脚本
|
||||
├── docs/ # 文档
|
||||
├── run.bat # 用户启动脚本（双击即用）
|
||||
└── manage.py # Django 管理入口
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 常见问题
|
||||
|
||||
### Q1: 双击 run.bat 后浏览器没打开？
|
||||
A: 手动打开浏览器访问 `http://127.0.0.1:8000/`。如果也无法访问，检查命令行是否有报错。
|
||||
|
||||
### Q2: 压缩时提示 `db.sqlite3-wal 被占用`？
|
||||
A: 有残留的 Python 进程持有 SQLite 连接。运行以下命令后重试：
|
||||
```powershell
|
||||
Get-Process python -ErrorAction SilentlyContinue | Stop-Process -Force
|
||||
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
### Q3: 地图不显示？
|
||||
A: CSV 需包含 lat/lon 列（列名含 `lat`/`latitude`/`lon`/`longitude`/`lng` 自动识别），缺失值不显示。页面会提示跳过的数量。
|
||||
|
||||
### Q4: 上传 CSV 后显示 "处理失败"？
|
||||
A: 在运行详情页查看错误消息。常见原因：
|
||||
- CSV 编码不是 UTF-8
|
||||
- 所有列都是非数值类型（无法聚类，返回 0 簇）
|
||||
- 数据量太小（< 3 条实体记录，HDBSCAN 自动跳过）
|
||||
|
||||
### Q5: 如何配置 LLM？
|
||||
A: 导航到「配置」页面，填写 base_url 和 API Key，点击「测试连通性」验证。支持 OpenAI 兼容的任何 API。
|
||||
|
||||
### Q6: 支持哪些 CSV 编码？
|
||||
A: 自动检测 UTF-8 BOM、UTF-16LE BOM、UTF-16BE BOM。无 BOM 时默认 UTF-8。中文路径完全支持。
|
||||
|
||||
### Q7: 不同 CSV 文件的列名不一致怎么办？
|
||||
A: 默认 `schema_strict=false`，系统自动取所有文件的列名并集，缺失列填 null。如需严格模式，在配置中将 `schema_strict` 改为 `true`。
|
||||
|
||||
### Q8: 项目太大，如何缩小？
|
||||
```powershell
|
||||
Remove-Item -Recurse *.pyc, __pycache__ -Force
|
||||
Remove-Item db.sqlite3-wal, db.sqlite3-shm -Force -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
from django.contrib import admin
|
||||
from .models import AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
||||
|
||||
|
||||
@admin.register(AnalysisRun)
|
||||
class AnalysisRunAdmin(admin.ModelAdmin):
|
||||
list_display = ('id', 'created_at', 'status', 'csv_glob', 'entity_count', 'cluster_count')
|
||||
list_filter = ('status', 'created_at')
|
||||
ordering = ('-created_at',)
|
||||
|
||||
|
||||
@admin.register(ClusterResult)
|
||||
class ClusterResultAdmin(admin.ModelAdmin):
|
||||
list_display = ('run', 'cluster_label', 'size', 'noise_ratio', 'silhouette_score')
|
||||
list_filter = ('run',)
|
||||
|
||||
|
||||
@admin.register(EntityProfile)
|
||||
class EntityProfileAdmin(admin.ModelAdmin):
|
||||
list_display = ('run', 'entity_value', 'cluster_label', 'embedding_x', 'embedding_y')
|
||||
list_filter = ('run', 'cluster_label')
|
||||
search_fields = ('entity_value',)
|
||||
|
||||
|
||||
@admin.register(ClusterFeature)
|
||||
class ClusterFeatureAdmin(admin.ModelAdmin):
|
||||
list_display = ('cluster', 'feature_name', 'mean', 'std', 'distinguishing_score')
|
||||
list_filter = ('cluster__run',)
|
||||
ordering = ('-distinguishing_score',)
|
||||
@@ -0,0 +1,7 @@
|
||||
from django.apps import AppConfig
|
||||
|
||||
|
||||
class AnalysisConfig(AppConfig):
|
||||
default_auto_field = 'django.db.models.BigAutoField'
|
||||
name = 'analysis'
|
||||
label = 'tianxuan_analysis'
|
||||
@@ -0,0 +1,494 @@
|
||||
"""CSV loading with per-file schema validation, BOM detection, and multi-file merge.
|
||||
|
||||
Supports multiple encodings and delimiter-separated files. The main entry
|
||||
point is :func:`load_csv_directory`, which globs files, detects BOM per file,
|
||||
validates schema consistency, and returns a merged :class:`pl.LazyFrame`.
|
||||
"""
|
||||
import glob
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import polars as pl
|
||||
import yaml
|
||||
|
||||
from analysis.type_classifier import classify_schema, DataType
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
SUPPORTED_ENCODINGS = ['utf-8', 'utf-8-sig', 'utf-16le', 'utf-16be', 'latin-1']
|
||||
|
||||
|
||||
# ── numeric coercion helper ──────────────────────────────────────────────
|
||||
|
||||
def _coerce_to_float(series: pl.Series) -> pl.Series:
|
||||
"""Convert string series to Float64.
|
||||
|
||||
Every entry that doesn't parse as a valid float becomes null.
|
||||
Handles blank strings, standalone '+'/'-', 'N/A', 'null', 'None', etc.
|
||||
"""
|
||||
series = series.cast(pl.Utf8).str.strip_chars()
|
||||
artifacts = ('', '+', '-', '.', '..', 'N/A', 'n/a', 'NA', 'null', 'None', 'NULL', '?', '---', '--', '/')
|
||||
is_artifact = series.is_in(pl.Series(artifacts).implode())
|
||||
result = []
|
||||
for v, bad in zip(series.to_list(), is_artifact.to_list()):
|
||||
if v is None or bad:
|
||||
result.append(None)
|
||||
else:
|
||||
# Check for "+"/"-" prefix artifacts (e.g., "+abc", "-xyz")
|
||||
# where the value isn't a genuine numeric string. Values like
|
||||
# "+5" or "-3.14" are still accepted as valid floats below.
|
||||
if len(v) > 1 and v[0] in '+-':
|
||||
try:
|
||||
float(v[1:]) # validate suffix minus prefix
|
||||
except ValueError:
|
||||
result.append(None)
|
||||
continue
|
||||
try:
|
||||
result.append(float(v))
|
||||
except (ValueError, TypeError):
|
||||
result.append(None)
|
||||
return pl.Series(result, dtype=pl.Float64)
|
||||
|
||||
|
||||
# ── BOM detection ───────────────────────────────────────────────────────
|
||||
|
||||
def detect_bom(path: str) -> str:
|
||||
"""Read the first 4 bytes of *path* and return the detected encoding.
|
||||
|
||||
Returns ``'utf-8'`` when no BOM is found.
|
||||
|
||||
Note: Python 3 ``open()`` fully supports Unicode/Chinese paths on all
|
||||
platforms (no encoding dance needed).
|
||||
"""
|
||||
with open(path, 'rb') as f:
|
||||
header = f.read(4)
|
||||
|
||||
if header[:3] == b'\xef\xbb\xbf':
|
||||
return 'utf-8-sig'
|
||||
if header[:2] == b'\xff\xfe':
|
||||
return 'utf-16le'
|
||||
if header[:2] == b'\xfe\xff':
|
||||
return 'utf-16be'
|
||||
return 'utf-8'
|
||||
|
||||
|
||||
# ── schema utilities ────────────────────────────────────────────────────
|
||||
|
||||
def collect_schema(
|
||||
path: str,
|
||||
encoding: str = 'utf-8',
|
||||
delimiter: str = ',',
|
||||
sample_rows: int = 10000,
|
||||
) -> dict:
|
||||
"""Return {column_name: dtype_string} for a single CSV file.
|
||||
|
||||
Uses :func:`pl.scan_csv` with *sample_rows* rows for type inference.
|
||||
"""
|
||||
lf = pl.scan_csv(
|
||||
str(path), # str() ensures Unicode/Chinese paths resolve correctly
|
||||
encoding=encoding,
|
||||
separator=delimiter,
|
||||
infer_schema_length=sample_rows,
|
||||
)
|
||||
schema = lf.collect_schema()
|
||||
return {name: str(dtype) for name, dtype in zip(schema.names(), schema.dtypes())}
|
||||
|
||||
|
||||
def _merge_schema_entries(
|
||||
schemas: list[dict],
|
||||
) -> dict:
|
||||
"""Merge multiple schema dicts, raising on type conflicts.
|
||||
|
||||
Returns a single {name: dtype} dict. If a column appears in multiple
|
||||
files with different dtypes the *last* non-null dtype wins (with a
|
||||
warning embedded in the result).
|
||||
"""
|
||||
merged: dict[str, str] = {}
|
||||
conflicts: list[str] = []
|
||||
for s in schemas:
|
||||
for name, dtype in s.items():
|
||||
if name in merged and merged[name] != dtype:
|
||||
conflicts.append(name)
|
||||
merged[name] = dtype # last wins
|
||||
return merged, conflicts
|
||||
|
||||
|
||||
def validate_schemas(schemas: list[dict]) -> None:
|
||||
"""Compare schemas across files and raise :class:`ValueError` on mismatch.
|
||||
|
||||
Checks that every file has the same set of column names. Type
|
||||
differences across files are **not** considered an error (Polars can
|
||||
cast), but differences in *column sets* are.
|
||||
"""
|
||||
if not schemas:
|
||||
return
|
||||
|
||||
col_sets = [set(s.keys()) for s in schemas]
|
||||
reference = col_sets[0]
|
||||
|
||||
for i, cs in enumerate(col_sets[1:], start=1):
|
||||
only_in_ref = reference - cs
|
||||
only_in_current = cs - reference
|
||||
if only_in_ref or only_in_current:
|
||||
msg_parts = []
|
||||
if only_in_ref:
|
||||
msg_parts.append(f"File #{i} missing columns: {sorted(only_in_ref)}")
|
||||
if only_in_current:
|
||||
msg_parts.append(f"File #{i} has extra columns: {sorted(only_in_current)}")
|
||||
raise ValueError(f"Schema mismatch: {'; '.join(msg_parts)}")
|
||||
|
||||
|
||||
# ── dtype inference from schema ─────────────────────────────────────────
|
||||
|
||||
def detect_schema_dtypes(
|
||||
lf: pl.LazyFrame,
|
||||
sample_rows: int = 10000,
|
||||
) -> dict:
|
||||
"""Infer Polars dtype strings from a LazyFrame's schema.
|
||||
|
||||
The schema is read without collecting the full data (the scan scans
|
||||
a small sample).
|
||||
"""
|
||||
schema = lf.collect_schema()
|
||||
return {name: str(dtype) for name, dtype in zip(schema.names(), schema.dtypes())}
|
||||
|
||||
|
||||
# ── config loading ──────────────────────────────────────────────────────
|
||||
|
||||
def load_config(config_path: Optional[str]) -> dict:
|
||||
"""Load a YAML config file; returns an empty dict if *config_path* is None."""
|
||||
if config_path is None:
|
||||
return {}
|
||||
path = Path(config_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Config file not found: {config_path}")
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
return yaml.safe_load(f) or {}
|
||||
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _file_count(paths: list[str]) -> int:
|
||||
return len(paths)
|
||||
|
||||
|
||||
def _resolve_encoding(encoding: str, file_encoding: str) -> str:
|
||||
"""BOM-detected encoding overrides user-supplied encoding.
|
||||
|
||||
Polars >= 1.0 requires ``'utf8'`` (not ``'utf-8'``).
|
||||
"""
|
||||
resolved = file_encoding if file_encoding != 'utf-8' else encoding
|
||||
return resolved.replace('-', '') if resolved in ('utf-8', 'utf-8-sig') else resolved
|
||||
|
||||
|
||||
# ── main entry ──────────────────────────────────────────────────────────
|
||||
|
||||
def load_csv_directory(
|
||||
glob_pattern: str,
|
||||
encoding: str = 'utf-8',
|
||||
delimiter: str = ',',
|
||||
config_path: Optional[str] = None,
|
||||
schema_override: Optional[dict] = None,
|
||||
sample_rows: int = 10000,
|
||||
schema_strict: bool = False,
|
||||
recursive: bool = False,
|
||||
head: Optional[int] = None,
|
||||
) -> tuple[pl.LazyFrame, dict, int, int, float]:
|
||||
"""Load, validate, and merge CSV files matching *glob_pattern*.
|
||||
|
||||
Args:
|
||||
glob_pattern: Glob pattern for CSV files (e.g. ``data/*.csv``).
|
||||
encoding: Fallback encoding when no BOM is detected.
|
||||
delimiter: Column delimiter character.
|
||||
config_path: Optional path to a YAML config (merged into metadata).
|
||||
schema_override: If given, force these {column: dtype} overrides.
|
||||
sample_rows: Rows to scan for dtype inference per file.
|
||||
schema_strict: When True, raise ValueError on column mismatch across
|
||||
files. When False (default), log warnings and merge with
|
||||
``diagonal_relaxed`` (missing columns get null).
|
||||
recursive: When True, use ``glob.glob(..., recursive=True)`` to
|
||||
support ``**/*.csv`` patterns.
|
||||
head: Optional row limit for preview scenarios. When set, applies
|
||||
``lf.head(head)`` before returning.
|
||||
|
||||
Returns:
|
||||
A tuple of ``(lazyframe, schema_dict, total_row_count, file_count,
|
||||
memory_estimate_mb)``.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If no files match the glob pattern.
|
||||
ValueError: If schemas are inconsistent across files and
|
||||
*schema_strict* is True.
|
||||
"""
|
||||
# Normalise path separators for cross-platform Unicode support
|
||||
glob_pattern = os.path.normpath(glob_pattern)
|
||||
|
||||
# Resolve file list
|
||||
paths = sorted(glob.glob(glob_pattern, recursive=recursive))
|
||||
if not paths and not recursive:
|
||||
# Fallback: try recursive pattern
|
||||
paths = sorted(glob.glob(glob_pattern, recursive=True))
|
||||
if not paths:
|
||||
raise FileNotFoundError(f"No files match glob pattern: {glob_pattern}")
|
||||
|
||||
# Load config (user-provided or fall back to project default)
|
||||
config = load_config(config_path)
|
||||
if not config:
|
||||
default_path = Path(__file__).parent.parent / 'config' / 'config.yaml'
|
||||
if default_path.exists():
|
||||
config = load_config(str(default_path))
|
||||
|
||||
# Collect per-file schema
|
||||
schemas: list[dict] = []
|
||||
lazyframes: list[pl.LazyFrame] = []
|
||||
for path in paths:
|
||||
file_encoding = detect_bom(path)
|
||||
actual_encoding = _resolve_encoding(encoding, file_encoding)
|
||||
|
||||
lf = pl.scan_csv(
|
||||
str(path), # str() ensures Unicode/Chinese paths resolve correctly
|
||||
encoding=actual_encoding,
|
||||
separator=delimiter,
|
||||
infer_schema_length=max(sample_rows, 1000),
|
||||
)
|
||||
schema = detect_schema_dtypes(lf, sample_rows=sample_rows)
|
||||
|
||||
# Apply schema override for this file
|
||||
if schema_override:
|
||||
for col, dtype in schema_override.items():
|
||||
if col in schema:
|
||||
schema[col] = dtype
|
||||
|
||||
schemas.append(schema)
|
||||
lazyframes.append(lf)
|
||||
|
||||
if schema_strict:
|
||||
# Raise on column mismatch
|
||||
validate_schemas(schemas)
|
||||
else:
|
||||
# Lenient mode: log warnings for column mismatches
|
||||
col_sets = [set(s.keys()) for s in schemas]
|
||||
reference = col_sets[0]
|
||||
for i, cs in enumerate(col_sets[1:], start=1):
|
||||
only_in_ref = reference - cs
|
||||
only_in_current = cs - reference
|
||||
if only_in_ref:
|
||||
logger.warning(
|
||||
"File #%d missing columns: %s (will be null)",
|
||||
i + 1, sorted(only_in_ref),
|
||||
)
|
||||
if only_in_current:
|
||||
logger.warning(
|
||||
"File #%d has extra columns: %s (will be null in other files)",
|
||||
i + 1, sorted(only_in_current),
|
||||
)
|
||||
|
||||
# Merge schemas (unify column ordering — first file wins order)
|
||||
merged_schema, _ = _merge_schema_entries(schemas)
|
||||
|
||||
# Concatenate all LazyFrames (union vertically)
|
||||
merged_lf: pl.LazyFrame = pl.concat(lazyframes, how='diagonal_relaxed')
|
||||
|
||||
# Log schema and size estimate
|
||||
total_row_count_est = len(paths) * (lazyframes[0].collect(streaming=True).height if lazyframes else 0)
|
||||
logger.info(f'[LOAD] files={len(paths)} total_rows_est={total_row_count_est}')
|
||||
for col_name, dtype in sorted(merged_schema.items()):
|
||||
logger.info(f'[LOAD_SCHEMA] col={col_name} dtype={dtype}')
|
||||
|
||||
# Apply schema override on merged frame (cast columns)
|
||||
if schema_override:
|
||||
casts = {}
|
||||
for col, dtype_str in schema_override.items():
|
||||
if col in merged_schema:
|
||||
try:
|
||||
polars_dtype = getattr(pl, dtype_str.split('(')[0], None)
|
||||
if polars_dtype:
|
||||
casts[col] = pl.col(col).cast(polars_dtype)
|
||||
except Exception:
|
||||
pass # skip uncastable overrides silently
|
||||
if casts:
|
||||
merged_lf = merged_lf.with_columns(list(casts.values()))
|
||||
|
||||
# ── Cleaning: drop outdated columns ──────────────────────────────────
|
||||
geoip_cfg = config.get('data', {}).get('geoip', {})
|
||||
drop_cols = geoip_cfg.get(
|
||||
'drop_columns',
|
||||
[], # Empty by default — lat/lon columns are now cleaned, not dropped
|
||||
)
|
||||
existing_drop = [c for c in drop_cols if c in merged_schema]
|
||||
if existing_drop:
|
||||
merged_lf = merged_lf.drop(existing_drop)
|
||||
merged_schema = {k: v for k, v in merged_schema.items() if k not in existing_drop}
|
||||
logger.info('[CLEAN] dropped columns: %s', existing_drop)
|
||||
|
||||
# ── Cleaning: BOOL_ENUM normalisation ────────────────────────────────
|
||||
try:
|
||||
type_map = classify_schema(
|
||||
merged_lf.head(1000),
|
||||
config_overrides=config.get('columns', {}),
|
||||
)
|
||||
except Exception:
|
||||
type_map = {}
|
||||
|
||||
bool_enum_cols = [
|
||||
c for c, t in type_map.items()
|
||||
if t == DataType.BOOL_ENUM and c in merged_schema
|
||||
]
|
||||
if bool_enum_cols:
|
||||
# Only clean string-typed BOOL_ENUM columns; native Boolean is already clean
|
||||
schema_dtypes = dict(zip(
|
||||
merged_lf.collect_schema().names(),
|
||||
merged_lf.collect_schema().dtypes(),
|
||||
))
|
||||
bool_exprs = []
|
||||
cleaned_names: list[str] = []
|
||||
for c in bool_enum_cols:
|
||||
if schema_dtypes.get(c) not in (pl.Utf8, pl.String):
|
||||
continue # native Boolean / numeric — skip
|
||||
c_str = pl.col(c).cast(pl.Utf8).str.strip_chars()
|
||||
cleaned = (
|
||||
pl.when(c_str == "")
|
||||
.then(pl.lit(None, pl.Utf8))
|
||||
.when(c_str.str.to_lowercase() == "+")
|
||||
.then(pl.lit("True"))
|
||||
.when(c_str.str.to_lowercase() == "-")
|
||||
.then(pl.lit("False"))
|
||||
.otherwise(c_str)
|
||||
)
|
||||
bool_exprs.append(cleaned.alias(c))
|
||||
cleaned_names.append(c)
|
||||
if bool_exprs:
|
||||
merged_lf = merged_lf.with_columns(bool_exprs)
|
||||
for c in cleaned_names:
|
||||
merged_schema[c] = 'Utf8'
|
||||
logger.info('[CLEAN] BOOL_ENUM columns normalised: %s', cleaned_names)
|
||||
|
||||
# ── Cleaning: LAT_LON normalisation ────────────────────────────────
|
||||
latlon_cols = [
|
||||
c for c, t in type_map.items()
|
||||
if t == DataType.LAT_LON and c in merged_schema
|
||||
]
|
||||
if latlon_cols:
|
||||
schema_dtypes_ll = dict(zip(
|
||||
merged_lf.collect_schema().names(),
|
||||
merged_lf.collect_schema().dtypes(),
|
||||
))
|
||||
latlon_exprs = []
|
||||
for c in latlon_cols:
|
||||
dtype = schema_dtypes_ll.get(c)
|
||||
if dtype in (pl.Utf8, pl.String):
|
||||
latlon_exprs.append(
|
||||
pl.col(c).map_batches(_coerce_to_float, return_dtype=pl.Float64).alias(c)
|
||||
)
|
||||
merged_schema[c] = 'Float64'
|
||||
if latlon_exprs:
|
||||
merged_lf = merged_lf.with_columns(latlon_exprs)
|
||||
logger.info('[CLEAN] LAT_LON columns normalised to Float64: %s', latlon_cols)
|
||||
|
||||
# ── Cleaning: Generic numeric coercion ────────────────────────────
|
||||
# For any remaining string column (not IPv4/URL/HEX/ENUM/etc.),
|
||||
# try to coerce to Float64. Whatever can't parse becomes null.
|
||||
# This prevents "mean on str column" crashes for ANY column.
|
||||
_SKIP_NUM_COERCE = (DataType.IPv4, DataType.URL, DataType.HEX,
|
||||
DataType.ENUM, DataType.LAT_LON)
|
||||
# Also protect columns with these name patterns (MAC addresses, etc.)
|
||||
_SKIP_NAME_PATTERNS = ('mac', 'mac_', '_mac', 'addr', 'address')
|
||||
schema_dtypes_gn = dict(zip(
|
||||
merged_lf.collect_schema().names(),
|
||||
merged_lf.collect_schema().dtypes(),
|
||||
))
|
||||
generic_numeric_cols = []
|
||||
for c, t in type_map.items():
|
||||
if t in _SKIP_NUM_COERCE:
|
||||
continue
|
||||
if c not in merged_schema:
|
||||
continue
|
||||
dtype = schema_dtypes_gn.get(c)
|
||||
if dtype not in (pl.Utf8, pl.String):
|
||||
continue # already numeric
|
||||
# Skip columns with protected name patterns
|
||||
cl = c.lower().replace('-', '_').replace(' ', '_')
|
||||
if any(p in cl for p in _SKIP_NAME_PATTERNS):
|
||||
continue
|
||||
generic_numeric_cols.append(c)
|
||||
|
||||
if generic_numeric_cols:
|
||||
num_exprs = []
|
||||
for c in generic_numeric_cols:
|
||||
num_exprs.append(
|
||||
pl.col(c).map_batches(_coerce_to_float, return_dtype=pl.Float64).alias(c)
|
||||
)
|
||||
if num_exprs:
|
||||
merged_lf = merged_lf.with_columns(num_exprs)
|
||||
for c in generic_numeric_cols:
|
||||
merged_schema[c] = 'Float64'
|
||||
logger.info('[CLEAN] Generic numeric coercion applied: %s', generic_numeric_cols)
|
||||
|
||||
# ── Cleaning: HEX preprocessing (hex_pairs_count) ────────────────────
|
||||
hex_cols = [
|
||||
c for c, t in type_map.items()
|
||||
if t == DataType.HEX and c in merged_schema
|
||||
]
|
||||
if hex_cols:
|
||||
hex_exprs = []
|
||||
for c in hex_cols:
|
||||
hex_exprs.append(
|
||||
pl.col(c).str.split(" ").list.len().alias(f"{c}_hex_pairs_count")
|
||||
)
|
||||
merged_lf = merged_lf.with_columns(hex_exprs)
|
||||
for c in hex_cols:
|
||||
merged_schema[f"{c}_hex_pairs_count"] = 'UInt32'
|
||||
logger.info('[CLEAN] HEX columns added hex_pairs_count: %s', hex_cols)
|
||||
|
||||
# ── Value normalization (TLS hex→enum, display→enum) ────────────────
|
||||
from analysis.value_normalizer import normalize_lf
|
||||
merged_lf = normalize_lf(merged_lf, type_map)
|
||||
# Only add _norm columns to schema if the source column exists in type_map
|
||||
_NORM_MAP = [
|
||||
('0ver', '0ver_norm'),
|
||||
('0cph', '0cph_norm'),
|
||||
('cipher-suite', 'cipher_suite_norm'),
|
||||
('0crv', '0crv_norm'),
|
||||
('ecdhe-named-curve', 'ecdhe_named_curve_norm'),
|
||||
('cnrs', 'cnrs_norm'),
|
||||
('isrs', 'isrs_norm'),
|
||||
]
|
||||
_added = []
|
||||
for src, tgt in _NORM_MAP:
|
||||
if src in type_map:
|
||||
merged_schema[tgt] = 'Utf8'
|
||||
_added.append(tgt)
|
||||
if _added:
|
||||
logger.info('[CLEAN] Value normalization columns added: %s', _added)
|
||||
|
||||
# Row count estimate (fast path: use first file row count × files)
|
||||
# For accurate count we'd need to collect, but that defeats lazy.
|
||||
# We estimate from the first file's scanned schema info.
|
||||
first_rows = lazyframes[0].collect(streaming=True).height if lazyframes else 0
|
||||
total_row_count = first_rows * len(paths)
|
||||
|
||||
# Memory estimate
|
||||
bytes_per_row = 0
|
||||
for col_name, dtype_str in merged_schema.items():
|
||||
numeric_part = dtype_str.split('(')[0].lower()
|
||||
size_map = {
|
||||
'int8': 1, 'int16': 2, 'int32': 4, 'int64': 8,
|
||||
'uint8': 1, 'uint16': 2, 'uint32': 4, 'uint64': 8,
|
||||
'float32': 4, 'float64': 8,
|
||||
'boolean': 1, 'bool': 1,
|
||||
'utf8': 50, 'string': 50, 'str': 50,
|
||||
'categorical': 8, 'cat': 8,
|
||||
'date': 4, 'datetime': 8, 'time': 8, 'duration': 8,
|
||||
}
|
||||
bytes_per_row += size_map.get(numeric_part, 8)
|
||||
memory_mb = (bytes_per_row * total_row_count) / (1024 * 1024)
|
||||
|
||||
# Apply head limit for preview (keeps LazyFrame lazy — rows are limited on collect)
|
||||
if head is not None:
|
||||
merged_lf = merged_lf.head(head)
|
||||
total_row_count = min(total_row_count, head)
|
||||
|
||||
return merged_lf, merged_schema, total_row_count, len(paths), memory_mb
|
||||
@@ -0,0 +1,333 @@
|
||||
"""Data profiling for LLM decision-making.
|
||||
|
||||
Produces column-level statistics and a correlation matrix, with built-in
|
||||
context‑window truncation so the output stays under a configurable budget
|
||||
(default 16 KB).
|
||||
"""
|
||||
import json
|
||||
from typing import Any, Optional
|
||||
|
||||
import polars as pl
|
||||
import numpy as np
|
||||
|
||||
|
||||
MAX_PROFILE_SIZE_KB = 16
|
||||
"""Maximum serialised profile size before truncation kicks in."""
|
||||
|
||||
MAX_SAMPLE_ROWS = 5000
|
||||
"""Maximum rows to collect for profiling."""
|
||||
|
||||
MAX_COLUMNS_FULL = 100
|
||||
"""Beyond this column count the profile is always truncated."""
|
||||
|
||||
MAX_CORRELATION_COLS = 20
|
||||
"""Maximum numeric columns to include in the correlation matrix."""
|
||||
|
||||
_ENTITY_KEYWORDS = frozenset({
|
||||
'ip', 'host', 'domain', 'user', 'client', 'src_ip', 'dst_ip',
|
||||
'source', 'destination', 'address', 'entity', 'name', 'id',
|
||||
'username', 'hostname', 'mac', 'email', 'phone', 'session',
|
||||
})
|
||||
|
||||
|
||||
def _is_float_dtype(dtype: pl.DataType) -> bool:
|
||||
return dtype in (pl.Float32, pl.Float64)
|
||||
|
||||
|
||||
def _is_int_dtype(dtype: pl.DataType) -> bool:
|
||||
return dtype in (pl.Int8, pl.Int16, pl.Int32, pl.Int64,
|
||||
pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64)
|
||||
|
||||
|
||||
def _is_numeric_dtype(dtype: pl.DataType) -> bool:
|
||||
return _is_float_dtype(dtype) or _is_int_dtype(dtype)
|
||||
|
||||
|
||||
def _is_string_dtype(dtype: pl.DataType) -> bool:
|
||||
return dtype in (pl.Utf8, pl.String, pl.Categorical)
|
||||
|
||||
|
||||
def _is_temporal_dtype(dtype: pl.DataType) -> bool:
|
||||
return dtype in (pl.Date, pl.Datetime, pl.Time, pl.Duration)
|
||||
|
||||
|
||||
# ── column statistics ───────────────────────────────────────────────────
|
||||
|
||||
def _compute_column_stats(series: pl.Series) -> dict:
|
||||
"""Return a dict of statistics for a single column, based on its dtype.
|
||||
|
||||
Numeric columns get min/max/mean/std/median/percentiles.
|
||||
String columns get unique count, missing count, and top-5 frequencies.
|
||||
Boolean columns get true/false/null counts.
|
||||
Temporal columns get min/max/null_count.
|
||||
"""
|
||||
dtype = series.dtype
|
||||
stats: dict = {
|
||||
'dtype': str(dtype),
|
||||
'null_count': int(series.null_count()),
|
||||
'non_null_count': int(len(series) - series.null_count()),
|
||||
}
|
||||
|
||||
if _is_numeric_dtype(dtype):
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) > 0:
|
||||
stats.update({
|
||||
'min': _safe_float(non_null.min()),
|
||||
'max': _safe_float(non_null.max()),
|
||||
'mean': _safe_float(non_null.mean()),
|
||||
'std': _safe_float(non_null.std()),
|
||||
'median': _safe_float(non_null.median()),
|
||||
'q25': _safe_float(non_null.quantile(0.25)),
|
||||
'q75': _safe_float(non_null.quantile(0.75)),
|
||||
'unique_count': int(non_null.n_unique()),
|
||||
})
|
||||
else:
|
||||
stats.update({
|
||||
'min': None, 'max': None, 'mean': None, 'std': None,
|
||||
'median': None, 'q25': None, 'q75': None, 'unique_count': 0,
|
||||
})
|
||||
stats['type_category'] = 'numeric'
|
||||
|
||||
elif _is_string_dtype(dtype):
|
||||
non_null = series.drop_nulls()
|
||||
stats['unique_count'] = int(non_null.n_unique()) if len(non_null) > 0 else 0
|
||||
stats['type_category'] = 'categorical'
|
||||
|
||||
# Top-5 value counts
|
||||
if len(non_null) > 0:
|
||||
vc = non_null.value_counts().sort('count', descending=True).head(5)
|
||||
# Polars >=1.0 value_counts() uses original col name, not 'index'
|
||||
val_col = vc.columns[0]
|
||||
stats['top_values'] = [
|
||||
{'value': str(row[val_col]), 'count': int(row['count'])}
|
||||
for row in vc.iter_rows(named=True)
|
||||
]
|
||||
else:
|
||||
stats['top_values'] = []
|
||||
|
||||
# Detect entity column
|
||||
stats['is_entity_candidate'] = str(series.name).lower() in _ENTITY_KEYWORDS
|
||||
|
||||
elif dtype == pl.Boolean:
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) > 0:
|
||||
true_count = int(non_null.sum())
|
||||
stats.update({
|
||||
'true_count': true_count,
|
||||
'false_count': int(len(non_null)) - true_count,
|
||||
})
|
||||
else:
|
||||
stats.update({'true_count': 0, 'false_count': 0})
|
||||
stats['type_category'] = 'boolean'
|
||||
|
||||
else:
|
||||
stats['type_category'] = 'other'
|
||||
stats['unique_count'] = int(series.drop_nulls().n_unique())
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def _safe_float(val: Any) -> Optional[float]:
|
||||
"""Convert *val* to float or return None on failure."""
|
||||
if val is None:
|
||||
return None
|
||||
try:
|
||||
f = float(val)
|
||||
return None if np.isnan(f) or np.isinf(f) else round(f, 6)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
# ── correlation matrix ──────────────────────────────────────────────────
|
||||
|
||||
def _compute_correlation_matrix(
|
||||
df: pl.DataFrame,
|
||||
top_n: int = MAX_CORRELATION_COLS,
|
||||
) -> Optional[list[dict]]:
|
||||
"""Pearson correlation for numeric columns.
|
||||
|
||||
Returns a list of ``{column_i, column_j, correlation}`` dicts for the
|
||||
top *top_n* columns by mean absolute correlation, sorted descending by
|
||||
absolute value. Returns *None* when fewer than 2 numeric columns exist.
|
||||
"""
|
||||
numeric_cols = [
|
||||
col for col in df.columns
|
||||
if _is_numeric_dtype(df[col].dtype)
|
||||
]
|
||||
if len(numeric_cols) < 2:
|
||||
return None
|
||||
|
||||
# Limit to top_n numeric columns (pick by variance — highest first)
|
||||
if len(numeric_cols) > top_n:
|
||||
variances = []
|
||||
for col in numeric_cols:
|
||||
v = df[col].drop_nulls().std()
|
||||
variances.append((col, 0.0 if v is None else float(v)))
|
||||
variances.sort(key=lambda x: x[1], reverse=True)
|
||||
numeric_cols = [c for c, _ in variances[:top_n]]
|
||||
|
||||
# Build correlation matrix
|
||||
try:
|
||||
import pandas as pd
|
||||
pdf = df.select(numeric_cols).to_pandas()
|
||||
corr = pdf.corr(method='pearson')
|
||||
pairs: list[dict] = []
|
||||
for i, col_i in enumerate(numeric_cols):
|
||||
for col_j in numeric_cols[i + 1:]:
|
||||
val = corr.loc[col_i, col_j]
|
||||
if np.isnan(val):
|
||||
continue
|
||||
pairs.append({
|
||||
'column_i': col_i,
|
||||
'column_j': col_j,
|
||||
'correlation': round(float(val), 4),
|
||||
})
|
||||
except ImportError:
|
||||
# Pandas not available — compute simple correlation matrix via numpy
|
||||
data = df.select(numeric_cols).to_numpy()
|
||||
data = np.nan_to_num(data, nan=0.0)
|
||||
corr_np = np.corrcoef(data.T)
|
||||
pairs = []
|
||||
for i in range(len(numeric_cols)):
|
||||
for j in range(i + 1, len(numeric_cols)):
|
||||
val = corr_np[i, j]
|
||||
if np.isnan(val):
|
||||
continue
|
||||
pairs.append({
|
||||
'column_i': numeric_cols[i],
|
||||
'column_j': numeric_cols[j],
|
||||
'correlation': round(float(val), 4),
|
||||
})
|
||||
|
||||
# Sort by absolute correlation descending
|
||||
pairs.sort(key=lambda x: abs(x['correlation']), reverse=True)
|
||||
return pairs[:top_n * 2] # cap output
|
||||
|
||||
|
||||
# ── truncation ──────────────────────────────────────────────────────────
|
||||
|
||||
def _estimate_json_size(obj: Any) -> int:
|
||||
"""Return the length of *obj* when serialised to JSON."""
|
||||
return len(json.dumps(obj, default=str, ensure_ascii=False))
|
||||
|
||||
|
||||
def truncate_profile(
|
||||
profile: dict,
|
||||
max_kb: int = MAX_PROFILE_SIZE_KB,
|
||||
) -> dict:
|
||||
"""Remove low-value columns from *profile* until it fits in *max_kb*.
|
||||
|
||||
Sets ``truncated=True`` and adds ``omitted_columns`` if truncation
|
||||
occurred. Columns with the least variance (or highest null rate) are
|
||||
removed first.
|
||||
"""
|
||||
max_bytes = max_kb * 1024
|
||||
if _estimate_json_size(profile) <= max_bytes:
|
||||
profile['truncated'] = False
|
||||
profile.setdefault('omitted_columns', [])
|
||||
return profile
|
||||
|
||||
column_stats = profile.get('column_stats', {})
|
||||
if not column_stats:
|
||||
profile['truncated'] = True
|
||||
return profile
|
||||
|
||||
# Score columns: low variance / high null rate → remove first
|
||||
scored: list[tuple[str, float]] = []
|
||||
for col_name, stats in column_stats.items():
|
||||
null_rate = stats.get('null_count', 0) / max(stats.get('non_null_count', 1), 1)
|
||||
# Numeric columns get a variance estimate
|
||||
std = stats.get('std')
|
||||
variance_score = float(std) if std is not None else 0.0
|
||||
# Remove low-variance OR high-null columns
|
||||
removal_score = -variance_score + null_rate * 100
|
||||
scored.append((col_name, removal_score))
|
||||
|
||||
scored.sort(key=lambda x: x[1]) # most valuable first
|
||||
|
||||
kept_columns = []
|
||||
omitted_columns = []
|
||||
for col_name, _ in scored:
|
||||
test_profile = dict(profile)
|
||||
test_profile['column_stats'] = {
|
||||
c: column_stats[c] for c in kept_columns + [col_name]
|
||||
}
|
||||
if _estimate_json_size(test_profile) <= max_bytes:
|
||||
kept_columns.append(col_name)
|
||||
else:
|
||||
omitted_columns.append(col_name)
|
||||
|
||||
profile['column_stats'] = {
|
||||
c: column_stats[c] for c in kept_columns
|
||||
}
|
||||
profile['truncated'] = True
|
||||
profile['omitted_columns'] = omitted_columns
|
||||
profile['truncation_note'] = (
|
||||
f"{len(omitted_columns)} of {len(omitted_columns) + len(kept_columns)} "
|
||||
f"columns omitted to stay under {max_kb} KB budget"
|
||||
)
|
||||
return profile
|
||||
|
||||
|
||||
# ── main profiling entry point ──────────────────────────────────────────
|
||||
|
||||
def profile_dataset(
|
||||
lf: pl.LazyFrame,
|
||||
sample_size: int = 1000,
|
||||
columns: Optional[list[str]] = None,
|
||||
max_kb: int = MAX_PROFILE_SIZE_KB,
|
||||
max_correlation_cols: int = MAX_CORRELATION_COLS,
|
||||
) -> dict:
|
||||
"""Compute column-level statistics and a correlation matrix.
|
||||
|
||||
Args:
|
||||
lf: LazyFrame to profile.
|
||||
sample_size: Maximum rows to collect (sampled if larger).
|
||||
columns: Subset of columns to profile (``None`` = all).
|
||||
max_kb: Truncation budget in KB.
|
||||
max_correlation_cols: Max numeric columns for correlation.
|
||||
|
||||
Returns:
|
||||
A dict with keys ``column_stats``, ``correlation_matrix``,
|
||||
``truncated``, ``omitted_columns``, ``total_columns``,
|
||||
``profiled_rows``, and ``sample_ratio``.
|
||||
"""
|
||||
# Collect (possibly sampled) dataframe
|
||||
df = lf.collect(streaming=True)
|
||||
total_rows = len(df)
|
||||
sample_ratio = 1.0
|
||||
|
||||
if total_rows > sample_size:
|
||||
df = df.sample(n=sample_size, seed=42)
|
||||
sample_ratio = sample_size / total_rows
|
||||
|
||||
# Determine columns to profile
|
||||
if columns:
|
||||
profile_columns = [c for c in columns if c in df.columns]
|
||||
else:
|
||||
profile_columns = list(df.columns)
|
||||
|
||||
if len(profile_columns) > MAX_COLUMNS_FULL:
|
||||
profile_columns = profile_columns[:MAX_COLUMNS_FULL]
|
||||
|
||||
# Per-column stats
|
||||
column_stats: dict = {}
|
||||
for col_name in profile_columns:
|
||||
column_stats[col_name] = _compute_column_stats(df[col_name])
|
||||
|
||||
# Correlation matrix
|
||||
correlation_matrix = _compute_correlation_matrix(df, top_n=max_correlation_cols)
|
||||
|
||||
result = {
|
||||
'column_stats': column_stats,
|
||||
'correlation_matrix': correlation_matrix,
|
||||
'truncated': False,
|
||||
'omitted_columns': [],
|
||||
'total_columns': len(df.columns),
|
||||
'profiled_columns': len(profile_columns),
|
||||
'profiled_rows': len(df),
|
||||
'total_rows': total_rows,
|
||||
'sample_ratio': round(sample_ratio, 4),
|
||||
}
|
||||
|
||||
return truncate_profile(result, max_kb=max_kb)
|
||||
@@ -0,0 +1,392 @@
|
||||
"""
|
||||
数据预处理校验阶段。在 load_csv_directory 之后、build_entity_profiles 之前自动运行。
|
||||
不阻塞流程，只记录 warning 到日志。
|
||||
"""
|
||||
import logging
|
||||
import json
|
||||
from typing import Any, Optional
|
||||
|
||||
import polars as pl
|
||||
import numpy as np
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.type_classifier import classify_column, DataType
|
||||
from analysis.ip_clustering import ip_to_int
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Mapping from Polars dtype strings to DataType for schema comparison
|
||||
_DTYPE_TO_DATATYPE: dict[str, DataType] = {
|
||||
'Int8': DataType.INT, 'Int16': DataType.INT, 'Int32': DataType.INT, 'Int64': DataType.INT,
|
||||
'UInt8': DataType.INT, 'UInt16': DataType.INT, 'UInt32': DataType.INT, 'UInt64': DataType.INT,
|
||||
'Float32': DataType.FLOAT, 'Float64': DataType.FLOAT,
|
||||
'Boolean': DataType.BOOL_ENUM, 'Bool': DataType.BOOL_ENUM,
|
||||
'String': DataType.STRING, 'Utf8': DataType.STRING, 'Categorical': DataType.STRING,
|
||||
'Date': DataType.TIMESTAMP, 'Datetime': DataType.TIMESTAMP,
|
||||
'Time': DataType.TIMESTAMP, 'Duration': DataType.TIMESTAMP,
|
||||
}
|
||||
|
||||
_MAX_OUTLIER_SAMPLE = 1000
|
||||
"""Max rows to sample for Z-score outlier detection."""
|
||||
|
||||
_MAX_IPV4_INVALID_SAMPLES = 10
|
||||
"""Max invalid IPv4 samples to include in the report."""
|
||||
|
||||
|
||||
def _dtype_to_datatype(dtype_str: str) -> Optional[DataType]:
|
||||
"""Map a Polars dtype string (e.g. ``'String'``, ``'Int64'``) to :class:`DataType`.
|
||||
|
||||
Strips any type parameters (e.g. ``Datetime(time_unit='us', time_zone=None)``)
|
||||
before lookup.
|
||||
"""
|
||||
base = dtype_str.split('(')[0].strip()
|
||||
return _DTYPE_TO_DATATYPE.get(base)
|
||||
|
||||
|
||||
def _safe_float(val: Any) -> Optional[float]:
|
||||
"""Convert *val* to float or return *None* on failure."""
|
||||
if val is None:
|
||||
return None
|
||||
try:
|
||||
f = float(val)
|
||||
return None if np.isnan(f) or np.isinf(f) else round(f, 6)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _is_acceptable_mismatch(schema_type: DataType, inferred_type: DataType) -> bool:
|
||||
"""Return *True* if a schema vs. inferred type mismatch is benign.
|
||||
|
||||
Acceptable mismatches:
|
||||
- Schema says ``STRING`` but classifier infers a more specific type
|
||||
(``IPv4``, ``URL``, ``HEX``, etc.) — this is expected because
|
||||
``data_loader`` only sees raw Polars dtypes while ``classify_column``
|
||||
applies value-based heuristics.
|
||||
- ``INT`` ↔ ``FLOAT`` — Polars can cast between numeric types freely.
|
||||
- ``ENUM`` ↔ ``BOOL_ENUM`` — both are low-cardinality categorical types.
|
||||
"""
|
||||
if schema_type == DataType.STRING:
|
||||
return True
|
||||
if {schema_type, inferred_type} == {DataType.INT, DataType.FLOAT}:
|
||||
return True
|
||||
if {schema_type, inferred_type} == {DataType.ENUM, DataType.BOOL_ENUM}:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate(dataset_id: str, strict: bool = False) -> dict:
|
||||
"""运行完整校验，返回 JSON 报告。
|
||||
|
||||
校验项
|
||||
------
|
||||
1. **列类型** — 用 :func:`~analysis.type_classifier.classify_column`
|
||||
判定每列类型，与 *schema* 对比矛盾。
|
||||
2. **缺失率** — 每列 ``null_count / len``，>50% 标记为高风险。
|
||||
3. **异常值** — 数值列的 ``min`` / ``max`` / ``std``，Z-score > 5 比例。
|
||||
4. **枚举分布** — ``ENUM`` / ``BOOL_ENUM`` 的值频率，特殊符号
|
||||
（``+``、空白）标记。
|
||||
5. **IPv4 有效性** — 对 ``IPv4`` 类型列的每个值调用
|
||||
:func:`~analysis.ip_clustering.ip_to_int` 验证。
|
||||
|
||||
输出
|
||||
----
|
||||
每条校验结果通过 :data:`logger` 以 ``[VALIDATE]`` 标记写入
|
||||
``info`` 或 ``warning`` 级别。
|
||||
|
||||
参数
|
||||
----------
|
||||
dataset_id:
|
||||
SessionStore 中的数据集 ID。
|
||||
strict:
|
||||
为 *True* 时遇到高风险项（缺失率 >50%、无效 IPv4 比例 >50%）
|
||||
抛出 :class:`RuntimeError`。默认为 *False*（仅记录 warning）。
|
||||
|
||||
返回
|
||||
-------
|
||||
dict
|
||||
包含 ``valid``、``columns``、``warnings``、``risks`` 等键的
|
||||
校验报告。
|
||||
"""
|
||||
store = SessionStore()
|
||||
entry = store.get_dataset(dataset_id)
|
||||
if entry is None:
|
||||
msg = f"Dataset '{dataset_id}' not found in SessionStore"
|
||||
logger.error(f'[VALIDATE] {msg}')
|
||||
return {'valid': False, 'columns': {}, 'warnings': [], 'risks': [msg]}
|
||||
|
||||
lf: pl.LazyFrame = entry['lazyframe']
|
||||
schema: dict = entry.get('schema', {})
|
||||
|
||||
# Materialise the LazyFrame
|
||||
try:
|
||||
df = lf.collect(streaming=True)
|
||||
except Exception as e:
|
||||
msg = f"Failed to collect LazyFrame for dataset '{dataset_id}': {e}"
|
||||
logger.error(f'[VALIDATE] {msg}')
|
||||
return {'valid': False, 'columns': {}, 'warnings': [], 'risks': [msg]}
|
||||
|
||||
total_rows = len(df)
|
||||
logger.info(f'[VALIDATE] dataset={dataset_id} rows={total_rows} columns={len(df.columns)}')
|
||||
|
||||
columns_report: dict[str, dict] = {}
|
||||
all_warnings: list[str] = []
|
||||
all_risks: list[str] = []
|
||||
|
||||
for col_name in df.columns:
|
||||
series = df[col_name]
|
||||
col_info: dict[str, Any] = {
|
||||
'name': col_name,
|
||||
'dtype': str(series.dtype),
|
||||
}
|
||||
col_warnings: list[str] = []
|
||||
|
||||
# ── 1. Column type classification ────────────────────────────────
|
||||
inferred_type = classify_column(col_name, series, config_type=None)
|
||||
col_info['inferred_type'] = inferred_type.name
|
||||
|
||||
schema_dtype_str = schema.get(col_name)
|
||||
if schema_dtype_str:
|
||||
schema_type = _dtype_to_datatype(schema_dtype_str)
|
||||
col_info['schema_type'] = schema_type.name if schema_type else schema_dtype_str
|
||||
if schema_type and schema_type != inferred_type:
|
||||
if not _is_acceptable_mismatch(schema_type, inferred_type):
|
||||
msg = (
|
||||
f"Column '{col_name}': schema type ({schema_type.name}) "
|
||||
f"differs from inferred type ({inferred_type.name})"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
all_warnings.append(msg)
|
||||
else:
|
||||
col_info['schema_type'] = None
|
||||
|
||||
# ── 2. Missing rate ──────────────────────────────────────────────
|
||||
null_count = int(series.null_count())
|
||||
null_rate = null_count / total_rows if total_rows > 0 else 0.0
|
||||
col_info['null_count'] = null_count
|
||||
col_info['null_rate'] = round(null_rate, 4)
|
||||
|
||||
if null_rate > 0.5:
|
||||
msg = (
|
||||
f"Column '{col_name}': null rate {null_rate:.1%} "
|
||||
f"exceeds 50% (high risk)"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
all_risks.append(msg)
|
||||
elif null_rate > 0.3:
|
||||
msg = (
|
||||
f"Column '{col_name}': null rate {null_rate:.1%} "
|
||||
f"exceeds 30%"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
all_warnings.append(msg)
|
||||
|
||||
# ── 3. Outlier detection (numeric columns) ───────────────────────
|
||||
if inferred_type in (DataType.INT, DataType.FLOAT):
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) > 1:
|
||||
try:
|
||||
col_info['min'] = _safe_float(non_null.min())
|
||||
col_info['max'] = _safe_float(non_null.max())
|
||||
col_info['std'] = _safe_float(non_null.std())
|
||||
|
||||
# Sample for Z-score computation
|
||||
if len(non_null) > _MAX_OUTLIER_SAMPLE:
|
||||
sample = non_null.sample(n=_MAX_OUTLIER_SAMPLE, seed=42)
|
||||
else:
|
||||
sample = non_null
|
||||
|
||||
num_values = sample.cast(pl.Float64).to_numpy()
|
||||
mean = float(np.nanmean(num_values))
|
||||
std = float(np.nanstd(num_values))
|
||||
if std > 0:
|
||||
z_scores = np.abs((num_values - mean) / std)
|
||||
outlier_ratio = float(np.mean(z_scores > 5))
|
||||
col_info['z_score_gt_5_ratio'] = round(outlier_ratio, 4)
|
||||
if outlier_ratio > 0.05:
|
||||
msg = (
|
||||
f"Column '{col_name}': {outlier_ratio:.1%} "
|
||||
f"values have |Z-score| > 5 (high outlier ratio)"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
all_warnings.append(msg)
|
||||
else:
|
||||
col_info['z_score_gt_5_ratio'] = 0.0
|
||||
except Exception as e:
|
||||
col_info['numeric_error'] = str(e)
|
||||
col_info['z_score_gt_5_ratio'] = 0.0
|
||||
else:
|
||||
col_info['z_score_gt_5_ratio'] = 0.0
|
||||
|
||||
# ── 4. Enum distribution ─────────────────────────────────────────
|
||||
if inferred_type in (DataType.ENUM, DataType.BOOL_ENUM):
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) > 0:
|
||||
vc = non_null.value_counts()
|
||||
val_col = vc.columns[0]
|
||||
cnt_col = vc.columns[1]
|
||||
total_non_null = len(non_null)
|
||||
distribution: dict[str, dict] = {}
|
||||
for row in vc.iter_rows(named=True):
|
||||
val = str(row[val_col])
|
||||
count = int(row[cnt_col])
|
||||
distribution[val] = {
|
||||
'count': count,
|
||||
'ratio': round(count / total_non_null, 4),
|
||||
}
|
||||
col_info['distribution'] = distribution
|
||||
|
||||
# Flag special symbols (empty string, whitespace-only, +)
|
||||
for val, info in distribution.items():
|
||||
stripped = val.strip()
|
||||
if stripped == '' or stripped == '+':
|
||||
msg = (
|
||||
f"Column '{col_name}': "
|
||||
f"value {repr(val)} appears {info['count']} times"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
all_warnings.append(msg)
|
||||
break # one warning per column
|
||||
|
||||
# ── 5. IPv4 validation ───────────────────────────────────────────
|
||||
if inferred_type == DataType.IPv4:
|
||||
non_null = series.drop_nulls()
|
||||
total_valid = 0
|
||||
total_invalid = 0
|
||||
invalid_samples: list[str] = []
|
||||
for v in non_null:
|
||||
ip_str = str(v).strip()
|
||||
if ip_to_int(ip_str) is not None:
|
||||
total_valid += 1
|
||||
else:
|
||||
total_invalid += 1
|
||||
if len(invalid_samples) < _MAX_IPV4_INVALID_SAMPLES:
|
||||
invalid_samples.append(ip_str)
|
||||
|
||||
col_info['ipv4_valid_count'] = total_valid
|
||||
col_info['ipv4_invalid_count'] = total_invalid
|
||||
col_info['ipv4_invalid_samples'] = invalid_samples
|
||||
|
||||
if total_invalid > 0:
|
||||
total_ipv4 = total_valid + total_invalid
|
||||
invalid_ratio = total_invalid / total_ipv4 if total_ipv4 > 0 else 1.0
|
||||
sample_str = str(invalid_samples[:5])
|
||||
msg = (
|
||||
f"Column '{col_name}': {total_invalid}/{total_ipv4} "
|
||||
f"({invalid_ratio:.1%}) values are invalid IPv4. "
|
||||
f"Samples: {sample_str}"
|
||||
)
|
||||
col_warnings.append(msg)
|
||||
if invalid_ratio > 0.5:
|
||||
all_risks.append(msg)
|
||||
else:
|
||||
all_warnings.append(msg)
|
||||
|
||||
col_info['warnings'] = col_warnings
|
||||
columns_report[col_name] = col_info
|
||||
|
||||
# ── Log all findings ─────────────────────────────────────────────────
|
||||
for col_name, col_info in columns_report.items():
|
||||
for w in col_info.get('warnings', []):
|
||||
logger.warning(f'[VALIDATE] {w}')
|
||||
|
||||
report_data: dict[str, Any] = {
|
||||
'valid': len(all_risks) == 0,
|
||||
'dataset_id': dataset_id,
|
||||
'total_rows': total_rows,
|
||||
'total_columns': len(df.columns),
|
||||
'columns': columns_report,
|
||||
'warnings': all_warnings,
|
||||
'risks': all_risks,
|
||||
}
|
||||
|
||||
n_warn = len(all_warnings)
|
||||
n_risk = len(all_risks)
|
||||
logger.info(
|
||||
f'[VALIDATE] Complete: valid={report_data["valid"]} '
|
||||
f'warnings={n_warn} risks={n_risk}'
|
||||
)
|
||||
|
||||
if strict and all_risks:
|
||||
raise RuntimeError(
|
||||
f'Validation strict mode: {len(all_risks)} risk(s) found:\n' +
|
||||
'\n'.join(f' - {r}' for r in all_risks)
|
||||
)
|
||||
|
||||
return report_data
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formatted report
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def report(dataset_id: str) -> str:
|
||||
"""返回格式化的校验报告字符串。
|
||||
|
||||
参数
|
||||
----------
|
||||
dataset_id:
|
||||
SessionStore 中的数据集 ID。
|
||||
|
||||
返回
|
||||
-------
|
||||
str
|
||||
人类可读的格式化报告。
|
||||
"""
|
||||
result = validate(dataset_id, strict=False)
|
||||
|
||||
lines: list[str] = []
|
||||
lines.append('=' * 60)
|
||||
lines.append(f'Data Validation Report — dataset: {dataset_id}')
|
||||
lines.append('=' * 60)
|
||||
lines.append(f' Valid: {result["valid"]}')
|
||||
lines.append(f' Total rows: {result.get("total_rows", "?")}')
|
||||
lines.append(f' Total columns: {result.get("total_columns", "?")}')
|
||||
lines.append(f' Warnings: {len(result.get("warnings", []))}')
|
||||
lines.append(f' Risks: {len(result.get("risks", []))}')
|
||||
lines.append('')
|
||||
|
||||
columns = result.get('columns', {})
|
||||
for col_name, col_info in sorted(columns.items()):
|
||||
lines.append(f' --- {col_name} ---')
|
||||
lines.append(f' dtype: {col_info.get("dtype", "?")}')
|
||||
lines.append(f' inferred: {col_info.get("inferred_type", "?")}')
|
||||
lines.append(f' null_rate: {col_info.get("null_rate", "?")}')
|
||||
if 'min' in col_info:
|
||||
cmin = col_info.get('min')
|
||||
cmax = col_info.get('max')
|
||||
lines.append(f' range: [{cmin}, {cmax}]')
|
||||
if 'z_score_gt_5_ratio' in col_info:
|
||||
lines.append(f' |Z|>5 ratio: {col_info["z_score_gt_5_ratio"]}')
|
||||
if 'distribution' in col_info:
|
||||
dist = col_info['distribution']
|
||||
top5 = sorted(dist.items(), key=lambda kv: kv[1]['count'], reverse=True)[:5]
|
||||
top5_str = ', '.join(f'{k}={v["count"]}' for k, v in top5)
|
||||
lines.append(f' top values: {top5_str}')
|
||||
if 'ipv4_invalid_count' in col_info and col_info['ipv4_invalid_count'] > 0:
|
||||
lines.append(
|
||||
f' invalid IPv4: {col_info["ipv4_invalid_count"]} '
|
||||
f'(samples: {col_info.get("ipv4_invalid_samples", [])})'
|
||||
)
|
||||
for w in col_info.get('warnings', []):
|
||||
lines.append(f' ! {w}')
|
||||
lines.append('')
|
||||
|
||||
if result.get('warnings'):
|
||||
lines.append('Warnings:')
|
||||
for w in result['warnings']:
|
||||
lines.append(f' - {w}')
|
||||
lines.append('')
|
||||
if result.get('risks'):
|
||||
lines.append('Risks:')
|
||||
for r in result['risks']:
|
||||
lines.append(f' # {r}')
|
||||
lines.append('')
|
||||
|
||||
lines.append('=' * 60)
|
||||
return '\n'.join(lines)
|
||||
@@ -0,0 +1,412 @@
|
||||
"""Aggregate flow-level data into entity-level profiles.
|
||||
|
||||
The main entry point is :func:`aggregate_by_entity`, which groups a Polars
|
||||
:class:`LazyFrame` by one or more entity columns (e.g. ``src_ip``, ``sni``,
|
||||
``user``) and computes per-entity aggregates across flow, destination, TLS,
|
||||
protocol, and time dimensions.
|
||||
|
||||
Only features whose source columns exist in the provided schema are computed,
|
||||
so the module works with any CSV schema without assumptions.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
import polars as pl
|
||||
|
||||
from .type_classifier import DataType
|
||||
|
||||
_agg_logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Keyword sets for schema-independent column discovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Flow stats — user exact column names only
|
||||
_BYTES_SENT_KEYWORDS = ('8ack',)
|
||||
_BYTES_REV_KEYWORDS = () # no user column for reverse bytes
|
||||
_DURATION_KEYWORDS = ('4dur',)
|
||||
_PACKETS_KEYWORDS = ('8ppk',)
|
||||
_PACKETS_REV_KEYWORDS = () # no user column for reverse packets
|
||||
|
||||
# Destination diversity — user exact column names only
|
||||
_DST_IP_KEYWORDS = (':ips', ':ipd')
|
||||
_DST_PORT_KEYWORDS = (':prs', ':prd')
|
||||
_DST_NET_KEYWORDS = () # no user column for destination network
|
||||
|
||||
# TLS features — user exact column names only
|
||||
_CIPHER_KEYWORDS = ('cipher_suite', '0cph')
|
||||
_TLS_VERSION_KEYWORDS = ('0ver',)
|
||||
_SNI_KEYWORDS = ('snam', 'cnam')
|
||||
|
||||
# Protocol features — user exact column names only
|
||||
_PROTO_KEYWORDS = ('1ipp',)
|
||||
_TCP_FLAGS_KEYWORDS = () # no user column for TCP flags
|
||||
|
||||
# Geo-location features — user exact column names only
|
||||
_LAT_KEYWORDS = ('latd',)
|
||||
_LON_KEYWORDS = ('lond',)
|
||||
|
||||
# Time patterns — user exact column names only
|
||||
_TIMESTAMP_KEYWORDS = ('time', 'timestamp', '8dbd')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IP subnet helpers for prefix-based aggregation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def ip_to_subnet(ip_str: str, mask: int) -> str:
|
||||
"""Convert ``10.0.1.15/24`` → ``10.0.1.0/24``, ``10.0.1.250/28`` →
|
||||
``10.0.1.240/28``.
|
||||
|
||||
Invalid IPs / ``None`` / empty string are returned unchanged.
|
||||
Supports ``/24`` (0–255 last octet) and ``/28`` (240–255 in steps of 16)
|
||||
masks.
|
||||
"""
|
||||
if not ip_str or not isinstance(ip_str, str):
|
||||
return ip_str
|
||||
try:
|
||||
parts = [int(o) for o in ip_str.strip().split('.')]
|
||||
if len(parts) != 4 or any(p < 0 or p > 255 for p in parts):
|
||||
return ip_str
|
||||
ip_int = (parts[0] << 24) + (parts[1] << 16) + (parts[2] << 8) + parts[3]
|
||||
subnet_int = ip_int & (0xFFFFFFFF << (32 - mask))
|
||||
return (
|
||||
f"{(subnet_int >> 24) & 255}."
|
||||
f"{(subnet_int >> 16) & 255}."
|
||||
f"{(subnet_int >> 8) & 255}."
|
||||
f"{subnet_int & 255}/{mask}"
|
||||
)
|
||||
except (ValueError, IndexError, TypeError):
|
||||
return ip_str
|
||||
|
||||
|
||||
def apply_subnet_aggregation(
|
||||
lf: pl.LazyFrame,
|
||||
ip_columns: list[str],
|
||||
subnet_masks: list[int],
|
||||
) -> pl.LazyFrame:
|
||||
"""Apply subnet masking to IP columns in the LazyFrame.
|
||||
|
||||
Each IP column is transformed to its subnet prefix for each mask.
|
||||
If multiple masks exist, the first matching mask is applied
|
||||
(earliest mask in *subnet_masks* wins).
|
||||
"""
|
||||
if not subnet_masks or not ip_columns:
|
||||
return lf
|
||||
schema = lf.collect_schema().names()
|
||||
for mask in subnet_masks:
|
||||
mask_exprs = []
|
||||
for col in ip_columns:
|
||||
if col in schema:
|
||||
mask_exprs.append(
|
||||
pl.col(col)
|
||||
.map_batches(
|
||||
lambda s, m=mask: s.cast(pl.Utf8).map_batches(
|
||||
lambda x, mm=m: pl.Series(
|
||||
[ip_to_subnet(v, mm) for v in x.to_list()]
|
||||
),
|
||||
return_dtype=pl.Utf8,
|
||||
),
|
||||
return_dtype=pl.Utf8,
|
||||
)
|
||||
.alias(col)
|
||||
)
|
||||
if mask_exprs:
|
||||
lf = lf.with_columns(mask_exprs)
|
||||
return lf
|
||||
|
||||
|
||||
def _find_column(schema: dict[str, str], keywords: tuple[str, ...]) -> Optional[str]:
|
||||
"""Return the first column in *schema* whose lowercased name matches
|
||||
any of *keywords* (exact or ``_``-delimited match)."""
|
||||
for col in schema:
|
||||
normalised = col.lower().lstrip(':+').replace('-', '_').replace('.', '_').replace(' ', '_')
|
||||
for kw in sorted(keywords):
|
||||
kw_s = kw.lower().lstrip(':+')
|
||||
if kw_s == normalised or normalised.endswith(f'_{kw_s}') or normalised.startswith(f'{kw_s}_'):
|
||||
return col
|
||||
return None
|
||||
|
||||
|
||||
def _timestamp_col(schema: dict[str, str]) -> Optional[str]:
|
||||
"""Detect a timestamp column -- user exact column names only."""
|
||||
for preference in ('timestamp', 'time', '8dbd'):
|
||||
if preference in schema:
|
||||
return preference
|
||||
return _find_column(schema, _TIMESTAMP_KEYWORDS)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Aggregation plan builder
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_aggregation_params(
|
||||
schema: dict[str, str],
|
||||
type_map: dict[str, DataType] | None = None,
|
||||
) -> list[tuple[str, pl.Expr]]:
|
||||
"""Build a list of ``(feature_name, pl.Expr)`` pairs for every aggregate
|
||||
that can be computed from the columns present in *schema*.
|
||||
|
||||
Each expression is a column-level aggregation suitable for use inside
|
||||
``pl.LazyFrame.group_by(...).agg(...)``.
|
||||
"""
|
||||
aggs: list[tuple[str, pl.Expr]] = []
|
||||
|
||||
# ── Flow stats ───────────────────────────────────────────────────────
|
||||
# flow_count: always computed via count()
|
||||
aggs.append(('flow_count', pl.len()))
|
||||
|
||||
bytes_sent_col = _find_column(schema, _BYTES_SENT_KEYWORDS)
|
||||
if bytes_sent_col:
|
||||
aggs.append(('total_bytes_sent', pl.sum(bytes_sent_col).cast(pl.Int64)))
|
||||
# Also store raw column name for downstream use
|
||||
aggs.append(('_bytes_sent_col', pl.lit(bytes_sent_col)))
|
||||
aggs.append(('_bytes_sent_expr', pl.lit(bytes_sent_col)))
|
||||
else:
|
||||
aggs.append(('total_bytes_sent', pl.lit(None, dtype=pl.Int64)))
|
||||
aggs.append(('_bytes_sent_col', pl.lit('')))
|
||||
aggs.append(('_bytes_sent_expr', pl.lit('')))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=bytes_sent matched={bytes_sent_col} dtype={schema.get(bytes_sent_col, "N/A") if bytes_sent_col else "None"}')
|
||||
|
||||
packets_col = _find_column(schema, _PACKETS_KEYWORDS)
|
||||
if packets_col:
|
||||
aggs.append(('total_packets', pl.sum(packets_col).cast(pl.Int64)))
|
||||
else:
|
||||
aggs.append(('total_packets', pl.lit(None, dtype=pl.Int64)))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=packets matched={packets_col} dtype={schema.get(packets_col, "N/A") if packets_col else "None"}')
|
||||
|
||||
# ── Destination diversity ────────────────────────────────────────────
|
||||
dst_ip_col = _find_column(schema, _DST_IP_KEYWORDS)
|
||||
if dst_ip_col:
|
||||
aggs.append(('unique_dst_ips', pl.n_unique(dst_ip_col)))
|
||||
aggs.append(('_dst_ip_col', pl.lit(dst_ip_col)))
|
||||
else:
|
||||
aggs.append(('unique_dst_ips', pl.lit(None, dtype=pl.UInt32)))
|
||||
aggs.append(('_dst_ip_col', pl.lit('')))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=dst_ip matched={dst_ip_col} dtype={schema.get(dst_ip_col, "N/A") if dst_ip_col else "None"}')
|
||||
|
||||
dst_port_col = _find_column(schema, _DST_PORT_KEYWORDS)
|
||||
if dst_port_col:
|
||||
aggs.append(('unique_dst_ports', pl.n_unique(dst_port_col)))
|
||||
aggs.append(('_dst_port_col', pl.lit(dst_port_col)))
|
||||
else:
|
||||
aggs.append(('unique_dst_ports', pl.lit(None, dtype=pl.UInt32)))
|
||||
aggs.append(('_dst_port_col', pl.lit('')))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=dst_port matched={dst_port_col} dtype={schema.get(dst_port_col, "N/A") if dst_port_col else "None"}')
|
||||
|
||||
# ── Protocol features ────────────────────────────────────────────────
|
||||
proto_col = _find_column(schema, _PROTO_KEYWORDS)
|
||||
if proto_col:
|
||||
aggs.append(('unique_protocols', pl.n_unique(proto_col)))
|
||||
aggs.append(('_proto_col', pl.lit(proto_col)))
|
||||
else:
|
||||
aggs.append(('unique_protocols', pl.lit(None, dtype=pl.UInt32)))
|
||||
aggs.append(('_proto_col', pl.lit('')))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=proto matched={proto_col} dtype={schema.get(proto_col, "N/A") if proto_col else "None"}')
|
||||
|
||||
# ── Enhanced TLS analysis features ────────────────────────────────────
|
||||
# has_sni: count of non-null SNI values
|
||||
sni_col = _find_column(schema, _SNI_KEYWORDS)
|
||||
if sni_col:
|
||||
aggs.append(('has_sni', pl.col(sni_col).is_not_null().sum()))
|
||||
aggs.append(('sni_missing_ratio', pl.col(sni_col).is_null().mean()))
|
||||
aggs.append(('avg_sni_length', pl.col(sni_col).cast(pl.Utf8).str.len_bytes().mean()))
|
||||
aggs.append(('max_sni_length', pl.col(sni_col).cast(pl.Utf8).str.len_bytes().max()))
|
||||
|
||||
# TLS version risk: look for _norm column
|
||||
tls_norm = _find_column(schema, ('0ver_norm',))
|
||||
if not tls_norm:
|
||||
tls_norm = _find_column(schema, ('0ver',))
|
||||
if tls_norm:
|
||||
aggs.append(('tls_modern_ratio',
|
||||
pl.when(pl.col(tls_norm).is_in(['tls_1_3'])).then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('tls_old_ratio',
|
||||
pl.when(pl.col(tls_norm).is_in(['tls_1_0', 'tls_1_1', 'ssl_2_0', 'ssl_3_0'])).then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('unique_tls_versions', pl.n_unique(tls_norm)))
|
||||
|
||||
# Modern cipher ratio (AEAD = modern)
|
||||
cph_norm = _find_column(schema, ('0cph_norm',))
|
||||
if not cph_norm:
|
||||
cph_norm = _find_column(schema, ('cipher_suite_norm',))
|
||||
if cph_norm:
|
||||
aggs.append(('aead_ratio',
|
||||
pl.when(pl.col(cph_norm).str.contains('gcm|chacha20|poly1305')).then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('weak_cipher_ratio',
|
||||
pl.when(pl.col(cph_norm).str.contains('cbc|rc4|null')).then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('ecdhe_ratio',
|
||||
pl.when(pl.col(cph_norm).str.contains('ecdhe')).then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('unique_ciphers', pl.n_unique(cph_norm)))
|
||||
|
||||
# Non-standard port detection (:prd is often String type)
|
||||
prd_col = _find_column(schema, (':prd',))
|
||||
if prd_col:
|
||||
prd_str = pl.col(prd_col).cast(pl.Utf8)
|
||||
aggs.append(('non_standard_port_ratio',
|
||||
pl.when(prd_str != '443').then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
aggs.append(('port_443_ratio',
|
||||
pl.when(prd_str == '443').then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
|
||||
# Bytes per packet analysis
|
||||
bytes_col = _find_column(schema, ('8ack',))
|
||||
pkts_col = _find_column(schema, ('8ppk',))
|
||||
if bytes_col and pkts_col:
|
||||
bpp = pl.col(bytes_col).cast(pl.Float64) / pl.col(pkts_col).cast(pl.Float64).clip(1)
|
||||
aggs.append(('avg_bytes_per_packet', bpp.mean()))
|
||||
aggs.append(('std_bytes_per_packet', bpp.std()))
|
||||
aggs.append(('min_bytes_per_packet', bpp.min()))
|
||||
aggs.append(('max_bytes_per_packet', bpp.max()))
|
||||
|
||||
# Duration analysis
|
||||
dur_col = _find_column(schema, ('4dur',))
|
||||
if dur_col:
|
||||
aggs.append(('avg_duration', pl.col(dur_col).cast(pl.Float64).mean()))
|
||||
aggs.append(('std_duration', pl.col(dur_col).cast(pl.Float64).std()))
|
||||
aggs.append(('max_duration', pl.col(dur_col).cast(pl.Float64).max()))
|
||||
|
||||
# Recoverable/resumed ratio
|
||||
cnrs_norm = _find_column(schema, ('cnrs_norm',))
|
||||
if cnrs_norm:
|
||||
aggs.append(('recoverable_ratio',
|
||||
pl.when(pl.col(cnrs_norm) == 'yes').then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
|
||||
isrs_norm = _find_column(schema, ('isrs_norm',))
|
||||
if isrs_norm:
|
||||
aggs.append(('resumed_ratio',
|
||||
pl.when(pl.col(isrs_norm) == 'yes').then(1)
|
||||
.otherwise(0).cast(pl.Float64).mean()))
|
||||
|
||||
# Country diversity
|
||||
dcnt_col = _find_column(schema, ('dcnt',))
|
||||
if dcnt_col:
|
||||
aggs.append(('countries_visited', pl.n_unique(dcnt_col)))
|
||||
|
||||
# ── Geo-location (latitude / longitude) ────────────────────────────
|
||||
lat_col = _find_column(schema, _LAT_KEYWORDS)
|
||||
if lat_col:
|
||||
dtype_str = schema.get(lat_col, '').lower()
|
||||
if 'str' in dtype_str or 'utf' in dtype_str:
|
||||
lat_expr = pl.mean(pl.col(lat_col).cast(pl.Float64))
|
||||
else:
|
||||
lat_expr = pl.mean(lat_col)
|
||||
aggs.append(('avg_latitude', lat_expr.cast(pl.Float64)))
|
||||
else:
|
||||
aggs.append(('avg_latitude', pl.lit(None, dtype=pl.Float64)))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=lat matched={lat_col} dtype={schema.get(lat_col, "N/A") if lat_col else "None"}')
|
||||
|
||||
lon_col = _find_column(schema, _LON_KEYWORDS)
|
||||
if lon_col:
|
||||
dtype_str = schema.get(lon_col, '').lower()
|
||||
if 'str' in dtype_str or 'utf' in dtype_str:
|
||||
lon_expr = pl.mean(pl.col(lon_col).cast(pl.Float64))
|
||||
else:
|
||||
lon_expr = pl.mean(lon_col)
|
||||
aggs.append(('avg_longitude', lon_expr.cast(pl.Float64)))
|
||||
else:
|
||||
aggs.append(('avg_longitude', pl.lit(None, dtype=pl.Float64)))
|
||||
_agg_logger.info(f'[FIND_COL] keywords=lon matched={lon_col} dtype={schema.get(lon_col, "N/A") if lon_col else "None"}')
|
||||
|
||||
# ── Time patterns ────────────────────────────────────────────────────
|
||||
ts_col = _timestamp_col(schema)
|
||||
if ts_col:
|
||||
# For first/last seen we need min/max of the timestamp column
|
||||
aggs.append(('first_seen', pl.min(ts_col)))
|
||||
aggs.append(('last_seen', pl.max(ts_col)))
|
||||
aggs.append(('_ts_col', pl.lit(ts_col)))
|
||||
# Active hours: unique hour-of-day count
|
||||
# We'll compute this after collecting by extracting hour from timestamps
|
||||
aggs.append(('_has_timestamp', pl.lit(True)))
|
||||
else:
|
||||
aggs.append(('first_seen', pl.lit(None, dtype=pl.Utf8)))
|
||||
aggs.append(('last_seen', pl.lit(None, dtype=pl.Utf8)))
|
||||
aggs.append(('_ts_col', pl.lit('')))
|
||||
aggs.append(('_has_timestamp', pl.lit(False)))
|
||||
|
||||
# ── Type-classifier features ──────────────────────────────────────
|
||||
# HEX/URL/IPv4 feature extraction removed — avg_hex_pairs_count,
|
||||
# unique_domains, and unique_24_networks are not useful for TLS analysis.
|
||||
|
||||
return aggs
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main aggregation entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Columns whose ``_``-prefixed names are internal bookkeeping, not user-facing
|
||||
_INTERNAL_COLUMNS = tuple([
|
||||
'_bytes_sent_col', '_dst_ip_col', '_dst_port_col',
|
||||
'_proto_col', '_ts_col', '_has_timestamp',
|
||||
'_bytes_sent_expr',
|
||||
])
|
||||
|
||||
|
||||
def aggregate_by_entity(
|
||||
lf: pl.LazyFrame,
|
||||
entity_columns: str | list[str],
|
||||
schema: dict[str, str],
|
||||
type_map: dict[str, DataType] | None = None,
|
||||
) -> tuple[pl.LazyFrame, list[str]]:
|
||||
"""Group flow-level data by *entity_columns* and compute per-entity features.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lf:
|
||||
A Polars :class:`LazyFrame` containing flow-level data.
|
||||
entity_columns:
|
||||
Column name(s) to group by (e.g. ``'src_ip'``, or ``['src_ip', 'dst_ip']``).
|
||||
schema:
|
||||
Dictionary mapping column names to dtype strings (e.g. from
|
||||
``SessionStore.get_dataset(...)['schema']``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
tuple[pl.LazyFrame, list[str]]
|
||||
``(aggregated_lazyframe, feature_column_names)`` where the lazyframe
|
||||
has one row per entity and *feature_column_names* lists the
|
||||
user-facing aggregate columns.
|
||||
"""
|
||||
# Normalise to list
|
||||
if isinstance(entity_columns, str):
|
||||
entity_columns = [entity_columns]
|
||||
|
||||
# Build aggregation expressions based on the schema (and optional type_map)
|
||||
agg_params = build_aggregation_params(schema, type_map=type_map)
|
||||
|
||||
# Separate feature names and expressions
|
||||
feature_names: list[str] = []
|
||||
exprs: list[pl.Expr] = []
|
||||
for name, expr in agg_params:
|
||||
exprs.append(expr.alias(name))
|
||||
if not name.startswith('_'):
|
||||
feature_names.append(name)
|
||||
|
||||
# Check entity columns exist
|
||||
missing = [c for c in entity_columns if c not in schema]
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"Entity column(s) '{missing}' not found in schema. "
|
||||
f"Available columns: {list(schema.keys())}"
|
||||
)
|
||||
|
||||
# Perform aggregation
|
||||
aggregated_lf = lf.group_by(entity_columns).agg(exprs)
|
||||
|
||||
# Post-process: compute active_hours_count if a timestamp column exists
|
||||
# (We compute this as a second pass since it requires extraction logic)
|
||||
any(name == '_has_timestamp' for name, _ in agg_params)
|
||||
|
||||
# Filter to only user-facing feature columns names (excluding _internal)
|
||||
user_features = [f for f in feature_names if f not in _INTERNAL_COLUMNS]
|
||||
|
||||
return aggregated_lf, user_features
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Auto-detect entity column from CSV schema using heuristic scoring.
|
||||
|
||||
Provides :func:`detect_entity_column` which scores all string/categorical
|
||||
columns by keyword-name match, unique-value ratio, and dtype, returning
|
||||
a ranked list of candidates and a recommended column.
|
||||
|
||||
Usage::
|
||||
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
|
||||
result = detect_entity_column(dataset_id)
|
||||
result['recommended'] # e.g. 'src_ip'
|
||||
result['candidates'] # sorted list of scored columns
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import polars as pl
|
||||
|
||||
from .session_store import SessionStore
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entity-related keywords (lowercased)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
ENTITY_KEYWORDS: tuple[str, ...] = (
|
||||
# User exact column names only — no generic/fuzzy keywords
|
||||
':ips', ':ipd', ':prs', ':prd',
|
||||
'server_ip', 'client_ip', 'scnt', 'dcnt',
|
||||
'cnam', 'snam', '0ver', '4dur', '8ses', '2tmo', '4ksz',
|
||||
'cnrs', 'isrs', '8ack', '8ppk', '8dbd',
|
||||
'1ipp', '4dbn', 'tabl', 'name', 'source_node',
|
||||
'cipher_suite', 'ecdhe_named_curve',
|
||||
'0cph', '0crv', '0rnd', '0rnt', 'row', 'time', 'timestamp',
|
||||
'latd', 'lond', 'ispn', 'orgn', 'city',
|
||||
)
|
||||
|
||||
|
||||
def _score_name(column: str) -> tuple[float, str]:
|
||||
"""Score a column name against *ENTITY_KEYWORDS*.
|
||||
|
||||
Returns ``(score, matched_keyword_or_empty)`` where *score* is 3.0
|
||||
for every matched keyword. Partial matches (keyword is a substring
|
||||
of the lowercased column name) also count.
|
||||
"""
|
||||
col_lower = column.lower().replace('-', '_').replace('.', '_').replace(' ', '_')
|
||||
for kw in ENTITY_KEYWORDS:
|
||||
if kw == col_lower or col_lower.endswith(f'_{kw}') or col_lower.startswith(f'{kw}_'):
|
||||
return 3.0, kw
|
||||
return 0.0, ''
|
||||
|
||||
|
||||
def detect_entity_column(dataset_id: str, entity_columns: Optional[list[str]] = None) -> dict:
|
||||
"""Score all string/categorical columns and return candidates.
|
||||
|
||||
The scoring algorithm:
|
||||
|
||||
* **Name match** (``+3``): column name (normalised) is a keyword, or
|
||||
ends/starts with a keyword via ``_`` separator.
|
||||
* **Unique ratio** (0–10): columns whose ratio of unique values to
|
||||
non-null values falls in the range ``[0.01, 0.50]`` receive up to
|
||||
10 points (``min(ratio * 100, 10)``).
|
||||
* **String dtype** (``+2``): ``Utf8`` / ``String`` / ``Categorical``
|
||||
columns get a bonus.
|
||||
* **Null penalty** (``*0.1``): if null ratio > 50%, score severely reduced.
|
||||
* **Penalty** (``-5``): unique ratio > 0.50 (likely random IDs or
|
||||
transaction-level identifiers).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
dataset_id:
|
||||
Dataset identifier in the session store.
|
||||
entity_columns:
|
||||
Optional list of column names to restrict detection to. When given,
|
||||
only these columns are scored (must exist in the schema).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict
|
||||
``candidates``: list of ``{column, score, unique_ratio, matched_keyword}``
|
||||
sorted descending by score.
|
||||
|
||||
``recommended``: highest-scoring column name (or ``''`` when none found).
|
||||
|
||||
``recommended_multi``: list of the top 3 candidate column names (may be
|
||||
shorter than 3 when fewer candidates exist).
|
||||
|
||||
``dataset_id``: the input identifier (echoed).
|
||||
"""
|
||||
store = SessionStore()
|
||||
entry = store.get_dataset(dataset_id)
|
||||
if entry is None:
|
||||
return {
|
||||
'dataset_id': dataset_id,
|
||||
'candidates': [],
|
||||
'recommended': '',
|
||||
'error': f'Dataset not found: {dataset_id}',
|
||||
}
|
||||
|
||||
lf: pl.LazyFrame = entry['lazyframe']
|
||||
schema: dict[str, str] = entry.get('schema', {})
|
||||
|
||||
# Identify string / categorical columns
|
||||
string_types = frozenset({'Utf8', 'String', 'Categorical', 'str', 'cat'})
|
||||
candidate_scores: list[dict] = []
|
||||
|
||||
try:
|
||||
# Collect a sample to compute unique ratios efficiently
|
||||
df_sample = lf.collect(streaming=True)
|
||||
except Exception as exc:
|
||||
return {
|
||||
'dataset_id': dataset_id,
|
||||
'candidates': [],
|
||||
'recommended': '',
|
||||
'error': f'Failed to collect dataset: {exc}',
|
||||
}
|
||||
|
||||
total_rows = len(df_sample)
|
||||
|
||||
for col_name, dtype_str in schema.items():
|
||||
# If entity_columns specified, skip columns not in the list
|
||||
if entity_columns is not None and col_name not in entity_columns:
|
||||
continue
|
||||
|
||||
# Only score string-like columns
|
||||
dtype_base = dtype_str.split('(')[0].strip()
|
||||
if dtype_base not in string_types:
|
||||
continue
|
||||
|
||||
series_all = df_sample[col_name]
|
||||
non_null_series = series_all.drop_nulls()
|
||||
non_null_count = len(non_null_series)
|
||||
null_count = total_rows - non_null_count
|
||||
|
||||
if non_null_count == 0:
|
||||
continue
|
||||
|
||||
unique_count = int(non_null_series.n_unique())
|
||||
unique_ratio = unique_count / non_null_count
|
||||
|
||||
# --- scoring ---
|
||||
name_score, matched_kw = _score_name(col_name)
|
||||
dtype_score = 2.0 # String/categorical bonus
|
||||
ratio_score = min(unique_ratio * 100.0, 10.0)
|
||||
penalty = -5.0 if unique_ratio > 0.50 else 0.0
|
||||
|
||||
total_score = name_score + dtype_score + ratio_score + penalty
|
||||
|
||||
# Null ratio penalty: if more than 50% null, severely reduce score
|
||||
if null_count / max(1, non_null_count) > 0.5:
|
||||
total_score *= 0.1
|
||||
|
||||
candidate_scores.append({
|
||||
'column': col_name,
|
||||
'score': round(total_score, 2),
|
||||
'unique_ratio': round(unique_ratio, 4),
|
||||
'matched_keyword': matched_kw,
|
||||
'non_null_count': non_null_count,
|
||||
'unique_count': unique_count,
|
||||
'null_count': null_count,
|
||||
})
|
||||
|
||||
# Sort descending by score, then by unique_count desc (prefer richer columns)
|
||||
candidate_scores.sort(key=lambda c: (c['score'], c['unique_count']), reverse=True)
|
||||
|
||||
recommended = candidate_scores[0]['column'] if candidate_scores else ''
|
||||
recommended_multi = [c['column'] for c in candidate_scores[:3]]
|
||||
|
||||
return {
|
||||
'dataset_id': dataset_id,
|
||||
'candidates': candidate_scores,
|
||||
'recommended': recommended,
|
||||
'recommended_multi': recommended_multi, # NEW
|
||||
'total_rows': total_rows,
|
||||
'scored_columns': len(candidate_scores),
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
"""
|
||||
离线 GeoIP 查询。内嵌 IPv4 区间 → 城市/经纬度映射。
|
||||
|
||||
从 ``data/geoip_data.txt`` 加载 IP 段数据，在模块首次导入时构建
|
||||
排序索引，提供 O(log N) 的 IP 查询能力。
|
||||
|
||||
数据约 200KB，覆盖 23 个主要城市 IP 段。
|
||||
可通过编辑 ``data/geoip_data.txt`` 扩展。
|
||||
"""
|
||||
import bisect
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# [(start_int, end_int, lat, lon, city, country), ...]
|
||||
# Built at module load time from data/geoip_data.txt
|
||||
_GEO_DATA: list[tuple[int, int, float, float, str, str]] = []
|
||||
|
||||
# Sorted list of start integers (for bisect lookup) + parallel data list
|
||||
_STARTS: list[int] = []
|
||||
_RANGES: list[tuple[int, int, float, float, str, str]] = []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IP conversion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def ip_to_int(ip_str: str) -> Optional[int]:
|
||||
"""Convert an IPv4 dotted-string to its 32-bit integer representation.
|
||||
|
||||
Returns ``None`` if *ip_str* is not a valid IPv4 address (including
|
||||
when any octet is outside 0-255).
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> ip_to_int('8.8.8.8')
|
||||
134744072
|
||||
>>> ip_to_int('1.0.0.0')
|
||||
16777216
|
||||
"""
|
||||
parts = ip_str.strip().split('.')
|
||||
if len(parts) != 4:
|
||||
return None
|
||||
try:
|
||||
octets = [int(p) for p in parts]
|
||||
if any(o < 0 or o > 255 for o in octets):
|
||||
return None
|
||||
return (octets[0] << 24) | (octets[1] << 16) | (octets[2] << 8) | octets[3]
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Index building
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _build_index() -> tuple[list[int], list[tuple[int, int, float, float, str, str]]]:
|
||||
"""Build sorted parallel lists for bisect-based IP lookup.
|
||||
|
||||
Processes ``_GEO_DATA``, sorts by start integer, and returns
|
||||
``(starts, ranges)`` where *starts* is a sorted list of start integers
|
||||
and *ranges* is the corresponding data list at the same indices.
|
||||
"""
|
||||
if not _GEO_DATA:
|
||||
return [], []
|
||||
|
||||
sorted_data = sorted(_GEO_DATA, key=lambda x: x[0])
|
||||
starts = [entry[0] for entry in sorted_data]
|
||||
return starts, sorted_data
|
||||
|
||||
|
||||
def _load_data_file() -> None:
|
||||
"""Load IP range data from ``data/geoip_data.txt`` into ``_GEO_DATA``.
|
||||
|
||||
Called once at module import time. Each non-blank, non-comment line
|
||||
in the file should contain six comma-separated fields::
|
||||
|
||||
start_ip,end_ip,lat,lon,city,country
|
||||
"""
|
||||
data_path = Path(__file__).parent.parent / 'data' / 'geoip_data.txt'
|
||||
|
||||
if not data_path.exists():
|
||||
logger.warning('[GEOIP] data file not found: %s', data_path)
|
||||
return
|
||||
|
||||
loaded: list[tuple[int, int, float, float, str, str]] = []
|
||||
skipped = 0
|
||||
|
||||
with open(data_path, 'r', encoding='utf-8') as f:
|
||||
for line_no, raw in enumerate(f, start=1):
|
||||
stripped = raw.strip()
|
||||
if not stripped or stripped.startswith('#'):
|
||||
continue
|
||||
|
||||
parts = [p.strip() for p in stripped.split(',')]
|
||||
if len(parts) != 6:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
start_ip_str, end_ip_str, lat_str, lon_str, city, country = parts
|
||||
|
||||
start_int = ip_to_int(start_ip_str)
|
||||
end_int = ip_to_int(end_ip_str)
|
||||
|
||||
if start_int is None or end_int is None:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
lat = float(lat_str)
|
||||
lon = float(lon_str)
|
||||
except ValueError:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
loaded.append((start_int, end_int, lat, lon, city, country))
|
||||
|
||||
_GEO_DATA.clear()
|
||||
_GEO_DATA.extend(loaded)
|
||||
|
||||
global _STARTS, _RANGES
|
||||
_STARTS, _RANGES = _build_index()
|
||||
|
||||
logger.info('[GEOIP] loaded %d ranges from %s (%d skipped)',
|
||||
len(loaded), data_path, skipped)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def lookup(ip_str: str) -> Optional[dict]:
|
||||
"""查询 IP 对应的地理位置。
|
||||
|
||||
Uses binary search (``bisect_right``) on the sorted start-IP list to
|
||||
locate the range *ip_str* falls within, then checks the end bound.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
ip_str:
|
||||
IPv4 address as a dotted string (e.g. ``'8.8.8.8'``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict or None
|
||||
A dictionary with keys ``lat``, ``lon``, ``city``, ``country``
|
||||
if the IP is found in the database; ``None`` otherwise.
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> result = lookup('8.8.8.8')
|
||||
>>> result['city']
|
||||
'Mountain View'
|
||||
>>> lookup('10.0.0.1') is None
|
||||
True
|
||||
"""
|
||||
target = ip_to_int(ip_str)
|
||||
if target is None:
|
||||
return None
|
||||
|
||||
if not _STARTS:
|
||||
return None
|
||||
|
||||
idx = bisect.bisect_right(_STARTS, target) - 1
|
||||
if idx < 0:
|
||||
return None
|
||||
|
||||
start, end, lat, lon, city, country = _RANGES[idx]
|
||||
if start <= target <= end:
|
||||
return {
|
||||
'lat': lat,
|
||||
'lon': lon,
|
||||
'city': city,
|
||||
'country': country,
|
||||
}
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level initialisation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_load_data_file()
|
||||
@@ -0,0 +1,91 @@
|
||||
"""IP address utilities for subnet clustering and conversion.
|
||||
|
||||
Provides helper functions to convert IPv4 strings to integers, compute
|
||||
subnet prefixes, and group IP lists by subnet.
|
||||
|
||||
Typical usage::
|
||||
|
||||
from analysis.ip_clustering import ip_to_int, get_subnet, cluster_ips
|
||||
|
||||
ip_int = ip_to_int('192.168.1.1') # → 3232235777
|
||||
subnet = get_subnet('192.168.1.5', 24) # → '192.168.1.0/24'
|
||||
groups = cluster_ips(ip_list, mask=24) # → {'192.168.1.0/24': [...]}
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
def ip_to_int(ip_str: str) -> int | None:
|
||||
"""Convert an IPv4 string to its 32-bit integer representation.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
ip_str:
|
||||
An IPv4 address string, e.g. ``'192.168.1.1'``.
|
||||
|
||||
Returns
|
||||
-------
|
||||
int | None
|
||||
The integer value of the address, or ``None`` if the input
|
||||
is not a valid IPv4 address.
|
||||
"""
|
||||
try:
|
||||
return int(ipaddress.IPv4Address(ip_str))
|
||||
except (ipaddress.AddressValueError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def get_subnet(ip_str: str, mask: int = 24) -> str | None:
|
||||
"""Compute the CIDR subnet string for an IP and mask length.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
ip_str:
|
||||
An IPv4 address string, e.g. ``'192.168.1.5'``.
|
||||
mask:
|
||||
Subnet mask length (default ``24``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
str | None
|
||||
Subnet string like ``'192.168.1.0/24'``, or ``None`` if the IP
|
||||
is not valid.
|
||||
"""
|
||||
try:
|
||||
ip = ipaddress.IPv4Address(ip_str)
|
||||
net = ipaddress.IPv4Network(f'{ip}/{mask}', strict=False)
|
||||
return str(net)
|
||||
except (ipaddress.AddressValueError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def cluster_ips(
|
||||
ip_list: list[str],
|
||||
mask: int = 24,
|
||||
) -> dict[str, list[str]]:
|
||||
"""Group a list of IPv4 addresses by their subnet.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
ip_list:
|
||||
List of IPv4 address strings.
|
||||
mask:
|
||||
Subnet mask length (default ``24``).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict[str, list[str]]
|
||||
Mapping of ``subnet`` → ``[ip, ip, ...]``. Invalid IPs are
|
||||
grouped under the key ``'__invalid__'``.
|
||||
"""
|
||||
groups: dict[str, list[str]] = defaultdict(list)
|
||||
for ip_str in ip_list:
|
||||
subnet = get_subnet(ip_str, mask)
|
||||
if subnet is not None:
|
||||
groups[subnet].append(ip_str)
|
||||
else:
|
||||
groups['__invalid__'].append(ip_str)
|
||||
# Convert to plain dict for stable serialisation
|
||||
return dict(groups)
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Django management command: ``python manage.py run_pipeline``
|
||||
|
||||
Runs the complete TLS analysis pipeline from CLI, no LLM required.
|
||||
Usage: uv run python manage.py run_pipeline <csv_glob> [--entity-col NAME] [--algo hdbscan|kmeans]
|
||||
"""
|
||||
import os, sys, json, asyncio
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
from django.utils import timezone
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = 'Run the full TLS analysis pipeline from CLI (no LLM needed)'
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument('csv_glob', type=str, help='Glob pattern for CSV files')
|
||||
parser.add_argument('--entity-col', type=str, default=None,
|
||||
help='Entity column (auto-detect if omitted)')
|
||||
parser.add_argument('--algo', type=str, default='hdbscan', choices=['hdbscan', 'kmeans'],
|
||||
help='Clustering algorithm (default: hdbscan)')
|
||||
parser.add_argument('--output', type=str, default=None,
|
||||
help='Output directory for exports (default: no export)')
|
||||
|
||||
def handle(self, *args, **options):
|
||||
import django; django.setup()
|
||||
from analysis.models import AnalysisRun
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
|
||||
csv_glob = options['csv_glob']
|
||||
entity_col = options['entity_col']
|
||||
algo = options['algo']
|
||||
output = options['output']
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
|
||||
# ── Step 1: Load CSV ─────────────────────────────────────────────
|
||||
self.stdout.write(f'[1/5] 加载 CSV: {csv_glob}')
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(csv_glob)
|
||||
ds_id = 'manual_ds'
|
||||
store.store_dataset(ds_id, lf, schema=schema, metadata={
|
||||
'row_count': row_count, 'file_count': file_count, 'csv_glob': csv_glob,
|
||||
})
|
||||
self.stdout.write(self.style.SUCCESS(f' → {row_count} rows, {file_count} files, {memory_mb:.1f} MB'))
|
||||
|
||||
# ── Step 2: Detect entity column ────────────────────────────────
|
||||
self.stdout.write('[2/5] 检测实体列')
|
||||
if entity_col:
|
||||
used_col = entity_col
|
||||
self.stdout.write(f' → 使用指定列: {used_col}')
|
||||
else:
|
||||
result = detect_entity_column(ds_id)
|
||||
candidates = result.get('candidates', [])
|
||||
used_col = result.get('recommended')
|
||||
if not used_col:
|
||||
self.stderr.write(self.style.ERROR(' ✗ 未检测到实体列，请用 --entity-col 手动指定'))
|
||||
return
|
||||
self.stdout.write(f' → 自动检测: {used_col} (共 {len(candidates)} 候选)')
|
||||
|
||||
# ── Step 3: Aggregate by entity ──────────────────────────────────
|
||||
self.stdout.write('[3/5] 实体聚合')
|
||||
agg_lf, feature_columns = aggregate_by_entity(lf, used_col, schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
entity_ds_id = 'manual_entity'
|
||||
store.store_dataset(entity_ds_id, agg_lf, schema=dict(zip(feature_columns, ['']*len(feature_columns))),
|
||||
metadata={'row_count': len(df_agg), 'entity_column': used_col, 'csv_glob': csv_glob})
|
||||
self.stdout.write(self.style.SUCCESS(f' → {len(df_agg)} 个实体, {len(feature_columns)} 维特征'))
|
||||
|
||||
# ── Step 4: Clustering ──────────────────────────────────────────
|
||||
self.stdout.write(f'[4/5] 聚类 ({algo})')
|
||||
from analysis.tool_registry import _handle_run_clustering
|
||||
# 只选用户面特征列（排除内部 _ 开头列 + 时间列）
|
||||
user_features = [
|
||||
c for c in feature_columns
|
||||
if not c.startswith('_') and c not in ('first_seen', 'last_seen', 'src_ip')
|
||||
][:10]
|
||||
self.stdout.write(f' → 聚类特征: {user_features}')
|
||||
result = asyncio.run(_handle_run_clustering(
|
||||
dataset_id=entity_ds_id,
|
||||
cluster_columns=user_features,
|
||||
algorithm=algo, params={}, random_state=42,
|
||||
))
|
||||
if 'error' in result:
|
||||
self.stderr.write(self.style.ERROR(f' ✗ 聚类失败: {result["error"]}'))
|
||||
return
|
||||
cluster_id = result.get('cluster_result_id', '')
|
||||
n_clusters = result.get('n_clusters', 0)
|
||||
quality = result.get('quality_metrics', {})
|
||||
self.stdout.write(self.style.SUCCESS(
|
||||
f' → {n_clusters} 个簇, 噪声比 {quality.get("noise_ratio", "N/A")}'
|
||||
))
|
||||
if quality.get('silhouette_score'):
|
||||
self.stdout.write(f' → Silhouette: {quality["silhouette_score"]:.4f}')
|
||||
|
||||
# ── Step 5: Feature extraction ───────────────────────────────────
|
||||
self.stdout.write('[5/5] 特征提取')
|
||||
from analysis.tool_registry import _handle_extract_features
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id=entity_ds_id, cluster_result_id=cluster_id,
|
||||
top_k=10, method='zscore', save_to_db=True,
|
||||
))
|
||||
if 'error' in feat_result:
|
||||
self.stderr.write(self.style.ERROR(f' ✗ 特征提取失败: {feat_result["error"]}'))
|
||||
else:
|
||||
self.stdout.write(self.style.SUCCESS(f' → {feat_result.get("n_features", 0)} 个特征已保存到数据库'))
|
||||
|
||||
# ── Export ───────────────────────────────────────────────────────
|
||||
if output:
|
||||
from analysis.tool_registry import _handle_export_results
|
||||
os.makedirs(output, exist_ok=True)
|
||||
asyncio.run(_handle_export_results(result_id=entity_ds_id, output_path=output, format='csv', overwrite=True))
|
||||
self.stdout.write(self.style.SUCCESS(f' → 导出到 {output}'))
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
run_record = AnalysisRun.objects.create(
|
||||
csv_glob=csv_glob, status='completed',
|
||||
total_flows=row_count, entity_count=len(df_agg),
|
||||
cluster_count=n_clusters, entity_column=used_col,
|
||||
)
|
||||
|
||||
# ── Step 6: PCA-2D embedding (after run_record exists) ─────────────
|
||||
try:
|
||||
from sklearn.preprocessing import StandardScaler
|
||||
from sklearn.decomposition import PCA
|
||||
import polars as pl, numpy as np
|
||||
num_cols = [c for c in df_agg.columns if df_agg[c].dtype in (
|
||||
pl.Int8,pl.Int16,pl.Int32,pl.Int64,pl.UInt8,pl.UInt16,pl.UInt32,pl.UInt64,pl.Float32,pl.Float64)]
|
||||
if len(num_cols) >= 2 and len(df_agg) > 2:
|
||||
mat = np.nan_to_num(StandardScaler().fit_transform(df_agg.select(num_cols).to_numpy()), nan=0.0)
|
||||
coords = PCA(n_components=2, random_state=42).fit_transform(mat)
|
||||
non_num = [c for c in df_agg.columns if c not in num_cols and not c.startswith('_')]
|
||||
ecol = non_num[0] if non_num else df_agg.columns[0]
|
||||
from analysis.models import EntityProfile as EP, ClusterResult as CR
|
||||
for i, row in enumerate(df_agg.iter_rows(named=True)):
|
||||
if i >= len(coords): break
|
||||
ev = str(row.get(ecol, ''))
|
||||
if ev:
|
||||
EP.objects.get_or_create(entity_value=ev, defaults={
|
||||
'run_id': run_record.id,
|
||||
'cluster_label': 0,
|
||||
'embedding_x': float(coords[i,0]),
|
||||
'embedding_y': float(coords[i,1])})
|
||||
except Exception as e:
|
||||
self.stderr.write(f' ⚠ PCA降维跳过: {e}')
|
||||
self.stdout.write(self.style.SUCCESS(
|
||||
f'\n*** 分析完成! Run ID: #{run_record.id}'
|
||||
f' 查看 Django: http://127.0.0.1:8000/runs/{run_record.id}/'
|
||||
))
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Django management command: ``python manage.py start_mcp``
|
||||
|
||||
Starts the embedded TLS Analyzer MCP server on stdio transport.
|
||||
IMPORTANT: All log output goes to stderr (stdout is the MCP protocol transport).
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
from django.core.management.base import BaseCommand, CommandError
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = 'Start the embedded TLS Analyzer MCP server on stdio transport'
|
||||
|
||||
def handle(self, *args, **options):
|
||||
# Ensure Django is fully initialised before importing server code
|
||||
import django
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
|
||||
django.setup()
|
||||
|
||||
# CRITICAL: MCP uses stdio for protocol transport.
|
||||
# All log messages MUST go to stderr, NOT stdout.
|
||||
print('Starting TLS Analyzer MCP server...', file=sys.stderr)
|
||||
print('Transport: stdio', file=sys.stderr)
|
||||
sys.stderr.flush()
|
||||
|
||||
try:
|
||||
from analysis.mcp_server import TLSAnalyzerMCPServer
|
||||
server = TLSAnalyzerMCPServer()
|
||||
server.run_sync()
|
||||
except KeyboardInterrupt:
|
||||
print('\nMCP server stopped.', file=sys.stderr)
|
||||
except Exception as e:
|
||||
raise CommandError(f'MCP server failed: {e}') from e
|
||||
@@ -0,0 +1,84 @@
|
||||
"""MCP Server embedded in Django.
|
||||
|
||||
Uses asyncio stdio transport. Registered via the Django management command
|
||||
``python manage.py start_mcp``.
|
||||
|
||||
Django ORM initialisation happens at the management-command level so that
|
||||
the server thread has access to models when needed (e.g. during feature
|
||||
extraction).
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
|
||||
from mcp.server import Server
|
||||
from mcp.server.stdio import stdio_server
|
||||
from mcp.types import TextContent
|
||||
|
||||
# Ensure Django is configured when this module is imported from the
|
||||
# management command. The management command calls django.setup()
|
||||
# before constructing the server.
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
|
||||
|
||||
from .session_store import SessionStore
|
||||
from .tool_registry import get_tools_meta, handle_call
|
||||
|
||||
|
||||
class TLSAnalyzerMCPServer:
|
||||
"""Embedded MCP server for the TLS Analyzer.
|
||||
|
||||
Exposes 11 tools over stdio transport. The :class:`SessionStore`
|
||||
singleton holds datasets and analysis results in memory for the
|
||||
lifetime of the server process.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.server = Server("tls-analyzer")
|
||||
self.store = SessionStore()
|
||||
self._setup_tools()
|
||||
|
||||
# ── tool registration ───────────────────────────────────────────────
|
||||
|
||||
def _setup_tools(self) -> None:
|
||||
"""Register all tools with the MCP server via the SDK decorator API.
|
||||
|
||||
Uses :func:`get_tools_meta` for tool metadata (name, description,
|
||||
inputSchema) and :func:`handle_call` for request routing.
|
||||
"""
|
||||
tools_meta = get_tools_meta()
|
||||
_tools_map = {t.name: t for t in tools_meta}
|
||||
|
||||
@self.server.list_tools()
|
||||
async def list_tools():
|
||||
return tools_meta
|
||||
|
||||
@self.server.call_tool()
|
||||
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
||||
try:
|
||||
result = await handle_call(name, arguments)
|
||||
return [TextContent(type="text", text=json.dumps(result, default=str))]
|
||||
except Exception as exc:
|
||||
error_body = json.dumps({
|
||||
'error': str(exc),
|
||||
'tool': name,
|
||||
'truncated': False,
|
||||
}, default=str)
|
||||
return [TextContent(type="text", text=error_body)]
|
||||
|
||||
# ── lifecycle ───────────────────────────────────────────────────────
|
||||
|
||||
async def run(self) -> None:
|
||||
"""Start the MCP server with stdio transport.
|
||||
|
||||
Blocks until the transport is closed.
|
||||
"""
|
||||
async with stdio_server() as (read, write):
|
||||
await self.server.run(
|
||||
read,
|
||||
write,
|
||||
self.server.create_initialization_options(),
|
||||
)
|
||||
|
||||
def run_sync(self) -> None:
|
||||
"""Synchronous entry point for the management command."""
|
||||
asyncio.run(self.run())
|
||||
@@ -0,0 +1,84 @@
|
||||
# Generated by Django 4.2.30 on 2026-07-13 14:36
|
||||
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
initial = True
|
||||
|
||||
dependencies = [
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='AnalysisRun',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('csv_glob', models.CharField(help_text='Glob pattern for CSV files', max_length=1024)),
|
||||
('config', models.JSONField(blank=True, default=dict, help_text='YAML config snapshot')),
|
||||
('status', models.CharField(choices=[('pending', 'Pending'), ('loading', 'Loading Data'), ('profiling', 'Profiling'), ('aggregating', 'Building Entity Profiles'), ('clustering', 'Clustering'), ('extracting', 'Extracting Features'), ('completed', 'Completed'), ('failed', 'Failed')], default='pending', max_length=32)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('updated_at', models.DateTimeField(auto_now=True)),
|
||||
('error_message', models.TextField(blank=True, default='')),
|
||||
('total_flows', models.IntegerField(blank=True, null=True)),
|
||||
('entity_count', models.IntegerField(blank=True, null=True)),
|
||||
('cluster_count', models.IntegerField(blank=True, null=True)),
|
||||
('entity_column', models.CharField(blank=True, default='', help_text='Auto-detected entity column name', max_length=256)),
|
||||
],
|
||||
options={
|
||||
'ordering': ['-created_at'],
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='ClusterResult',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('cluster_label', models.IntegerField(help_text='Cluster label (-1 for noise)')),
|
||||
('size', models.IntegerField(help_text='Number of entities in this cluster')),
|
||||
('proportion', models.FloatField(blank=True, help_text='Proportion of total entities', null=True)),
|
||||
('noise_ratio', models.FloatField(blank=True, help_text='Noise ratio of the clustering', null=True)),
|
||||
('silhouette_score', models.FloatField(blank=True, help_text='Silhouette score (sampled if large)', null=True)),
|
||||
('run', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='clusters', to='tianxuan_analysis.analysisrun')),
|
||||
],
|
||||
options={
|
||||
'unique_together': {('run', 'cluster_label')},
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='EntityProfile',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('entity_value', models.CharField(help_text='Entity identifier (IP, domain, etc.)', max_length=512)),
|
||||
('cluster_label', models.IntegerField(blank=True, help_text='Assigned cluster (-1=noise)', null=True)),
|
||||
('feature_json', models.JSONField(blank=True, default=dict, help_text='Aggregated entity features as dict')),
|
||||
('embedding_x', models.FloatField(blank=True, help_text='PCA-2D X coordinate', null=True)),
|
||||
('embedding_y', models.FloatField(blank=True, help_text='PCA-2D Y coordinate', null=True)),
|
||||
('cluster', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='entities', to='tianxuan_analysis.clusterresult')),
|
||||
('run', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='entities', to='tianxuan_analysis.analysisrun')),
|
||||
],
|
||||
options={
|
||||
'unique_together': {('run', 'entity_value')},
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='ClusterFeature',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('feature_name', models.CharField(max_length=256)),
|
||||
('mean', models.FloatField(blank=True, null=True)),
|
||||
('std', models.FloatField(blank=True, null=True)),
|
||||
('median', models.FloatField(blank=True, null=True)),
|
||||
('p25', models.FloatField(blank=True, null=True)),
|
||||
('p75', models.FloatField(blank=True, null=True)),
|
||||
('missing_rate', models.FloatField(blank=True, null=True)),
|
||||
('distinguishing_score', models.FloatField(blank=True, help_text='Z-score or ANOVA F-score', null=True)),
|
||||
('distinguishing_method', models.CharField(default='zscore', max_length=16)),
|
||||
('cluster', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='features', to='tianxuan_analysis.clusterresult')),
|
||||
],
|
||||
options={
|
||||
'unique_together': {('cluster', 'feature_name')},
|
||||
},
|
||||
),
|
||||
]
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
# Generated by Django 4.2.30 on 2026-07-16 05:11
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('tianxuan_analysis', '0001_initial'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='analysisrun',
|
||||
name='progress_msg',
|
||||
field=models.CharField(blank=True, default='', max_length=256),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='analysisrun',
|
||||
name='progress_pct',
|
||||
field=models.IntegerField(default=0),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='analysisrun',
|
||||
name='run_log',
|
||||
field=models.TextField(blank=True, default=''),
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,94 @@
|
||||
from django.db import models
|
||||
|
||||
|
||||
class AnalysisRun(models.Model):
|
||||
"""Tracks a single analysis pipeline execution."""
|
||||
STATUS_CHOICES = [
|
||||
('pending', 'Pending'),
|
||||
('loading', 'Loading Data'),
|
||||
('profiling', 'Profiling'),
|
||||
('aggregating', 'Building Entity Profiles'),
|
||||
('clustering', 'Clustering'),
|
||||
('extracting', 'Extracting Features'),
|
||||
('completed', 'Completed'),
|
||||
('failed', 'Failed'),
|
||||
]
|
||||
|
||||
csv_glob = models.CharField(max_length=1024, help_text="Glob pattern for CSV files")
|
||||
config = models.JSONField(default=dict, blank=True, help_text="YAML config snapshot")
|
||||
status = models.CharField(max_length=32, choices=STATUS_CHOICES, default='pending')
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
updated_at = models.DateTimeField(auto_now=True)
|
||||
error_message = models.TextField(blank=True, default='')
|
||||
progress_pct = models.IntegerField(default=0)
|
||||
progress_msg = models.CharField(max_length=256, default='', blank=True)
|
||||
run_log = models.TextField(blank=True, default='')
|
||||
|
||||
# Summary stats populated after completion
|
||||
total_flows = models.IntegerField(null=True, blank=True)
|
||||
entity_count = models.IntegerField(null=True, blank=True)
|
||||
cluster_count = models.IntegerField(null=True, blank=True)
|
||||
entity_column = models.CharField(max_length=256, blank=True, default='',
|
||||
help_text="Auto-detected entity column name")
|
||||
|
||||
class Meta:
|
||||
ordering = ['-created_at']
|
||||
|
||||
def __str__(self):
|
||||
return f"Run #{self.id} ({self.created_at:%Y-%m-%d %H:%M}) - {self.status}"
|
||||
|
||||
|
||||
class ClusterResult(models.Model):
|
||||
"""Stores clustering results for a single analysis run."""
|
||||
run = models.ForeignKey(AnalysisRun, on_delete=models.CASCADE, related_name='clusters')
|
||||
cluster_label = models.IntegerField(help_text="Cluster label (-1 for noise)")
|
||||
size = models.IntegerField(help_text="Number of entities in this cluster")
|
||||
proportion = models.FloatField(null=True, blank=True, help_text="Proportion of total entities")
|
||||
noise_ratio = models.FloatField(null=True, blank=True, help_text="Noise ratio of the clustering")
|
||||
silhouette_score = models.FloatField(null=True, blank=True, help_text="Silhouette score (sampled if large)")
|
||||
|
||||
class Meta:
|
||||
unique_together = ['run', 'cluster_label']
|
||||
|
||||
def __str__(self):
|
||||
return f"Cluster #{self.cluster_label} (n={self.size})"
|
||||
|
||||
|
||||
class EntityProfile(models.Model):
|
||||
"""One row per entity (user/host) per analysis run."""
|
||||
run = models.ForeignKey(AnalysisRun, on_delete=models.CASCADE, related_name='entities')
|
||||
entity_value = models.CharField(max_length=512, help_text="Entity identifier (IP, domain, etc.)")
|
||||
cluster_label = models.IntegerField(null=True, blank=True, help_text="Assigned cluster (-1=noise)")
|
||||
cluster = models.ForeignKey(ClusterResult, on_delete=models.SET_NULL, null=True, blank=True,
|
||||
related_name='entities')
|
||||
feature_json = models.JSONField(default=dict, blank=True,
|
||||
help_text="Aggregated entity features as dict")
|
||||
embedding_x = models.FloatField(null=True, blank=True, help_text="PCA-2D X coordinate")
|
||||
embedding_y = models.FloatField(null=True, blank=True, help_text="PCA-2D Y coordinate")
|
||||
|
||||
class Meta:
|
||||
unique_together = ['run', 'entity_value']
|
||||
|
||||
def __str__(self):
|
||||
return f"Entity: {self.entity_value} (Cluster {self.cluster_label})"
|
||||
|
||||
|
||||
class ClusterFeature(models.Model):
|
||||
"""Per-cluster statistical features and distinguishing scores."""
|
||||
cluster = models.ForeignKey(ClusterResult, on_delete=models.CASCADE, related_name='features')
|
||||
feature_name = models.CharField(max_length=256)
|
||||
mean = models.FloatField(null=True, blank=True)
|
||||
std = models.FloatField(null=True, blank=True)
|
||||
median = models.FloatField(null=True, blank=True)
|
||||
p25 = models.FloatField(null=True, blank=True)
|
||||
p75 = models.FloatField(null=True, blank=True)
|
||||
missing_rate = models.FloatField(null=True, blank=True)
|
||||
distinguishing_score = models.FloatField(null=True, blank=True,
|
||||
help_text="Z-score or ANOVA F-score")
|
||||
distinguishing_method = models.CharField(max_length=16, default='zscore')
|
||||
|
||||
class Meta:
|
||||
unique_together = ['cluster', 'feature_name']
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.feature_name} (score={self.distinguishing_score:.3f})"
|
||||
@@ -0,0 +1,251 @@
|
||||
"""Thread-safe in-memory session store for datasets and results.
|
||||
|
||||
Singleton pattern with RLock guard. Stores:
|
||||
- Datasets: dataset_id -> (LazyFrame, schema, metadata)
|
||||
- Cluster results: result_id -> (labels, n_clusters, params, parent_dataset_id)
|
||||
- Feature results: result_id -> (features, parent_cluster_result_id)
|
||||
"""
|
||||
import threading
|
||||
import gc
|
||||
import uuid
|
||||
from typing import Any, Optional
|
||||
|
||||
import polars as pl
|
||||
|
||||
|
||||
# Approximate per-element byte sizes for memory estimation
|
||||
_DTYPE_BYTE_SIZES = {
|
||||
# Polars dtype strings
|
||||
'Int8': 1, 'Int16': 2, 'Int32': 4, 'Int64': 8,
|
||||
'UInt8': 1, 'UInt16': 2, 'UInt32': 4, 'UInt64': 8,
|
||||
'Float32': 4, 'Float64': 8,
|
||||
'Boolean': 1, 'Bool': 1,
|
||||
'Utf8': 50, 'String': 50, 'Categorical': 8,
|
||||
'Date': 4, 'Datetime': 8, 'Time': 8, 'Duration': 8,
|
||||
'List': 16, 'Object': 16, 'Null': 0,
|
||||
# Polars __str__ representations
|
||||
'i32': 4, 'i64': 8, 'u32': 4, 'u64': 8,
|
||||
'f32': 4, 'f64': 8, 'str': 50, 'bool': 1,
|
||||
}
|
||||
|
||||
|
||||
def _estimate_dtype_bytes(dtype: Any) -> int:
|
||||
"""Guess per-element byte size from a Polars dtype."""
|
||||
key = str(dtype)
|
||||
for prefix, size in _DTYPE_BYTE_SIZES.items():
|
||||
if key.startswith(prefix) or key == prefix:
|
||||
return size
|
||||
return 8 # fallback
|
||||
|
||||
|
||||
def _generate_id(prefix: str = 'ds') -> str:
|
||||
"""Generate a short unique identifier."""
|
||||
return f"{prefix}_{uuid.uuid4().hex[:12]}"
|
||||
|
||||
|
||||
class SessionStore:
|
||||
"""Singleton in-memory store for datasets and analysis results.
|
||||
|
||||
Thread-safe via RLock. Datasets store Polars LazyFrames (query plans,
|
||||
not materialised data), so clone/shallow-copy is effectively free.
|
||||
"""
|
||||
|
||||
_instance: Optional['SessionStore'] = None
|
||||
_lock: threading.RLock = threading.RLock()
|
||||
|
||||
def __new__(cls) -> 'SessionStore':
|
||||
if cls._instance is None:
|
||||
with cls._lock:
|
||||
if cls._instance is None:
|
||||
cls._instance = super().__new__(cls)
|
||||
cls._instance._stores: dict[str, dict[str, Any]] = {}
|
||||
cls._instance._datalock = threading.RLock()
|
||||
return cls._instance
|
||||
|
||||
# ── dataset operations ──────────────────────────────────────────────
|
||||
|
||||
def store_dataset(
|
||||
self,
|
||||
dataset_id: str,
|
||||
lazyframe: pl.LazyFrame,
|
||||
schema: Optional[dict] = None,
|
||||
metadata: Optional[dict] = None,
|
||||
parent_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Store a LazyFrame under *dataset_id*.
|
||||
|
||||
Args:
|
||||
dataset_id: Unique key for the dataset.
|
||||
lazyframe: Polars LazyFrame (query plan, not materialised).
|
||||
schema: Column schema dict {name: dtype_string}.
|
||||
metadata: Optional dict (row_count, file_count, …).
|
||||
parent_id: If this is a derived dataset (filter/clone/…), the
|
||||
source dataset_id.
|
||||
"""
|
||||
with self._datalock:
|
||||
self._stores[dataset_id] = {
|
||||
'type': 'dataset',
|
||||
'lazyframe': lazyframe,
|
||||
'schema': schema or {},
|
||||
'metadata': metadata or {},
|
||||
'parent_id': parent_id,
|
||||
}
|
||||
|
||||
def get_dataset(self, dataset_id: str) -> Optional[dict]:
|
||||
"""Retrieve a stored dataset entry (or *None*)."""
|
||||
with self._datalock:
|
||||
entry = self._stores.get(dataset_id)
|
||||
if entry and entry.get('type') == 'dataset':
|
||||
return entry
|
||||
return None
|
||||
|
||||
def drop_dataset(self, dataset_id: str) -> bool:
|
||||
"""Remove a dataset from the store and force garbage collection.
|
||||
|
||||
Returns True if the dataset existed.
|
||||
"""
|
||||
with self._datalock:
|
||||
existed = dataset_id in self._stores and self._stores[dataset_id].get('type') == 'dataset'
|
||||
if existed:
|
||||
del self._stores[dataset_id]
|
||||
if existed:
|
||||
gc.collect()
|
||||
return existed
|
||||
|
||||
def clone_dataset(self, dataset_id: str) -> Optional[str]:
|
||||
"""Shallow clone — LazyFrame is a query plan so no data is copied.
|
||||
|
||||
Returns the new dataset_id, or *None* if source does not exist.
|
||||
"""
|
||||
with self._datalock:
|
||||
entry = self.get_dataset(dataset_id)
|
||||
if entry is None:
|
||||
return None
|
||||
new_id = _generate_id('clone')
|
||||
self._stores[new_id] = {
|
||||
'type': 'dataset',
|
||||
'lazyframe': entry['lazyframe'], # same plan pointer
|
||||
'schema': dict(entry['schema']),
|
||||
'metadata': dict(entry.get('metadata', {})),
|
||||
'parent_id': dataset_id,
|
||||
}
|
||||
return new_id
|
||||
|
||||
def list_datasets(self) -> list[dict]:
|
||||
"""Return summary dict for every stored dataset."""
|
||||
with self._datalock:
|
||||
results = []
|
||||
for ds_id, entry in self._stores.items():
|
||||
if entry.get('type') != 'dataset':
|
||||
continue
|
||||
meta = entry.get('metadata', {})
|
||||
results.append({
|
||||
'dataset_id': ds_id,
|
||||
'parent_id': entry.get('parent_id'),
|
||||
'row_count': meta.get('row_count'),
|
||||
'file_count': meta.get('file_count'),
|
||||
'columns': list(entry.get('schema', {}).keys()),
|
||||
'column_count': len(entry.get('schema', {})),
|
||||
})
|
||||
return results
|
||||
|
||||
def memory_estimate_mb(self, dataset_id: str) -> float:
|
||||
"""Rough memory estimate based on schema × row_count."""
|
||||
with self._datalock:
|
||||
entry = self.get_dataset(dataset_id)
|
||||
if entry is None:
|
||||
return 0.0
|
||||
schema = entry.get('schema', {})
|
||||
row_count = entry.get('metadata', {}).get('row_count', 0)
|
||||
if not row_count or not schema:
|
||||
return 0.0
|
||||
bytes_per_row = sum(_estimate_dtype_bytes(dtype) for dtype in schema.values())
|
||||
return (bytes_per_row * row_count) / (1024 * 1024)
|
||||
|
||||
# ── cluster-result operations ───────────────────────────────────────
|
||||
|
||||
def store_cluster_result(
|
||||
self,
|
||||
result_id: str,
|
||||
labels: list[int],
|
||||
n_clusters: int,
|
||||
params: dict[str, Any],
|
||||
parent_dataset_id: str,
|
||||
) -> None:
|
||||
"""Store a clustering result."""
|
||||
with self._datalock:
|
||||
self._stores[result_id] = {
|
||||
'type': 'cluster_result',
|
||||
'labels': labels,
|
||||
'n_clusters': n_clusters,
|
||||
'params': params,
|
||||
'parent_dataset_id': parent_dataset_id,
|
||||
}
|
||||
|
||||
def get_cluster_result(self, result_id: str) -> Optional[dict]:
|
||||
"""Retrieve a stored cluster result."""
|
||||
with self._datalock:
|
||||
entry = self._stores.get(result_id)
|
||||
if entry and entry.get('type') == 'cluster_result':
|
||||
return entry
|
||||
return None
|
||||
|
||||
def drop_cluster_result(self, result_id: str) -> bool:
|
||||
"""Remove a cluster result."""
|
||||
with self._datalock:
|
||||
existed = result_id in self._stores and self._stores[result_id].get('type') == 'cluster_result'
|
||||
if existed:
|
||||
del self._stores[result_id]
|
||||
return existed
|
||||
|
||||
# ── feature-result operations ───────────────────────────────────────
|
||||
|
||||
def store_feature_result(
|
||||
self,
|
||||
result_id: str,
|
||||
features: list[dict[str, Any]],
|
||||
parent_cluster_result_id: str,
|
||||
) -> None:
|
||||
"""Store extracted features per cluster."""
|
||||
with self._datalock:
|
||||
self._stores[result_id] = {
|
||||
'type': 'feature_result',
|
||||
'features': features,
|
||||
'parent_cluster_result_id': parent_cluster_result_id,
|
||||
}
|
||||
|
||||
def get_feature_result(self, result_id: str) -> Optional[dict]:
|
||||
"""Retrieve a stored feature result."""
|
||||
with self._datalock:
|
||||
entry = self._stores.get(result_id)
|
||||
if entry and entry.get('type') == 'feature_result':
|
||||
return entry
|
||||
return None
|
||||
|
||||
def drop_feature_result(self, result_id: str) -> bool:
|
||||
"""Remove a feature result."""
|
||||
with self._datalock:
|
||||
existed = result_id in self._stores and self._stores[result_id].get('type') == 'feature_result'
|
||||
if existed:
|
||||
del self._stores[result_id]
|
||||
return existed
|
||||
|
||||
# ── general ─────────────────────────────────────────────────────────
|
||||
|
||||
def drop_all(self) -> int:
|
||||
"""Drop every entry and run gc. Returns count of removed entries."""
|
||||
with self._datalock:
|
||||
count = len(self._stores)
|
||||
self._stores.clear()
|
||||
if count:
|
||||
gc.collect()
|
||||
return count
|
||||
|
||||
def __repr__(self) -> str:
|
||||
with self._datalock:
|
||||
types = {}
|
||||
for entry in self._stores.values():
|
||||
t = entry.get('type', 'unknown')
|
||||
types[t] = types.get(t, 0) + 1
|
||||
parts = [f"{k}={v}" for k, v in sorted(types.items())]
|
||||
return f"<SessionStore {'; '.join(parts)}>"
|
||||
@@ -0,0 +1,12 @@
|
||||
from django import template
|
||||
|
||||
register = template.Library()
|
||||
|
||||
|
||||
@register.filter
|
||||
def getlist(value, key):
|
||||
"""Get all values for a key from QueryDict and join with ', '."""
|
||||
if hasattr(value, 'getlist'):
|
||||
vals = value.getlist(key)
|
||||
return ', '.join(vals) if vals else ''
|
||||
return ''
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,350 @@
|
||||
"""Data type classifier for column-level type inference.
|
||||
|
||||
Provides :class:`DataType` enumeration and functions to classify Polars
|
||||
columns based on name heuristics and value sampling.
|
||||
|
||||
Typical usage::
|
||||
|
||||
from analysis.type_classifier import classify_column, classify_schema, DataType
|
||||
|
||||
# Single column
|
||||
dtype = classify_column('src_ip', series, config_type='ipv4')
|
||||
|
||||
# Full schema
|
||||
type_map = classify_schema(lazyframe, config_overrides={'src_ip': 'ipv4'})
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from enum import Enum
|
||||
|
||||
import polars as pl
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DataType enumeration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class DataType(Enum):
|
||||
"""Enumerated column types recognised by the classifier engine."""
|
||||
|
||||
INT = 1
|
||||
FLOAT = 2
|
||||
ENUM = 3
|
||||
HEX = 4
|
||||
URL = 5
|
||||
IPv4 = 6
|
||||
LAT_LON = 7
|
||||
TIMESTAMP = 8
|
||||
BOOL_ENUM = 9
|
||||
STRING = 10
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Detection patterns (raw strings for regex)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
HEX_PATTERN = r'^[0-9a-f]{2}(?:\s[0-9a-f]{2})*$'
|
||||
"""Matches ``"0e a6 3f"``, ``"ab cd ef"`` etc. (space-separated hex pairs)."""
|
||||
|
||||
URL_PATTERN = r'^https?://'
|
||||
"""Matches URLs starting with ``http://`` or ``https://``."""
|
||||
|
||||
IPv4_PATTERN = r'^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'
|
||||
"""Matches ``"192.168.1.1"`` style IPv4 addresses."""
|
||||
|
||||
MAC_PATTERN = r'^([0-9A-Fa-f]{2}[:.-]){5}[0-9A-Fa-f]{2}$'
|
||||
"""Matches MAC addresses with ``:``, ``-``, or ``.`` separators."""
|
||||
|
||||
BOOL_ENUM_VALUES = (
|
||||
'', ' ', '+', '-', '0', '1',
|
||||
'true', 'false', 'yes', 'no', 't', 'f',
|
||||
)
|
||||
"""String values considered boolean / binary indicators."""
|
||||
|
||||
LAT_LON_NAMES = ('lat', 'latitude', 'lon', 'lng', 'longitude')
|
||||
|
||||
# Name-to-type heuristics (checked after config override, before values)
|
||||
_NAME_TYPE_MAP: list[tuple[re.Pattern, DataType]] = [
|
||||
# User column names - EXACT match only (no generic fuzzy patterns)
|
||||
# IPv4 columns
|
||||
(re.compile(r'^:ips$|^:ipd$|^server.ip$|^client.ip$', re.I), DataType.IPv4),
|
||||
# ENUM columns
|
||||
(re.compile(r'^:prs$|^:prd$', re.I), DataType.ENUM),
|
||||
(re.compile(r'^(?:scnt|dcnt|cnrs|isrs|1ipp|4dbn)$', re.I), DataType.ENUM),
|
||||
(re.compile(r'^0ver$', re.I), DataType.ENUM),
|
||||
# FLOAT columns
|
||||
(re.compile(r'^4dur$|^8ses$|^2tmo$|^0rnt$', re.I), DataType.FLOAT),
|
||||
# INT columns
|
||||
(re.compile(r'^4ksz$|^8ack$|^8ppk$|^row$', re.I), DataType.INT),
|
||||
# HEX columns
|
||||
(re.compile(r'^0cph$|^0crv$', re.I), DataType.HEX),
|
||||
# TIMESTAMP columns
|
||||
(re.compile(r'^8dbd$|^time$|^timestamp$', re.I), DataType.TIMESTAMP),
|
||||
# Dot-notation suffixes for lat/lon (e.g. :ips.latd, :ipd.lond)
|
||||
(re.compile(r'\.latd$', re.I), DataType.LAT_LON),
|
||||
(re.compile(r'\.lond$', re.I), DataType.LAT_LON),
|
||||
# Dot-notation STRING suffixes (e.g. :ips.ispn, :ipd.city)
|
||||
(re.compile(r'\.(ispn|orgn|city)$', re.I), DataType.STRING),
|
||||
# Standalone string columns - exact match
|
||||
(re.compile(r'^(?:cnam|snam|tabl|name|0rnd|source.node|ecdhe.named.curve)$', re.I), DataType.STRING),
|
||||
]
|
||||
|
||||
# Mapping from config.yaml type strings to DataType
|
||||
_CONFIG_TYPE_MAP: dict[str, DataType] = {
|
||||
'int': DataType.INT,
|
||||
'float': DataType.FLOAT,
|
||||
'enum': DataType.ENUM,
|
||||
'hex': DataType.HEX,
|
||||
'url': DataType.URL,
|
||||
'ipv4': DataType.IPv4,
|
||||
'lat_lon': DataType.LAT_LON,
|
||||
'timestamp': DataType.TIMESTAMP,
|
||||
'bool_enum': DataType.BOOL_ENUM,
|
||||
'string': DataType.STRING,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Column classification (single column)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _name_to_type(name: str) -> DataType | None:
|
||||
"""Match column name against known heuristics.
|
||||
|
||||
Returns a :class:`DataType` if the name matches strongly, otherwise
|
||||
``None`` (fall through to value-based detection).
|
||||
"""
|
||||
for pattern, dtype in _NAME_TYPE_MAP:
|
||||
if pattern.search(name):
|
||||
return dtype
|
||||
return None
|
||||
|
||||
|
||||
def _is_port_value(v: str) -> bool:
|
||||
"""Check if *v* is an integer string in the registered port range (1-65535)."""
|
||||
try:
|
||||
n = int(v)
|
||||
return 1 <= n <= 65535
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _can_float(v: str) -> bool:
|
||||
"""Return ``True`` when *v* can be parsed as a Python float."""
|
||||
try:
|
||||
float(v)
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _values_to_type(series: pl.Series) -> DataType:
|
||||
"""Infer column type by sampling non-null values.
|
||||
|
||||
Detection order: IPv4 → MAC guard → HEX → URL → PORT → BOOL_ENUM
|
||||
→ LAT_LON → ENUM → FLOAT → STRING
|
||||
"""
|
||||
dtype = series.dtype
|
||||
|
||||
# Boolean dtype ──────────────────────────────────────────────────────
|
||||
if dtype == pl.Boolean:
|
||||
return DataType.BOOL_ENUM
|
||||
|
||||
# Numeric dtypes ─────────────────────────────────────────────────────
|
||||
if dtype in (pl.Float32, pl.Float64):
|
||||
# Check LAT_LON possibility before returning FLOAT
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) > 0:
|
||||
sample = non_null.head(100)
|
||||
sample_strs = [str(v).strip().lower() for v in sample]
|
||||
non_blank = [v for v in sample_strs if v not in ('', ' ', '+', '-')]
|
||||
if non_blank:
|
||||
try:
|
||||
floats = [float(v) for v in non_blank]
|
||||
max_abs = max(abs(f) for f in floats)
|
||||
if (
|
||||
all(-180 <= f <= 180 for f in floats)
|
||||
and any('.' in v for v in non_blank)
|
||||
and max_abs > 10
|
||||
):
|
||||
return DataType.LAT_LON
|
||||
except ValueError:
|
||||
pass
|
||||
return DataType.FLOAT
|
||||
|
||||
if dtype in (pl.Int8, pl.Int16, pl.Int32, pl.Int64,
|
||||
pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64):
|
||||
return DataType.INT
|
||||
|
||||
# String / categorical types ─────────────────────────────────────────
|
||||
non_null = series.drop_nulls()
|
||||
if len(non_null) == 0:
|
||||
return DataType.STRING
|
||||
|
||||
sample = non_null.head(100)
|
||||
# Convert all to string representations for pattern matching
|
||||
sample_strs = [str(v).strip().lower() for v in sample]
|
||||
|
||||
# ── 1. IPv4 detection ──────────────────────────────────────────────
|
||||
if all(re.match(IPv4_PATTERN, v) for v in sample_strs):
|
||||
if all(_valid_ip_octets(v) for v in sample_strs):
|
||||
return DataType.IPv4
|
||||
|
||||
# ── 2. MAC guard (skip HEX if values look like MAC addresses) ──────
|
||||
is_mac = all(re.match(MAC_PATTERN, v) for v in sample_strs)
|
||||
|
||||
# ── 3. HEX detection ───────────────────────────────────────────────
|
||||
if not is_mac and all(re.match(HEX_PATTERN, v) for v in sample_strs):
|
||||
return DataType.HEX
|
||||
|
||||
# ── 4. URL detection ───────────────────────────────────────────────
|
||||
if all(re.match(URL_PATTERN, v) for v in sample_strs):
|
||||
return DataType.URL
|
||||
|
||||
# ── 5. PORT detection → ENUM ───────────────────────────────────────
|
||||
if all(_is_port_value(v) for v in sample_strs):
|
||||
return DataType.ENUM
|
||||
|
||||
# ── 6. BOOL_ENUM detection ─────────────────────────────────────────
|
||||
if all(v in BOOL_ENUM_VALUES for v in sample_strs):
|
||||
return DataType.BOOL_ENUM
|
||||
|
||||
# ── 7. LAT_LON detection (value-based) ──────────────────────────────
|
||||
non_blank = [v for v in sample_strs if v not in ('', ' ', '+', '-')]
|
||||
if non_blank:
|
||||
try:
|
||||
floats = [float(v) for v in non_blank]
|
||||
max_abs = max(abs(f) for f in floats)
|
||||
if (
|
||||
all(-180 <= f <= 180 for f in floats)
|
||||
and any('.' in v for v in non_blank)
|
||||
and max_abs > 10
|
||||
):
|
||||
return DataType.LAT_LON
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# ── 8. ENUM detection (few unique values) ──────────────────────────
|
||||
unique_count = non_null.n_unique()
|
||||
if unique_count < 20:
|
||||
return DataType.ENUM
|
||||
|
||||
# ── 9. FLOAT detection (>80 % can float) ───────────────────────────
|
||||
float_count = sum(1 for v in sample_strs if _can_float(v))
|
||||
if float_count / len(sample_strs) > 0.8:
|
||||
# Guard: values with leading "+"/"-" (e.g., "+5", "-abc") that
|
||||
# happen to parse as float should not force a FLOAT classification
|
||||
# when non-float values exist in the column. This catches string
|
||||
# columns where a "+" prefix is used as a text marker (phone codes,
|
||||
# IDs, error codes) rather than a numeric sign.
|
||||
if any(v.startswith('+') or v.startswith('-') for v in sample_strs):
|
||||
if float_count < len(sample_strs):
|
||||
return DataType.STRING
|
||||
return DataType.FLOAT
|
||||
|
||||
return DataType.STRING
|
||||
|
||||
|
||||
def _valid_ip_octets(ip_str: str) -> bool:
|
||||
"""Check that each octet in an IPv4 string is 0-255."""
|
||||
parts = ip_str.split('.')
|
||||
if len(parts) != 4:
|
||||
return False
|
||||
for p in parts:
|
||||
try:
|
||||
n = int(p)
|
||||
if n < 0 or n > 255:
|
||||
return False
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def classify_column(
|
||||
name: str,
|
||||
series: pl.Series,
|
||||
config_type: str | None = None,
|
||||
) -> DataType:
|
||||
"""Classify a single column based on its *name* and *series* values.
|
||||
|
||||
Priority
|
||||
--------
|
||||
1. **config_type** — user override from ``config.yaml``.
|
||||
If provided (e.g. ``'ipv4'``), returns the corresponding
|
||||
:class:`DataType` immediately.
|
||||
2. **Value-based** pattern matching — samples up to 100 non-null
|
||||
values and tests against IPv4 → HEX → URL → PORT → BOOL_ENUM
|
||||
→ LAT_LON → ENUM → FLOAT in order.
|
||||
3. **Name-based** heuristics — only consulted when value-based
|
||||
detection returned ``STRING`` (i.e. no value pattern matched).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
name:
|
||||
Column name (used for name-based heuristics).
|
||||
series:
|
||||
Polars :class:`Series` containing the column's data.
|
||||
config_type:
|
||||
Optional user override (case-insensitive, matched to
|
||||
:data:`_CONFIG_TYPE_MAP`).
|
||||
|
||||
Returns
|
||||
-------
|
||||
DataType
|
||||
"""
|
||||
# 1. Config override (highest priority)
|
||||
if config_type is not None:
|
||||
mapped = _CONFIG_TYPE_MAP.get(config_type.lower().strip())
|
||||
if mapped is not None:
|
||||
return mapped
|
||||
|
||||
# 2. Value-based sampling
|
||||
result = _values_to_type(series)
|
||||
|
||||
# 3. Name-based heuristics (only when values returned STRING)
|
||||
if result == DataType.STRING:
|
||||
name_match = _name_to_type(name)
|
||||
if name_match is not None:
|
||||
return name_match
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema classification (all columns)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def classify_schema(
|
||||
lf: pl.LazyFrame,
|
||||
config_overrides: dict[str, str] | None = None,
|
||||
) -> dict[str, DataType]:
|
||||
"""Classify all columns in a :class:`LazyFrame`.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lf:
|
||||
A Polars :class:`LazyFrame` whose columns to classify.
|
||||
config_overrides:
|
||||
Optional mapping of ``{col_name: type_string}`` from a
|
||||
``config.yaml`` ``columns:`` section. These take highest
|
||||
priority (see :func:`classify_column`).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict[str, DataType]
|
||||
Mapping from column name to inferred :class:`DataType`.
|
||||
"""
|
||||
config_overrides = config_overrides or {}
|
||||
df_sample = lf.collect(streaming=True)
|
||||
result: dict[str, DataType] = {}
|
||||
for col_name in df_sample.columns:
|
||||
config_type = config_overrides.get(col_name)
|
||||
result[col_name] = classify_column(
|
||||
name=col_name,
|
||||
series=df_sample[col_name],
|
||||
config_type=config_type,
|
||||
)
|
||||
return result
|
||||
@@ -0,0 +1,24 @@
|
||||
from django.urls import path
|
||||
from . import views
|
||||
|
||||
app_name = 'analysis'
|
||||
urlpatterns = [
|
||||
path('', views.dashboard, name='dashboard'),
|
||||
path('upload/', views.upload_page, name='upload'),
|
||||
path('upload/csv/', views.upload_csv, name='upload_csv'),
|
||||
path('analyze/manual/', views.manual_page, name='manual'),
|
||||
path('analyze/run/', views.run_analysis, name='run_analysis'),
|
||||
path('analyze/auto/', views.auto_page, name='auto'),
|
||||
path('analyze/llm/', views.run_llm_analysis_view, name='run_llm_analysis'),
|
||||
path('runs/', views.run_list, name='run_list'),
|
||||
path('runs/<int:run_id>/', views.run_detail, name='run_detail'),
|
||||
path('runs/<int:run_id>/status/', views.run_status_api, name='run_status'),
|
||||
path('clusters/<int:run_id>/', views.cluster_overview, name='cluster_overview'),
|
||||
path('clusters/<int:run_id>/<int:cluster_label>/', views.cluster_detail, name='cluster_detail'),
|
||||
path('entities/<int:entity_id>/', views.entity_profile, name='entity_profile'),
|
||||
path('config/', views.config_view, name='config'),
|
||||
path('api/llm/test/', views.llm_test, name='llm_test'),
|
||||
path('runs/<int:run_id>/delete/', views.delete_upload, name='delete_upload'),
|
||||
path('logs/', views.log_viewer, name='log_viewer'),
|
||||
path('globe/', views.globe_view, name='globe'),
|
||||
]
|
||||
@@ -0,0 +1,308 @@
|
||||
"""TLS value normalization — maps hex wire format, IANA names, and display
|
||||
names to unified underscore-connected enum names.
|
||||
|
||||
Usage::
|
||||
|
||||
from analysis.value_normalizer import normalize_lf
|
||||
|
||||
lf = pl.LazyFrame({...})
|
||||
type_map = classify_schema(lf) # {col_name: DataType}
|
||||
lf = normalize_lf(lf, type_map)
|
||||
# -> LazyFrame with added {col}_norm columns
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import polars as pl
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# TLS Version map — hex wire format & display names
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
TLS_VERSION_MAP: dict[str, str] = {
|
||||
# hex wire format (space-separated hex pairs)
|
||||
'03 04': 'tls_1_3',
|
||||
'03 03': 'tls_1_2',
|
||||
'03 02': 'tls_1_1',
|
||||
'03 01': 'tls_1_0',
|
||||
'02 00': 'ssl_2_0',
|
||||
'03 00': 'ssl_3_0',
|
||||
# display formats
|
||||
'tlsv1.3': 'tls_1_3',
|
||||
'tls 1.3': 'tls_1_3',
|
||||
'1.3': 'tls_1_3',
|
||||
'tlsv1.2': 'tls_1_2',
|
||||
'tls 1.2': 'tls_1_2',
|
||||
'1.2': 'tls_1_2',
|
||||
'tlsv1.1': 'tls_1_1',
|
||||
'tls 1.1': 'tls_1_1',
|
||||
'1.1': 'tls_1_1',
|
||||
'tlsv1.0': 'tls_1_0',
|
||||
'tls 1.0': 'tls_1_0',
|
||||
'1.0': 'tls_1_0',
|
||||
'tlsv1': 'tls_1_0',
|
||||
'tls 1': 'tls_1_0',
|
||||
'1': 'tls_1_0',
|
||||
'sslv2': 'ssl_2_0',
|
||||
'ssl 2': 'ssl_2_0',
|
||||
'2': 'ssl_2_0',
|
||||
'sslv3': 'ssl_3_0',
|
||||
'ssl 3': 'ssl_3_0',
|
||||
'3': 'ssl_3_0',
|
||||
}
|
||||
"""TLS version mapping.
|
||||
|
||||
Keys are hex wire format (``'03 04'``) or display names (``'TLSv1.3'``,
|
||||
``'tls 1.3'``, etc.). Matching is case-insensitive. Unmatched values
|
||||
produce ``'unknown_version'``.
|
||||
"""
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# Cipher Suite maps
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
CIPHER_SUITE_MAP: dict[str, str] = {
|
||||
# hex wire format → canonical underscore name
|
||||
'13 01': 'tls_aes_128_gcm_sha256',
|
||||
'13 02': 'tls_aes_256_gcm_sha384',
|
||||
'13 03': 'tls_chacha20_poly1305_sha256',
|
||||
'c0 2b': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
|
||||
'c0 2c': 'tls_ecdhe_ecdsa_aes_256_gcm_sha384',
|
||||
'c0 2d': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
|
||||
'c0 2f': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
|
||||
'c0 30': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
|
||||
'cc a9': 'tls_ecdhe_ecdsa_chacha20_poly1305_sha256',
|
||||
'cc ac': 'tls_ecdhe_rsa_chacha20_poly1305_sha256',
|
||||
'00 9c': 'tls_rsa_aes_128_gcm_sha256',
|
||||
'00 9d': 'tls_rsa_aes_256_gcm_sha384',
|
||||
'00 3c': 'tls_rsa_aes_128_cbc_sha256',
|
||||
'00 3d': 'tls_rsa_aes_256_cbc_sha256',
|
||||
'c0 09': 'tls_ecdhe_ecdsa_aes_128_cbc_sha256',
|
||||
'c0 13': 'tls_ecdhe_rsa_aes_128_cbc_sha256',
|
||||
'00 35': 'tls_rsa_aes_128_cbc_sha',
|
||||
'00 2f': 'tls_rsa_aes_128_cbc_sha',
|
||||
'00 05': 'tls_rsa_rc4_128_sha',
|
||||
'00 ff': 'tls_rsa_null_md5',
|
||||
}
|
||||
"""Cipher suite hex → canonical name. Unmatched hex produces
|
||||
``'unknown_cipher_0x{hex}'``."""
|
||||
|
||||
_CIPHER_DISPLAY_NORMALIZED: dict[str, str] = {
|
||||
# Normalized key (lowercase, separators removed) → canonical name
|
||||
'tlsaes128gcmsha256': 'tls_aes_128_gcm_sha256',
|
||||
'tlsaes256gcmsha384': 'tls_aes_256_gcm_sha384',
|
||||
'aes128gcm': 'tls_aes_128_gcm_sha256',
|
||||
'aes256gcm': 'tls_aes_256_gcm_sha384',
|
||||
'chacha20poly1305': 'tls_chacha20_poly1305_sha256',
|
||||
'ecdheecdsaaes128gcm': 'tls_ecdhe_ecdsa_aes_128_gcm_sha256',
|
||||
'ecdhersaaes128gcm': 'tls_ecdhe_rsa_aes_128_gcm_sha256',
|
||||
'ecdhersaaes256gcm': 'tls_ecdhe_rsa_aes_256_gcm_sha384',
|
||||
'rsaaes128gcm': 'tls_rsa_aes_128_gcm_sha256',
|
||||
'rsaaes256gcm': 'tls_rsa_aes_256_gcm_sha384',
|
||||
}
|
||||
"""Cipher suite display-name normalisation lookup.
|
||||
|
||||
Keys are IANA/display names with separators removed and lowercased
|
||||
(e.g. ``'TLS_AES_128_GCM_SHA256'`` → ``'tlsaes128gmcsha256'``).
|
||||
"""
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# Named Curve maps
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
CURVE_MAP: dict[str, str] = {
|
||||
# hex wire format → canonical name
|
||||
'00 1d': 'x25519',
|
||||
'00 17': 'secp256r1',
|
||||
'00 18': 'secp384r1',
|
||||
'00 19': 'secp521r1',
|
||||
'00 1e': 'x448',
|
||||
# display names
|
||||
'x25519': 'x25519',
|
||||
'curve25519': 'x25519',
|
||||
'secp256r1': 'secp256r1',
|
||||
'prime256v1': 'secp256r1',
|
||||
'p256': 'secp256r1',
|
||||
'p-256': 'secp256r1',
|
||||
'p_256': 'secp256r1',
|
||||
'secp384r1': 'secp384r1',
|
||||
'p384': 'secp384r1',
|
||||
'p-384': 'secp384r1',
|
||||
'p_384': 'secp384r1',
|
||||
'secp521r1': 'secp521r1',
|
||||
'p521': 'secp521r1',
|
||||
'p-521': 'secp521r1',
|
||||
'p_521': 'secp521r1',
|
||||
'x448': 'x448',
|
||||
'curve448': 'x448',
|
||||
}
|
||||
"""Named curve mapping.
|
||||
|
||||
Keys are hex wire format (``'00 1d'``) or display names (``'X25519'``,
|
||||
``'prime256v1'``, etc.). Matching is case-insensitive.
|
||||
"""
|
||||
|
||||
|
||||
def _build_when_chain(
|
||||
v: pl.Expr,
|
||||
mapping: dict[str, str],
|
||||
fallback: pl.Expr,
|
||||
) -> pl.Expr:
|
||||
"""Build a ``pl.when()...then()...otherwise()`` chain from *mapping*.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
v:
|
||||
Polars expression representing the column value (already
|
||||
lowercased and stripped).
|
||||
mapping:
|
||||
``{lookup_key: canonical_value}`` dict.
|
||||
fallback:
|
||||
Expression for unmatched values.
|
||||
|
||||
Returns
|
||||
-------
|
||||
pl.Expr
|
||||
A chain ``pl.when(...).then(...)...otherwise(fallback)``.
|
||||
"""
|
||||
expr = pl.when(v.is_null()).then(pl.lit(None, pl.Utf8))
|
||||
for key, val in mapping.items():
|
||||
expr = expr.when(v == key).then(pl.lit(val))
|
||||
return expr.otherwise(fallback)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# Per-column normalisation functions
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _norm_tls_version(col: str) -> pl.Expr:
|
||||
"""Normalise a TLS version column (``0ver``).
|
||||
|
||||
Accepts hex wire format (``'03 04'``) or display names
|
||||
(``'TLSv1.3'``, ``'tls 1.3'``, ``'1.3'``, ``'SSLv3'``, etc.).
|
||||
Unmatched values produce ``'unknown_version'``.
|
||||
"""
|
||||
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
|
||||
return _build_when_chain(v, TLS_VERSION_MAP, pl.lit('unknown_version'))
|
||||
|
||||
|
||||
def _norm_cipher(col: str) -> pl.Expr:
|
||||
"""Normalise a cipher-suite hex column (``0cph``).
|
||||
|
||||
Unmatched hex values produce ``'unknown_cipher_0x{hex}'`` where
|
||||
``{hex}`` is the original (stripped, lowercased) value.
|
||||
"""
|
||||
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
|
||||
fallback = pl.lit('unknown_cipher_0x') + v
|
||||
return _build_when_chain(v, CIPHER_SUITE_MAP, fallback)
|
||||
|
||||
|
||||
def _norm_cipher_display(col: str) -> pl.Expr:
|
||||
"""Normalise a cipher-suite display-name column (``cipher-suite``).
|
||||
|
||||
The column value is lowercased and all separators (spaces, hyphens,
|
||||
underscores) are removed before lookup.
|
||||
|
||||
Unmatched names produce ``'{normalised}_unknown'``.
|
||||
"""
|
||||
raw = pl.col(col).cast(pl.Utf8).str.strip_chars()
|
||||
v = raw.str.to_lowercase().str.replace_all(r'[\s\-_]', '')
|
||||
fallback = v + pl.lit('_unknown')
|
||||
return _build_when_chain(v, _CIPHER_DISPLAY_NORMALIZED, fallback)
|
||||
|
||||
|
||||
def _norm_curve(col: str) -> pl.Expr:
|
||||
"""Normalise a named-curve hex column (``0crv``).
|
||||
|
||||
Unmatched hex values produce ``'unknown_curve_0x{hex}'``.
|
||||
"""
|
||||
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
|
||||
fallback = pl.lit('unknown_curve_0x') + v
|
||||
return _build_when_chain(v, CURVE_MAP, fallback)
|
||||
|
||||
|
||||
def _norm_curve_display(col: str) -> pl.Expr:
|
||||
"""Normalise a named-curve display-name column (``ecdhe-named-curve``).
|
||||
|
||||
Accepts names like ``'X25519'``, ``'prime256v1'``, ``'P-256'``, etc.
|
||||
Unmatched names produce ``'unknown_curve_{name}'`` where ``{name}``
|
||||
is the original value lowercased.
|
||||
"""
|
||||
v = pl.col(col).cast(pl.Utf8).str.strip_chars().str.to_lowercase()
|
||||
fallback = pl.lit('unknown_curve_') + v
|
||||
return _build_when_chain(v, CURVE_MAP, fallback)
|
||||
|
||||
|
||||
def _norm_cnrs(col: str) -> pl.Expr:
|
||||
"""Normalise a CNRS / ISRS column.
|
||||
|
||||
``'+'`` (optionally surrounded by whitespace) → ``'yes'``.
|
||||
Blank, null, or any other value → ``'no'``.
|
||||
|
||||
These columns are **never** missing; blank always means ``'no'``.
|
||||
"""
|
||||
c = pl.col(col).cast(pl.Utf8).str.strip_chars()
|
||||
return (
|
||||
pl.when(c.is_null() | (c == ''))
|
||||
.then(pl.lit('no'))
|
||||
.when(c.str.to_lowercase() == '+')
|
||||
.then(pl.lit('yes'))
|
||||
.otherwise(pl.lit('no'))
|
||||
)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
# Public API
|
||||
# ═════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def normalize_lf(lf: pl.LazyFrame, type_map: dict) -> pl.LazyFrame:
|
||||
"""Add ``{col}_norm`` columns with normalised values.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lf:
|
||||
A Polars :class:`LazyFrame` containing TLS-related columns.
|
||||
type_map:
|
||||
A ``{column_name: ...}`` dict (e.g. from
|
||||
:func:`~analysis.type_classifier.classify_schema`). Column
|
||||
names present in the map receive normalisation.
|
||||
|
||||
Returns
|
||||
-------
|
||||
pl.LazyFrame
|
||||
A new :class:`LazyFrame` with ``{col}_norm`` columns appended.
|
||||
"""
|
||||
exprs: list[pl.Expr] = []
|
||||
|
||||
if '0ver' in type_map:
|
||||
exprs.append(_norm_tls_version('0ver').alias('0ver_norm'))
|
||||
|
||||
if '0cph' in type_map:
|
||||
exprs.append(_norm_cipher('0cph').alias('0cph_norm'))
|
||||
|
||||
if 'cipher-suite' in type_map:
|
||||
exprs.append(
|
||||
_norm_cipher_display('cipher-suite').alias('cipher_suite_norm')
|
||||
)
|
||||
|
||||
if '0crv' in type_map:
|
||||
exprs.append(_norm_curve('0crv').alias('0crv_norm'))
|
||||
|
||||
if 'ecdhe-named-curve' in type_map:
|
||||
exprs.append(
|
||||
_norm_curve_display('ecdhe-named-curve').alias(
|
||||
'ecdhe_named_curve_norm'
|
||||
)
|
||||
)
|
||||
|
||||
if 'cnrs' in type_map:
|
||||
exprs.append(_norm_cnrs('cnrs').alias('cnrs_norm'))
|
||||
|
||||
if 'isrs' in type_map:
|
||||
exprs.append(_norm_cnrs('isrs').alias('isrs_norm'))
|
||||
|
||||
if exprs:
|
||||
lf = lf.with_columns(exprs)
|
||||
return lf
|
||||
@@ -0,0 +1,909 @@
|
||||
import json
|
||||
import time
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import threading
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from django.shortcuts import render, get_object_or_404, redirect
|
||||
from django.http import HttpResponse, JsonResponse
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
|
||||
from config import get_config, save_config, Config
|
||||
from .models import AnalysisRun, ClusterResult, EntityProfile
|
||||
|
||||
|
||||
def dashboard(request):
|
||||
"""Home page: recent runs overview."""
|
||||
runs = AnalysisRun.objects.all()[:20]
|
||||
return render(request, 'analysis/dashboard.html', {'runs': runs})
|
||||
|
||||
|
||||
def run_list(request):
|
||||
"""List all analysis runs with pagination."""
|
||||
page = int(request.GET.get('page', 1))
|
||||
per_page = 20
|
||||
total = AnalysisRun.objects.count()
|
||||
runs = AnalysisRun.objects.all()[(page - 1) * per_page: page * per_page]
|
||||
return render(request, 'analysis/run_list.html', {
|
||||
'runs': runs,
|
||||
'page': page,
|
||||
'total_pages': (total + per_page - 1) // per_page if total else 1,
|
||||
})
|
||||
|
||||
|
||||
def run_detail(request, run_id):
|
||||
"""Details for a single analysis run."""
|
||||
run = get_object_or_404(AnalysisRun, id=run_id)
|
||||
clusters = run.clusters.all().order_by('-size')
|
||||
return render(request, 'analysis/run_detail.html', {
|
||||
'run': run,
|
||||
'clusters': clusters,
|
||||
})
|
||||
|
||||
|
||||
def _extract_lat(val):
|
||||
"""Try to extract a latitude from *val* (number or str)."""
|
||||
if val is None:
|
||||
return None
|
||||
try:
|
||||
v = float(val)
|
||||
return None if v < -90 or v > 90 else v
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _extract_lon(val):
|
||||
"""Try to extract a longitude from *val* (number or str)."""
|
||||
if val is None:
|
||||
return None
|
||||
try:
|
||||
v = float(val)
|
||||
return None if v < -180 or v > 180 else v
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def cluster_overview(request, run_id):
|
||||
"""Overview of all clusters for a run, with PCA scatter data and geo scatter data."""
|
||||
run = get_object_or_404(AnalysisRun, id=run_id)
|
||||
clusters = run.clusters.exclude(cluster_label=-1).order_by('-size')
|
||||
noise = run.clusters.filter(cluster_label=-1).first()
|
||||
|
||||
# Build PCA scatter data for Chart.js
|
||||
entities = run.entities.exclude(embedding_x=None).exclude(embedding_y=None)
|
||||
scatter_data = [
|
||||
{'x': e.embedding_x, 'y': e.embedding_y, 'label': e.cluster_label, 'entity': e.entity_value}
|
||||
for e in entities
|
||||
]
|
||||
|
||||
# Build geo scatter data from feature_json (lat/lon)
|
||||
geo_data = []
|
||||
geo_skipped = 0
|
||||
for e in run.entities.all():
|
||||
fj = e.feature_json
|
||||
if not fj:
|
||||
continue
|
||||
lat = _extract_lat(fj.get('avg_latitude'))
|
||||
lon = _extract_lon(fj.get('avg_longitude'))
|
||||
if lat is not None and lon is not None:
|
||||
geo_data.append({
|
||||
'x': lon,
|
||||
'y': lat,
|
||||
'label': e.cluster_label if e.cluster_label is not None else -1,
|
||||
'entity': e.entity_value,
|
||||
})
|
||||
else:
|
||||
geo_skipped += 1
|
||||
|
||||
# Cluster sizes for bar chart
|
||||
cluster_sizes = [{'label': c.cluster_label, 'size': c.size, 'silhouette': c.silhouette_score}
|
||||
for c in clusters]
|
||||
|
||||
# Top features per cluster from ClusterFeature
|
||||
top_features = {}
|
||||
for c in clusters:
|
||||
features_qs = c.features.all().order_by('-distinguishing_score')[:10]
|
||||
top_features[str(c.cluster_label)] = [
|
||||
{'feature_name': f.feature_name, 'score': f.distinguishing_score, 'mean': f.mean}
|
||||
for f in features_qs
|
||||
]
|
||||
|
||||
return render(request, 'analysis/cluster_overview.html', {
|
||||
'run': run,
|
||||
'clusters': clusters,
|
||||
'noise': noise,
|
||||
'scatter_data_json': json.dumps(scatter_data),
|
||||
'geo_data_json': json.dumps(geo_data),
|
||||
'geo_skipped': geo_skipped,
|
||||
'geo_count': len(geo_data),
|
||||
'cluster_sizes_json': json.dumps(cluster_sizes),
|
||||
'top_features_json': json.dumps(top_features),
|
||||
})
|
||||
|
||||
|
||||
def cluster_detail(request, run_id, cluster_label):
|
||||
"""Detailed view of a single cluster with entity list."""
|
||||
run = get_object_or_404(AnalysisRun, id=run_id)
|
||||
cluster = get_object_or_404(ClusterResult, run=run, cluster_label=cluster_label)
|
||||
features = cluster.features.all().order_by('-distinguishing_score')[:50]
|
||||
entities = cluster.entities.all()[:50]
|
||||
|
||||
return render(request, 'analysis/cluster_detail.html', {
|
||||
'run': run,
|
||||
'cluster': cluster,
|
||||
'features': features,
|
||||
'entities': entities,
|
||||
})
|
||||
|
||||
|
||||
def entity_profile(request, entity_id):
|
||||
"""Detailed profile for a single entity."""
|
||||
entity = get_object_or_404(EntityProfile, id=entity_id)
|
||||
run = entity.run
|
||||
|
||||
# Compute deviation from cluster mean
|
||||
deviations = {}
|
||||
if entity.feature_json and entity.cluster:
|
||||
cluster_features = {
|
||||
f.feature_name: {'mean': f.mean, 'std': f.std}
|
||||
for f in entity.cluster.features.all()
|
||||
}
|
||||
for col, val in entity.feature_json.items():
|
||||
if col in cluster_features and cluster_features[col].get('std') and cluster_features[col]['std'] > 1e-10:
|
||||
deviations[col] = {
|
||||
'value': val,
|
||||
'cluster_mean': cluster_features[col]['mean'],
|
||||
'zscore': (val - cluster_features[col]['mean']) / cluster_features[col]['std'],
|
||||
}
|
||||
|
||||
return render(request, 'analysis/entity_profile.html', {
|
||||
'entity': entity,
|
||||
'run': run,
|
||||
'deviations': deviations,
|
||||
})
|
||||
|
||||
|
||||
def start_analysis(request):
|
||||
"""Placeholder: API endpoint to trigger analysis via MCP."""
|
||||
return JsonResponse({'status': 'not_implemented', 'message': 'Use MCP tools to trigger analysis'})
|
||||
|
||||
|
||||
def config_view(request):
|
||||
"""Config page: GET renders current config, POST saves updates."""
|
||||
cfg = get_config()
|
||||
|
||||
if request.method == 'POST':
|
||||
# Build a new Config from POST data (fall back to current values)
|
||||
server = Config.Server(
|
||||
host=request.POST.get('server_host', cfg.server.host),
|
||||
port=int(request.POST.get('server_port', cfg.server.port)),
|
||||
debug=request.POST.get('server_debug') == 'true',
|
||||
)
|
||||
data = Config.Data(
|
||||
schema_strict=request.POST.get('data_schema_strict') == 'true',
|
||||
recursive=request.POST.get('data_recursive') == 'true',
|
||||
)
|
||||
clustering = Config.Clustering(
|
||||
algorithm=request.POST.get('clustering_algorithm', cfg.clustering.algorithm),
|
||||
min_cluster_size=int(request.POST.get('clustering_min_cluster_size', cfg.clustering.min_cluster_size)),
|
||||
random_state=cfg.clustering.random_state,
|
||||
)
|
||||
llm = Config.LLM(
|
||||
enabled=request.POST.get('llm_enabled') == 'true',
|
||||
base_url=request.POST.get('llm_base_url', cfg.llm.base_url),
|
||||
api_key=request.POST.get('llm_api_key', cfg.llm.api_key),
|
||||
model=request.POST.get('llm_model', cfg.llm.model),
|
||||
)
|
||||
cfg = Config(server=server, data=data, clustering=clustering, llm=llm)
|
||||
save_config(cfg)
|
||||
# Re-render with saved flag
|
||||
return render(request, 'tianxuan/config.html', {'config': cfg, 'saved': True})
|
||||
|
||||
return render(request, 'tianxuan/config.html', {'config': cfg})
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def llm_test(request):
|
||||
"""LLM connectivity test: POST JSON {base_url, api_key, model} → chat/completions."""
|
||||
if request.method != 'POST':
|
||||
return JsonResponse({'success': False, 'message': '仅支持 POST 请求', 'latency_ms': 0}, status=405)
|
||||
|
||||
try:
|
||||
body = json.loads(request.body)
|
||||
except json.JSONDecodeError:
|
||||
return JsonResponse({'success': False, 'message': '无效的 JSON 请求体', 'latency_ms': 0})
|
||||
|
||||
base_url = (body.get('base_url') or '').rstrip('/')
|
||||
api_key = body.get('api_key') or ''
|
||||
model = body.get('model') or ''
|
||||
|
||||
if not base_url:
|
||||
return JsonResponse({'success': False, 'message': 'Base URL 不能为空', 'latency_ms': 0})
|
||||
if not api_key:
|
||||
return JsonResponse({'success': False, 'message': 'API Key 不能为空', 'latency_ms': 0})
|
||||
if not model:
|
||||
return JsonResponse({'success': False, 'message': 'Model 不能为空', 'latency_ms': 0})
|
||||
|
||||
url = f'{base_url}/chat/completions'
|
||||
payload = json.dumps({
|
||||
'model': model,
|
||||
'messages': [{'role': 'user', 'content': 'respond with ok'}],
|
||||
'max_tokens': 5,
|
||||
}).encode('utf-8')
|
||||
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=payload,
|
||||
headers={
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': f'Bearer {api_key}',
|
||||
},
|
||||
method='POST',
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
elapsed = int((time.monotonic() - start) * 1000)
|
||||
data = json.loads(resp.read().decode('utf-8'))
|
||||
if 'choices' in data and len(data['choices']) > 0:
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=True latency={elapsed}ms')
|
||||
return JsonResponse({
|
||||
'success': True,
|
||||
'message': f'连接成功，模型 {model} 返回正常',
|
||||
'latency_ms': elapsed,
|
||||
})
|
||||
else:
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=True latency={elapsed}ms')
|
||||
return JsonResponse({
|
||||
'success': True,
|
||||
'message': f'连接成功，但响应中无 choices（原始响应已记录）',
|
||||
'latency_ms': elapsed,
|
||||
})
|
||||
except urllib.error.HTTPError as e:
|
||||
elapsed = int((time.monotonic() - start) * 1000)
|
||||
try:
|
||||
detail = json.loads(e.read().decode('utf-8', errors='replace'))
|
||||
msg = detail.get('error', {}).get('message', str(e))
|
||||
except Exception:
|
||||
msg = str(e)
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
|
||||
return JsonResponse({'success': False, 'message': f'HTTP {e.code}: {msg}', 'latency_ms': elapsed})
|
||||
except urllib.error.URLError as e:
|
||||
elapsed = int((time.monotonic() - start) * 1000)
|
||||
reason = str(e.reason) if e.reason else '连接失败'
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
|
||||
return JsonResponse({'success': False, 'message': f'连接失败: {reason}', 'latency_ms': elapsed})
|
||||
except TimeoutError:
|
||||
elapsed = int((time.monotonic() - start) * 1000)
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
|
||||
return JsonResponse({'success': False, 'message': '请求超时（5秒）', 'latency_ms': elapsed})
|
||||
except Exception as e:
|
||||
elapsed = int((time.monotonic() - start) * 1000)
|
||||
logger.info(f'[LLM_TEST] base_url={base_url} model={model} success=False latency={elapsed}ms')
|
||||
return JsonResponse({'success': False, 'message': f'未知错误: {str(e)}', 'latency_ms': elapsed})
|
||||
|
||||
|
||||
# ── Upload page ─────────────────────────────────────────────────────────
|
||||
|
||||
def upload_page(request):
|
||||
"""CSV upload page."""
|
||||
runs = AnalysisRun.objects.all().order_by('-created_at')[:20]
|
||||
return render(request, 'tianxuan/upload.html', {'runs': runs})
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def upload_csv(request):
|
||||
"""Handle CSV upload AJAX POST."""
|
||||
if request.method != 'POST':
|
||||
return JsonResponse({'error': 'POST required'}, status=405)
|
||||
files = request.FILES.getlist('files')
|
||||
if not files:
|
||||
return JsonResponse({'error': 'No files uploaded'})
|
||||
if len(files) > 10000:
|
||||
return JsonResponse({'error': '一次最多上传 10000 个文件'})
|
||||
for f in files:
|
||||
if f.size > 500 * 1024 * 1024:
|
||||
return JsonResponse({'warning': f'文件 {f.name} 超过 500MB，建议分割后上传'})
|
||||
total_size = sum(f.size for f in files)
|
||||
if total_size > 50 * 1024 * 1024 * 1024:
|
||||
return JsonResponse({'error': '总大小超过 50GB 限制'})
|
||||
from django.utils import timezone
|
||||
ts = timezone.now().strftime('%Y%m%d_%H%M%S')
|
||||
import platform
|
||||
if platform.system() == 'Windows':
|
||||
base = Path(os.environ.get('APPDATA', '')) / 'TianXuan' / 'data' / 'uploads'
|
||||
else:
|
||||
base = Path.home() / '.tianxuan' / 'data' / 'uploads'
|
||||
upload_dir = base / ts
|
||||
upload_dir.mkdir(parents=True, exist_ok=True)
|
||||
saved = []
|
||||
for f in files:
|
||||
path = upload_dir / f.name
|
||||
with open(path, 'wb+') as dst:
|
||||
for chunk in f.chunks():
|
||||
dst.write(chunk)
|
||||
saved.append(str(path))
|
||||
# Auto-process in background
|
||||
run = AnalysisRun.objects.create(
|
||||
csv_glob=str(upload_dir / '*.csv'),
|
||||
status='loading',
|
||||
)
|
||||
t = threading.Thread(target=_background_process, args=(run.id, upload_dir), daemon=True)
|
||||
t.start()
|
||||
return JsonResponse({'run_id': run.id, 'files': saved, 'status': 'loading'})
|
||||
|
||||
|
||||
def delete_upload(request, run_id):
|
||||
"""Delete uploaded files and AnalysisRun."""
|
||||
run = get_object_or_404(AnalysisRun, id=run_id)
|
||||
import shutil
|
||||
upload_dir = None
|
||||
if run.csv_glob:
|
||||
p = Path(run.csv_glob).parent
|
||||
if p.exists():
|
||||
shutil.rmtree(p, ignore_errors=True)
|
||||
run.delete()
|
||||
return JsonResponse({'deleted': True})
|
||||
|
||||
|
||||
import traceback, logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _background_process(run_id, upload_dir):
|
||||
"""Background: load → detect → aggregate → ready."""
|
||||
import django
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
django.setup()
|
||||
from analysis.models import AnalysisRun
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.session_store import SessionStore
|
||||
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
store = SessionStore()
|
||||
try:
|
||||
# Check upload directory still exists (may have been temp-cleaned)
|
||||
if not upload_dir.is_dir():
|
||||
msg = '上传数据目录已不存在，请重新上传'
|
||||
logger.error('[BACKGROUND] %s (run_id=%s, upload_dir=%s)', msg, run_id, upload_dir)
|
||||
run.status = 'failed'
|
||||
run.error_message = msg
|
||||
run.save(update_fields=['status', 'error_message'])
|
||||
return
|
||||
|
||||
run.status = 'loading'; run.progress_pct = 10; run.progress_msg = '正在加载数据...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
lf, schema, row_count, file_count, mem = load_csv_directory(
|
||||
str(upload_dir / '*.csv'), schema_strict=False
|
||||
)
|
||||
ds_id = f'upload_{run_id}'
|
||||
store.store_dataset(ds_id, lf, schema=schema, metadata={
|
||||
'row_count': row_count, 'file_count': file_count, 'upload_dir': str(upload_dir),
|
||||
'csv_glob': str(upload_dir / '*.csv'),
|
||||
})
|
||||
run.total_flows = row_count; run.save(update_fields=['total_flows'])
|
||||
|
||||
run.status = 'profiling'; run.progress_pct = 30; run.progress_msg = '正在分析数据特征...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
entity_result = detect_entity_column(ds_id)
|
||||
entity_col = entity_result.get('recommended', '')
|
||||
entity_cols_multi = entity_result.get('recommended_multi', [])
|
||||
# Store as comma-separated (multi) or single-column
|
||||
if entity_cols_multi:
|
||||
entity_cols = entity_cols_multi[:2] # top 2
|
||||
run.entity_column = ','.join(entity_cols)
|
||||
else:
|
||||
entity_cols = [entity_col] if entity_col else []
|
||||
run.entity_column = entity_col
|
||||
run.save(update_fields=['entity_column'])
|
||||
|
||||
if entity_cols:
|
||||
run.status = 'aggregating'; run.progress_pct = 50; run.progress_msg = '正在构建实体画像...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_cols, schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
entity_ds_id = f'entity_{run_id}'
|
||||
store.store_dataset(entity_ds_id, agg_lf,
|
||||
schema=dict(zip(features, [''] * len(features))),
|
||||
metadata={'row_count': len(df_agg), 'entity_column': ','.join(entity_cols),
|
||||
'csv_glob': str(upload_dir / '*.csv')})
|
||||
run.entity_count = len(df_agg)
|
||||
run.entity_column = ','.join(entity_cols)
|
||||
run.save(update_fields=['entity_column', 'entity_count'])
|
||||
|
||||
run.status = 'ready'; run.progress_pct = 60; run.progress_msg = '数据准备就绪'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
except Exception:
|
||||
tb = traceback.format_exc()
|
||||
logger.error(tb)
|
||||
run.status = 'failed'
|
||||
run.run_log += f'\n[ERROR] {tb}'
|
||||
run.save(update_fields=['run_log'])
|
||||
run.error_message = tb
|
||||
run.save(update_fields=['status', 'error_message'])
|
||||
|
||||
|
||||
# ── Manual analysis page ───────────────────────────────────────────────
|
||||
|
||||
def manual_page(request):
|
||||
"""4-step manual analysis wizard."""
|
||||
step = int(request.GET.get('step', 1))
|
||||
runs = AnalysisRun.objects.filter(status='ready').order_by('-created_at')[:20]
|
||||
ctx = {'step': step, 'runs': runs}
|
||||
# Step 2: pass entity candidates and schema for the selected run
|
||||
if step >= 2:
|
||||
run_id = request.GET.get('run_id')
|
||||
if run_id:
|
||||
try:
|
||||
run = AnalysisRun.objects.get(id=int(run_id))
|
||||
ctx['selected_run'] = run
|
||||
# Try to get entity detection candidates from SessionStore
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
store = SessionStore()
|
||||
ds_id = f'upload_{run_id}'
|
||||
entry = store.get_dataset(ds_id)
|
||||
if entry is not None:
|
||||
schema = entry.get('schema', {})
|
||||
ctx['schema_keys'] = list(schema.keys())
|
||||
# Build numeric-only feature column options with dtype labels
|
||||
numeric_types = {'Int8', 'Int16', 'Int32', 'Int64',
|
||||
'UInt8', 'UInt16', 'UInt32', 'UInt64',
|
||||
'Float32', 'Float64'}
|
||||
ctx['feature_options'] = [
|
||||
{'name': k, 'dtype': v}
|
||||
for k, v in schema.items()
|
||||
if v.split('(')[0].strip() in numeric_types
|
||||
]
|
||||
# Detect entity candidates
|
||||
det_result = detect_entity_column(ds_id)
|
||||
if not det_result.get('error'):
|
||||
ctx['entity_candidates'] = det_result.get('candidates', [])
|
||||
# Pre-select stored entity columns (possibly comma-separated multi)
|
||||
entity_cols_str = run.entity_column
|
||||
if entity_cols_str:
|
||||
ctx['selected_entity_cols'] = entity_cols_str.split(',')
|
||||
except (AnalysisRun.DoesNotExist, ValueError):
|
||||
pass
|
||||
# Step 3: pass config defaults
|
||||
from config import get_config
|
||||
cfg = get_config()
|
||||
ctx['default_algo'] = cfg.clustering.algorithm
|
||||
ctx['default_min_size'] = cfg.clustering.min_cluster_size
|
||||
return render(request, 'tianxuan/manual.html', ctx)
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def run_analysis(request):
|
||||
"""AJAX POST: run analysis pipeline and return result page URL."""
|
||||
if request.method != 'POST':
|
||||
return JsonResponse({'error': 'POST required'}, status=405)
|
||||
import json
|
||||
body = json.loads(request.body)
|
||||
run_id = body.get('run_id')
|
||||
entity_col = body.get('entity_column', '')
|
||||
entity_columns = body.get('entity_columns')
|
||||
feature_columns = body.get('feature_columns')
|
||||
algorithm = body.get('algorithm', 'hdbscan')
|
||||
min_cluster_size = int(body.get('min_cluster_size', 5))
|
||||
|
||||
t = threading.Thread(
|
||||
target=_run_analysis_worker,
|
||||
args=(run_id, entity_col, entity_columns, feature_columns, algorithm, min_cluster_size),
|
||||
daemon=True,
|
||||
)
|
||||
t.start()
|
||||
return JsonResponse({'status': 'started', 'redirect': f'/runs/{run_id}/'})
|
||||
|
||||
|
||||
def _run_analysis_worker(run_id, entity_col, entity_columns, feature_columns, algorithm, min_cluster_size):
|
||||
"""Background: cluster + extract features for a run."""
|
||||
import django, os
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
django.setup()
|
||||
from analysis.models import AnalysisRun
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.tool_registry import _cluster_sync, _handle_extract_features
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from asgiref.sync import async_to_sync
|
||||
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
store = SessionStore()
|
||||
try:
|
||||
# If entity_columns provided, re-aggregate from the raw upload dataset
|
||||
if entity_columns:
|
||||
upload_ds_id = f'upload_{run_id}'
|
||||
upload_entry = store.get_dataset(upload_ds_id)
|
||||
if upload_entry is None:
|
||||
run.error_message = '请先上传并等待预处理完成'
|
||||
run.status = 'failed'
|
||||
run.save(update_fields=['status', 'error_message'])
|
||||
return
|
||||
upload_lf = upload_entry['lazyframe']
|
||||
upload_schema = upload_entry.get('schema', {})
|
||||
agg_lf, agg_features = aggregate_by_entity(upload_lf, entity_columns, upload_schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
agg_schema = {n: str(d) for n, d in zip(df_agg.columns, [df_agg[c].dtype for c in df_agg.columns])}
|
||||
entity_ds_id = f'entity_{run_id}_custom'
|
||||
store.store_dataset(
|
||||
dataset_id=entity_ds_id,
|
||||
lazyframe=df_agg.lazy(),
|
||||
schema=agg_schema,
|
||||
metadata={'row_count': len(df_agg), 'entity_columns': entity_columns,
|
||||
'csv_glob': run.csv_glob},
|
||||
)
|
||||
run.entity_column = ','.join(entity_columns)
|
||||
run.entity_count = len(df_agg)
|
||||
run.save(update_fields=['entity_column', 'entity_count'])
|
||||
else:
|
||||
entity_ds_id = f'entity_{run_id}'
|
||||
entry = store.get_dataset(entity_ds_id)
|
||||
if entry is None:
|
||||
run.error_message = '请先上传并等待预处理完成'
|
||||
run.status = 'failed'
|
||||
run.save(update_fields=['status', 'error_message'])
|
||||
return
|
||||
|
||||
run.status = 'clustering'; run.progress_pct = 70; run.progress_msg = '正在进行聚类分析...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
entry = store.get_dataset(entity_ds_id)
|
||||
schema = entry.get('schema', {})
|
||||
if feature_columns:
|
||||
entity_cols_set = set()
|
||||
if hasattr(run, 'entity_column') and run.entity_column:
|
||||
entity_cols_set = {c.strip() for c in run.entity_column.split(',')}
|
||||
feature_cols = [c for c in feature_columns if c in schema and c not in entity_cols_set]
|
||||
else:
|
||||
entity_cols_set = set()
|
||||
if hasattr(run, 'entity_column') and run.entity_column:
|
||||
entity_cols_set = {c.strip() for c in run.entity_column.split(',')}
|
||||
feature_cols = [c for c in schema.keys() if not c.startswith('_') and c not in entity_cols_set][:10]
|
||||
result = _cluster_sync(
|
||||
dataset_id=entity_ds_id,
|
||||
cluster_columns=feature_cols,
|
||||
algorithm=algorithm,
|
||||
params={'min_cluster_size': min_cluster_size},
|
||||
random_state=42,
|
||||
)
|
||||
if 'error' in result:
|
||||
err = result['error']
|
||||
if 'numeric' in err.lower():
|
||||
entry = store.get_dataset(entity_ds_id)
|
||||
if entry:
|
||||
schema = entry.get('schema', {})
|
||||
numeric_types = ('Int8', 'Int16', 'Int32', 'Int64',
|
||||
'UInt8', 'UInt16', 'UInt32', 'UInt64',
|
||||
'Float32', 'Float64')
|
||||
available = [f"{k}({v})" for k, v in schema.items()
|
||||
if v.split('(')[0].strip() in numeric_types and not k.startswith('_')]
|
||||
err += f"。可用数值列: {available}"
|
||||
raise Exception(err)
|
||||
cluster_id = result.get('cluster_result_id', '')
|
||||
run.cluster_count = result.get('n_clusters', 0)
|
||||
run.save(update_fields=['cluster_count'])
|
||||
|
||||
run.status = 'extracting'; run.progress_pct = 90; run.progress_msg = '正在提取聚类特征...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
async_to_sync(_handle_extract_features)(
|
||||
dataset_id=entity_ds_id,
|
||||
cluster_result_id=cluster_id,
|
||||
top_k=10, method='zscore', save_to_db=True,
|
||||
)
|
||||
run.status = 'completed'; run.progress_pct = 100; run.progress_msg = '分析完成'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg'])
|
||||
except Exception:
|
||||
tb = traceback.format_exc()
|
||||
logger.error(tb)
|
||||
last_pct = run.progress_pct
|
||||
run.status = 'failed'
|
||||
run.progress_msg = f'失败: {tb[:200]}'
|
||||
run.run_log += f'\n[ERROR] {tb}'
|
||||
run.save(update_fields=['run_log'])
|
||||
run.error_message = tb
|
||||
run.save(update_fields=['status', 'progress_msg', 'error_message'])
|
||||
|
||||
|
||||
# ── LLM auto analysis page ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def _run_llm_analysis(run_id, dataset_id, entity_col):
|
||||
"""Background: run LLM-driven analysis pipeline with progress callback."""
|
||||
import django, os
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
django.setup()
|
||||
from analysis.models import AnalysisRun
|
||||
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
|
||||
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
|
||||
def progress_callback(step, tool_name, result):
|
||||
nonlocal run
|
||||
try:
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
pct = min(10 + step * 6, 95)
|
||||
run.progress_pct = pct
|
||||
run.progress_msg = f'步骤 {step}: {tool_name}'
|
||||
if result:
|
||||
summary = str(result)[:200]
|
||||
run.run_log += f'[{step}] {tool_name}: {summary}\n'
|
||||
else:
|
||||
run.run_log += f'[{step}] {tool_name}\n'
|
||||
run.save(update_fields=['progress_pct', 'progress_msg', 'run_log'])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
from config import get_config
|
||||
cfg = get_config()
|
||||
llm_cfg = LLMConfig(cfg.llm.base_url, cfg.llm.api_key, cfg.llm.model)
|
||||
run.run_log = '启动 LLM 分析...\n'
|
||||
run.status = 'loading'
|
||||
run.progress_pct = 5
|
||||
run.progress_msg = 'LLM 初始化中...'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg', 'run_log'])
|
||||
|
||||
result = run_llm_pipeline(dataset_id, entity_col, config=llm_cfg, callback=progress_callback)
|
||||
|
||||
if 'error' in result:
|
||||
run.status = 'failed'
|
||||
run.progress_msg = f'失败: {result["error"]}'
|
||||
run.run_log += f'\n[ERROR] {result["error"]}'
|
||||
else:
|
||||
run.status = 'completed'
|
||||
run.progress_pct = 100
|
||||
run.progress_msg = 'LLM 分析完成'
|
||||
run.run_log += f'\n[完成] {result.get("result", "")}'
|
||||
run.save(update_fields=['status', 'progress_pct', 'progress_msg', 'run_log'])
|
||||
except Exception:
|
||||
tb = traceback.format_exc()
|
||||
logger.error(tb)
|
||||
try:
|
||||
run = AnalysisRun.objects.get(id=run_id)
|
||||
run.status = 'failed'
|
||||
run.progress_msg = f'失败: {tb[:200]}'
|
||||
run.run_log += f'\n[ERROR] {tb}'
|
||||
run.error_message = tb
|
||||
run.save(update_fields=['status', 'progress_msg', 'run_log', 'error_message'])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def run_llm_analysis_view(request):
|
||||
"""AJAX POST: run LLM-driven analysis pipeline."""
|
||||
if request.method != 'POST':
|
||||
return JsonResponse({'error': 'POST required'}, status=405)
|
||||
import json
|
||||
body = json.loads(request.body)
|
||||
run_id = body.get('run_id')
|
||||
entity_col = body.get('entity_column', '')
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
store = SessionStore()
|
||||
upload_ds_id = f'upload_{run_id}'
|
||||
entry = store.get_dataset(upload_ds_id)
|
||||
|
||||
if entry is None:
|
||||
# Try entity dataset as fallback
|
||||
entity_ds_id = f'entity_{run_id}'
|
||||
entry = store.get_dataset(entity_ds_id)
|
||||
if entry is None:
|
||||
return JsonResponse({'error': '数据集不存在，请先上传并等待预处理完成'})
|
||||
|
||||
dataset_id = upload_ds_id if store.get_dataset(upload_ds_id) else f'entity_{run_id}'
|
||||
|
||||
t = threading.Thread(
|
||||
target=_run_llm_analysis,
|
||||
args=(run_id, dataset_id, entity_col),
|
||||
daemon=True,
|
||||
)
|
||||
t.start()
|
||||
return JsonResponse({'status': 'started', 'run_id': run_id})
|
||||
|
||||
|
||||
def auto_page(request):
|
||||
"""LLM-driven auto analysis page."""
|
||||
runs = AnalysisRun.objects.filter(status='ready').order_by('-created_at')[:20]
|
||||
cfg = get_config()
|
||||
return render(request, 'tianxuan/auto.html', {
|
||||
'runs': runs,
|
||||
'llm_configured': bool(cfg.llm.base_url and cfg.llm.api_key),
|
||||
})
|
||||
|
||||
|
||||
# ── Progress API ───────────────────────────────────────────────────────
|
||||
|
||||
def run_status_api(request, run_id):
|
||||
"""Return current run status as JSON."""
|
||||
run = get_object_or_404(AnalysisRun, id=run_id)
|
||||
return JsonResponse({
|
||||
'id': run.id,
|
||||
'status': run.status,
|
||||
'entity_count': run.entity_count,
|
||||
'cluster_count': run.cluster_count,
|
||||
'error_message': run.error_message,
|
||||
'progress_pct': run.progress_pct,
|
||||
'progress_msg': run.progress_msg,
|
||||
'run_log': run.run_log[-2000:], # last 2000 chars
|
||||
})
|
||||
|
||||
|
||||
def log_viewer(request):
|
||||
"""Display last 200 lines of the app log (DEBUG only).
|
||||
With ?pos=N, returns text: line_count\\nnew_content for AJAX polling.
|
||||
"""
|
||||
from django.conf import settings
|
||||
if not settings.DEBUG:
|
||||
return HttpResponse('Log viewer only available in DEBUG mode', status=403)
|
||||
log_path = settings.BASE_DIR / 'logs' / 'tianxuan.log'
|
||||
pos = request.GET.get('pos')
|
||||
if pos is not None:
|
||||
# AJAX polling mode: return new content since byte position
|
||||
try:
|
||||
pos = int(pos)
|
||||
except ValueError:
|
||||
pos = 0
|
||||
if log_path.exists():
|
||||
with open(log_path, 'r', encoding='utf-8') as f:
|
||||
total = f.seek(0, 2) # seek to end
|
||||
if pos >= total:
|
||||
return HttpResponse(f'{total}\n', content_type='text/plain')
|
||||
f.seek(pos)
|
||||
new_content = f.read()
|
||||
return HttpResponse(f'{total}\n{new_content}', content_type='text/plain')
|
||||
return HttpResponse('0\n', content_type='text/plain')
|
||||
|
||||
# HTML page mode
|
||||
lines = []
|
||||
if log_path.exists():
|
||||
with open(log_path, 'r', encoding='utf-8') as f:
|
||||
lines = f.readlines()[-200:]
|
||||
return render(request, 'tianxuan/log_viewer.html', {'lines': lines})
|
||||
|
||||
# ── 3D Globe view ─────────────────────────────────────────────────────────
|
||||
|
||||
def _extract_flows_from_df(df, MAX_ROWS):
|
||||
"""Extract flow data (lat/lon/TLS/bytes) from a Polars DataFrame."""
|
||||
from analysis.geoip import lookup as geo_lookup
|
||||
TLS_HEX_MAP = {'0303': 'TLSv1.2', '0304': 'TLSv1.3', '0302': 'TLSv1.1', '0301': 'TLSv1.0'}
|
||||
|
||||
src_lat_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
|
||||
('src_latitude', 'src_lat', 'source_latitude', 'source_lat', ':ips_lat')), None)
|
||||
src_lon_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
|
||||
('src_longitude', 'src_lon', 'source_longitude', 'source_lon', ':ips_lon')), None)
|
||||
dst_lat_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
|
||||
('dst_latitude', 'dst_lat', 'dest_latitude', 'dest_lat', ':ipd_lat')), None)
|
||||
dst_lon_col = next((c for c in df.columns if c.lower().replace('-','_').replace(' ','_') in
|
||||
('dst_longitude', 'dst_lon', 'dest_longitude', 'dest_lon', ':ipd_lon')), None)
|
||||
src_ip_col = next((c for c in df.columns if 'src_ip' in c.lower() or ':ips' in c.lower()), None)
|
||||
dst_ip_col = next((c for c in df.columns if 'dst_ip' in c.lower() or ':ipd' in c.lower()), None)
|
||||
tls_col = next((c for c in df.columns if 'tls_version' in c.lower() or '_tlsver' in c.lower() or 'version' in c.lower()), None)
|
||||
bytes_col = next((c for c in df.columns if 'bytes_sent' in c.lower() or 'bytes' in c.lower()), None)
|
||||
|
||||
flows = []
|
||||
for row in df.iter_rows(named=True):
|
||||
slat = slon = dlat = dlon = None
|
||||
|
||||
if src_lat_col and src_lon_col:
|
||||
slat = _extract_lat(row.get(src_lat_col))
|
||||
slon = _extract_lon(row.get(src_lon_col))
|
||||
if dst_lat_col and dst_lon_col:
|
||||
dlat = _extract_lat(row.get(dst_lat_col))
|
||||
dlon = _extract_lon(row.get(dst_lon_col))
|
||||
|
||||
if (slat is None or slon is None) and src_ip_col:
|
||||
sg = geo_lookup(str(row.get(src_ip_col, '')))
|
||||
if sg:
|
||||
slat, slon = sg['lat'], sg['lon']
|
||||
if (dlat is None or dlon is None) and dst_ip_col:
|
||||
dg = geo_lookup(str(row.get(dst_ip_col, '')))
|
||||
if dg:
|
||||
dlat, dlon = dg['lat'], dg['lon']
|
||||
|
||||
if slat is None or slon is None or dlat is None or dlon is None:
|
||||
continue
|
||||
|
||||
tls_val = str(row.get(tls_col or '', ''))
|
||||
# Check hex format first (e.g. "03 03" or "0303")
|
||||
hex_key = tls_val.replace(' ', '').strip()
|
||||
if hex_key in TLS_HEX_MAP:
|
||||
tls_label = TLS_HEX_MAP[hex_key]
|
||||
elif '1.3' in tls_val:
|
||||
tls_label = 'TLSv1.3'
|
||||
elif '1.2' in tls_val:
|
||||
tls_label = 'TLSv1.2'
|
||||
else:
|
||||
tls_label = 'other'
|
||||
|
||||
bs = float(row.get(bytes_col or '', 0) or 0)
|
||||
flows.append({
|
||||
'slat': slat, 'slon': slon,
|
||||
'dlat': dlat, 'dlon': dlon,
|
||||
'tls': tls_label,
|
||||
'bytes': bs,
|
||||
})
|
||||
return flows
|
||||
|
||||
|
||||
def globe_view(request):
|
||||
import json
|
||||
from analysis.geoip import lookup as geo_lookup
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.models import AnalysisRun
|
||||
from analysis.session_store import SessionStore
|
||||
|
||||
MAX_ROWS_PER_RUN = 300 # Limit per run to prevent browser overload
|
||||
|
||||
# Check for ?data=dataset_id parameter (direct data loading from session store)
|
||||
data_param = request.GET.get('data')
|
||||
if data_param:
|
||||
store = SessionStore()
|
||||
entry = store.get_dataset(data_param)
|
||||
flows = []
|
||||
all_runs = list(AnalysisRun.objects.filter(status='completed').order_by('-id').values('id', 'csv_glob'))
|
||||
if entry is not None:
|
||||
lf = entry['lazyframe']
|
||||
try:
|
||||
df = lf.head(MAX_ROWS_PER_RUN).collect()
|
||||
flows = _extract_flows_from_df(df, MAX_ROWS_PER_RUN)
|
||||
except Exception:
|
||||
pass
|
||||
return render(request, 'tianxuan/globe.html', {
|
||||
'geo_flows': json.dumps(flows),
|
||||
'all_runs': all_runs,
|
||||
'selected_ids': [],
|
||||
})
|
||||
|
||||
# Get all completed runs for the selector
|
||||
all_runs = list(AnalysisRun.objects.filter(status='completed').order_by('-id').values('id', 'csv_glob'))
|
||||
|
||||
# Get selected run IDs from query params (multiple checkboxes)
|
||||
selected_ids_list = request.GET.getlist('runs')
|
||||
selected_runs = []
|
||||
if selected_ids_list:
|
||||
for sid in selected_ids_list:
|
||||
sid = sid.strip()
|
||||
if sid.isdigit():
|
||||
try:
|
||||
r = AnalysisRun.objects.get(id=int(sid), status='completed')
|
||||
selected_runs.append(r)
|
||||
except AnalysisRun.DoesNotExist:
|
||||
pass
|
||||
|
||||
if not selected_runs:
|
||||
# Default: latest completed run
|
||||
latest = AnalysisRun.objects.filter(status='completed').order_by('-id').first()
|
||||
if latest:
|
||||
selected_runs = [latest]
|
||||
|
||||
flows = []
|
||||
|
||||
for run in selected_runs:
|
||||
try:
|
||||
# Check upload directory exists before accessing
|
||||
if run.csv_glob:
|
||||
glob_parent = Path(run.csv_glob).parent
|
||||
if not glob_parent.is_dir():
|
||||
logger.warning('[GLOBE] Upload directory missing for run %s: %s', run.id, glob_parent)
|
||||
continue
|
||||
lf, schema, rc, fc, mem = load_csv_directory(run.csv_glob)
|
||||
df = lf.head(MAX_ROWS_PER_RUN).collect()
|
||||
flows.extend(_extract_flows_from_df(df, MAX_ROWS_PER_RUN))
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
resp = render(request, 'tianxuan/globe.html', {
|
||||
'geo_flows': json.dumps(flows),
|
||||
'all_runs': all_runs,
|
||||
'selected_ids': [r.id for r in selected_runs],
|
||||
})
|
||||
resp['Cache-Control'] = 'no-cache, no-store, must-revalidate'
|
||||
resp['Pragma'] = 'no-cache'
|
||||
resp['Expires'] = '0'
|
||||
return resp
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
"""Configuration loader for TianXuan."""
|
||||
from .loader import Config, get_config, save_config
|
||||
|
||||
__all__ = ['Config', 'get_config', 'save_config']
|
||||
@@ -0,0 +1,87 @@
|
||||
server:
|
||||
host: 127.0.0.1
|
||||
port: 8000
|
||||
debug: false
|
||||
data:
|
||||
schema_strict: false
|
||||
recursive: false
|
||||
geoip:
|
||||
enabled: true
|
||||
ip_columns: ["src_ip", "dst_ip"]
|
||||
drop_columns: [] # 默认不清除经纬列——由加载器自动清洗
|
||||
clustering:
|
||||
algorithm: hdbscan
|
||||
min_cluster_size: 5
|
||||
random_state: 42
|
||||
llm:
|
||||
enabled: false
|
||||
base_url: ''
|
||||
api_key: ''
|
||||
model: gpt-4
|
||||
entity:
|
||||
subnet_masks: [24, 28]
|
||||
ip_columns: ["src_ip", "dst_ip", ":ips", ":ipd", "server-ip", "client-ip"]
|
||||
columns:
|
||||
src_ip: ipv4
|
||||
dst_ip: ipv4
|
||||
":ips": ipv4
|
||||
":ipd": ipv4
|
||||
server-ip: ipv4
|
||||
client-ip: ipv4
|
||||
server_ip: ipv4
|
||||
client_ip: ipv4
|
||||
bytes_sent: int
|
||||
bytes_rev: int
|
||||
duration: float
|
||||
tls_version: enum
|
||||
cipher_suite: hex
|
||||
dst_url: url
|
||||
lat: lat_lon
|
||||
lon: lat_lon
|
||||
timestamp: timestamp
|
||||
# User column overrides
|
||||
"0ver": enum
|
||||
":prs": enum
|
||||
":prd": enum
|
||||
scnt: enum
|
||||
dcnt: enum
|
||||
cnam: string
|
||||
snam: string
|
||||
"4dur": float
|
||||
"8ses": float
|
||||
"2tmo": float
|
||||
"4ksz": int
|
||||
cnrs: enum
|
||||
isrs: enum
|
||||
"8ack": int
|
||||
"8ppk": int
|
||||
"8dbd": timestamp
|
||||
"1ipp": enum
|
||||
"4dbn": enum
|
||||
tabl: string
|
||||
"name": string
|
||||
source-node: string
|
||||
source_node: string
|
||||
ecdhe-named-curve: string
|
||||
ecdhe_named_curve: string
|
||||
"0cph": hex
|
||||
"0crv": hex
|
||||
"0rnd": string
|
||||
"0rnt": float
|
||||
"row": int
|
||||
"time": timestamp
|
||||
":ips.latd": lat_lon
|
||||
":ipd.latd": lat_lon
|
||||
":ips.lond": lat_lon
|
||||
":ipd.lond": lat_lon
|
||||
":ips.ispn": string
|
||||
":ipd.ispn": string
|
||||
":ips.orgn": string
|
||||
":ipd.orgn": string
|
||||
":ips.city": string
|
||||
":ipd.city": string
|
||||
"+.latd": lat_lon
|
||||
"+.lond": lat_lon
|
||||
"+.ispn": string
|
||||
"+.orgn": string
|
||||
"+.city": string
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Configuration loader with Pydantic model, YAML file, and caching."""
|
||||
|
||||
import pathlib
|
||||
|
||||
import yaml
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Config(BaseModel):
|
||||
"""Typed configuration for TianXuan."""
|
||||
|
||||
class Server(BaseModel):
|
||||
host: str = '127.0.0.1'
|
||||
port: int = 8000
|
||||
debug: bool = False
|
||||
|
||||
class Data(BaseModel):
|
||||
schema_strict: bool = False
|
||||
recursive: bool = False
|
||||
|
||||
class Clustering(BaseModel):
|
||||
algorithm: str = 'hdbscan'
|
||||
min_cluster_size: int = 5
|
||||
random_state: int = 42
|
||||
|
||||
class LLM(BaseModel):
|
||||
enabled: bool = False
|
||||
base_url: str = ''
|
||||
api_key: str = ''
|
||||
model: str = 'gpt-4'
|
||||
|
||||
class Entity(BaseModel):
|
||||
subnet_masks: list[int] = []
|
||||
ip_columns: list[str] = ['src_ip', 'dst_ip']
|
||||
|
||||
entity: Entity = Entity()
|
||||
server: Server = Server()
|
||||
data: Data = Data()
|
||||
clustering: Clustering = Clustering()
|
||||
llm: LLM = LLM()
|
||||
|
||||
|
||||
_config_path = pathlib.Path(__file__).parent / 'config.yaml'
|
||||
_config_cache = None
|
||||
_config_mtime = 0
|
||||
|
||||
|
||||
def get_config() -> Config:
|
||||
"""Return the cached Config, reloading if the YAML file has changed.
|
||||
|
||||
If *config.yaml* does not exist, returns an empty ``Config()`` with
|
||||
all default values.
|
||||
"""
|
||||
global _config_cache, _config_mtime
|
||||
|
||||
if _config_path.exists():
|
||||
mtime = _config_path.stat().st_mtime
|
||||
if _config_cache is None or mtime > _config_mtime:
|
||||
with open(_config_path, 'r', encoding='utf-8') as f:
|
||||
data = yaml.safe_load(f) or {}
|
||||
_config_cache = Config(**data)
|
||||
_config_mtime = mtime
|
||||
else:
|
||||
if _config_cache is None:
|
||||
_config_cache = Config()
|
||||
|
||||
return _config_cache
|
||||
|
||||
|
||||
def save_config(cfg: Config) -> None:
|
||||
"""Persist a Config instance back to *config.yaml* and update the cache."""
|
||||
global _config_cache, _config_mtime
|
||||
with open(_config_path, 'w', encoding='utf-8') as f:
|
||||
yaml.dump(
|
||||
cfg.model_dump(mode='python'),
|
||||
f,
|
||||
default_flow_style=False,
|
||||
allow_unicode=True,
|
||||
sort_keys=False,
|
||||
)
|
||||
_config_cache = cfg
|
||||
_config_mtime = _config_path.stat().st_mtime
|
||||
@@ -0,0 +1,41 @@
|
||||
# start_ip,end_ip,lat,lon,city,country
|
||||
# === North America ===
|
||||
1.0.0.0,1.0.0.255,37.386,-122.083,Mountain View,US
|
||||
8.8.8.0,8.8.8.255,37.386,-122.083,Mountain View,US
|
||||
72.14.192.0,72.14.255.255,40.7128,-74.0060,New York,US
|
||||
74.125.0.0,74.125.255.255,40.7128,-74.0060,New York,US
|
||||
66.102.0.0,66.102.255.255,34.0522,-118.2437,Los Angeles,US
|
||||
64.0.0.0,64.255.255.255,43.6532,-79.3832,Toronto,CA
|
||||
# === East Asia ===
|
||||
1.2.4.0,1.2.4.255,39.9042,116.4074,Beijing,CN
|
||||
123.125.114.0,123.125.114.255,39.9042,116.4074,Beijing,CN
|
||||
101.95.0.0,101.95.255.255,31.2304,121.4737,Shanghai,CN
|
||||
58.0.0.0,58.255.255.255,22.3193,114.1694,Hong Kong,HK
|
||||
14.0.0.0,14.255.255.255,22.3193,114.1694,Hong Kong,HK
|
||||
1.160.0.0,1.160.255.255,25.0330,121.5654,Taipei,TW
|
||||
101.0.0.0,101.0.255.255,25.0330,121.5654,Taipei,TW
|
||||
27.0.0.0,27.0.255.255,35.6762,139.6503,Tokyo,JP
|
||||
210.130.0.0,210.130.255.255,35.6762,139.6503,Tokyo,JP
|
||||
110.0.0.0,110.0.255.255,37.5665,126.9780,Seoul,KR
|
||||
121.78.0.0,121.78.255.255,37.5665,126.9780,Seoul,KR
|
||||
# === Southeast Asia ===
|
||||
116.14.0.0,116.14.255.255,1.3521,103.8198,Singapore,SG
|
||||
58.8.0.0,58.8.255.255,13.7563,100.5018,Bangkok,TH
|
||||
# === South Asia ===
|
||||
14.96.0.0,14.96.255.255,19.0760,72.8777,Mumbai,IN
|
||||
# === Oceania ===
|
||||
1.40.0.0,1.40.255.255,-33.8688,151.2093,Sydney,AU
|
||||
149.135.0.0,149.135.255.255,-33.8688,151.2093,Sydney,AU
|
||||
# === Europe ===
|
||||
5.10.0.0,5.10.255.255,51.5074,-0.1278,London,GB
|
||||
81.2.0.0,81.2.255.255,51.5074,-0.1278,London,GB
|
||||
90.0.0.0,90.0.255.255,48.8566,2.3522,Paris,FR
|
||||
46.0.0.0,46.0.255.255,50.1109,8.6821,Frankfurt,DE
|
||||
145.0.0.0,145.0.255.255,52.3676,4.9041,Amsterdam,NL
|
||||
84.203.0.0,84.203.255.255,53.3498,-6.2603,Dublin,IE
|
||||
78.0.0.0,78.0.255.255,59.3293,18.0686,Stockholm,SE
|
||||
95.0.0.0,95.0.255.255,55.7558,37.6173,Moscow,RU
|
||||
# === Middle East ===
|
||||
91.72.0.0,91.72.255.255,25.2048,55.2708,Dubai,AE
|
||||
# === South America ===
|
||||
177.0.0.0,177.0.255.255,-23.5505,-46.6333,Sao Paulo,BR
|
||||
BIN
Binary file not shown.
+818
@@ -0,0 +1,818 @@
|
||||
# 天璇 (TianXuan) 分析工作流
|
||||
|
||||
> 从原始 TLS 流 CSV 到实体画像、聚类结果、区分特征的完整流水线文档。
|
||||
|
||||
---
|
||||
|
||||
## 目录
|
||||
|
||||
1. [数据准备](#1-数据准备)
|
||||
2. [数据加载与配置](#2-数据加载与配置)
|
||||
3. [数据校验](#3-数据校验)
|
||||
4. [数据清洗](#4-数据清洗)
|
||||
5. [实体识别](#5-实体识别)
|
||||
6. [类型感知聚合](#6-类型感知聚合)
|
||||
7. [聚类分析](#7-聚类分析)
|
||||
8. [特征提取](#8-特征提取)
|
||||
9. [可视化](#9-可视化)
|
||||
10. [LLM 集成](#10-llm-集成)
|
||||
11. [跨机部署](#11-跨机部署)
|
||||
|
||||
---
|
||||
|
||||
## 1. 数据准备
|
||||
|
||||
### 1.1 CSV 格式要求
|
||||
|
||||
天璇接收以 **CSV 格式**存放的 TLS 流记录文件。每行代表一条网络流。
|
||||
|
||||
| 要求 | 说明 |
|
||||
|------|------|
|
||||
| **分隔符** | 逗号 `,`（默认），支持自定义分隔符 |
|
||||
| **文件扩展名** | `.csv` |
|
||||
| **多文件** | 支持 glob 通配符、`**/*.csv` 递归扫描、自动合并 |
|
||||
| **压缩包** | `.zip` 文件自动解压（通过上传界面） |
|
||||
| **文件数量** | 无硬性限制（合并时使用 Polars `diagonal_relaxed` 模式） |
|
||||
|
||||
### 1.2 编码建议
|
||||
|
||||
| 编码 | BOM 标记 | 自动检测 |
|
||||
|------|---------|---------|
|
||||
| **UTF-8**（推荐） | 无 / `EF BB BF` | ✅ 默认 |
|
||||
| UTF-8 with BOM | `EF BB BF` | ✅ 自动检测 |
|
||||
| UTF-16LE | `FF FE` | ✅ 自动检测 |
|
||||
| UTF-16BE | `FE FF` | ✅ 自动检测 |
|
||||
| Latin-1 | 无 | ✅ 手动指定 |
|
||||
|
||||
> **⚠️ 生产建议**：始终使用 **UTF-8 无 BOM** 编码。跨机器部署时强烈建议通过 `set PYTHONUTF8=1` 强制 Python 使用 UTF-8 模式（`run.bat` 已包含该设置）。
|
||||
|
||||
### 1.3 推荐列
|
||||
|
||||
系统基于列名关键词自动识别以下维度：
|
||||
|
||||
| 维度 | 推荐列名（不区分大小写） |
|
||||
|------|------------------------|
|
||||
| **源 IP** | `src_ip`、`source_ip`、`src_addr` |
|
||||
| **目的 IP** | `dst_ip`、`dest_ip`、`destination_ip`、`dst_addr`、`ip_dst` |
|
||||
| **字节数（上行）** | `bytes_sent`、`src_bytes`、`bytes_out`、`up_bytes`、`upload` |
|
||||
| **字节数（下行）** | `bytes_rev`、`bytes_received`、`dst_bytes`、`down_bytes`、`download` |
|
||||
| **持续时间** | `duration`、`dur`、`elapsed`、`time_delta`、`flow_duration` |
|
||||
| **包数** | `packets`、`pkts`、`packet_count`、`total_packets` |
|
||||
| **TLS 版本** | `tls_version`、`version`、`ssl_version`、`tlsver` |
|
||||
| **加密套件** | `cipher_suite`、`cipher`、`tls_cipher`、`ssl_cipher` |
|
||||
| **SNI** | `sni`、`server_name`、`tls_sni` |
|
||||
| **协议** | `proto`、`protocol`、`l4_proto`、`ip_proto` |
|
||||
| **目的端口** | `dst_port`、`destination_port`、`dest_port`、`port_dst` |
|
||||
| **目标 URL** | `dst_url`、`url`、`uri` |
|
||||
| **时间戳** | `timestamp`、`ts`、`time`、`datetime`、`first_seen` |
|
||||
| **纬度** | `lat`、`latitude`、`latitud`、`y` |
|
||||
| **经度** | `lon`、`lng`、`longitude`、`longitud`、`x` |
|
||||
|
||||
> 列名通过下划线分词匹配关键词（如 `flow_duration` 匹配 `duration`）。匹配算法见 `analysis/entity_aggregator.py` 中的 `_find_column()`。
|
||||
|
||||
### 1.4 命名规范
|
||||
|
||||
- **不要求**列名在所有文件中完全一致（默认 `schema_strict=false`）
|
||||
- 不同名的列自动取并集，缺失列填 `null`
|
||||
- 建议团队内统一列名，以获得最佳的自动识别效果
|
||||
|
||||
---
|
||||
|
||||
## 2. 数据加载与配置
|
||||
|
||||
### 2.1 加载入口
|
||||
|
||||
**主函数**：`analysis/data_loader.py::load_csv_directory()`
|
||||
|
||||
```python
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
|
||||
glob_pattern="data/*.csv",
|
||||
encoding="utf-8",
|
||||
delimiter=",",
|
||||
schema_strict=False,
|
||||
recursive=False,
|
||||
)
|
||||
```
|
||||
|
||||
**MCP 工具**（通过 `tool_registry.py` 暴露）：
|
||||
|
||||
```
|
||||
load_data(csv_glob="data/*.csv", schema_strict=False, recursive=False)
|
||||
```
|
||||
|
||||
**CLI 管道**（一键运行）：
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv
|
||||
```
|
||||
|
||||
### 2.2 列类型配置
|
||||
|
||||
在 `config/config.yaml` 中通过 `columns` 映射手动指定列类型。配置优先级高于自动推断：
|
||||
|
||||
```yaml
|
||||
columns:
|
||||
src_ip: ipv4 # 强制识别为 IPv4
|
||||
dst_ip: ipv4
|
||||
bytes_sent: int # 强制为整数类型
|
||||
bytes_rev: int
|
||||
duration: float # 强制为浮点类型
|
||||
tls_version: enum # 枚举类型（低基数分类变量）
|
||||
cipher_suite: hex # 十六进制数据
|
||||
dst_url: url # URL 字符串
|
||||
lat: lat_lon # 地理纬度
|
||||
lon: lat_lon # 地理经度
|
||||
timestamp: timestamp # 时间戳
|
||||
```
|
||||
|
||||
支持的类型字符串及映射：
|
||||
|
||||
| 配置值 | `DataType` | 聚合方式 |
|
||||
|--------|-----------|---------|
|
||||
| `int` | `INT` | mean / sum |
|
||||
| `float` | `FLOAT` | mean / sum |
|
||||
| `enum` | `ENUM` | mode（众数） |
|
||||
| `hex` | `HEX` | count（hex 对数统计） |
|
||||
| `url` | `URL` | domain（提取域名） |
|
||||
| `ipv4` | `IPv4` | subnet（/24 网络） |
|
||||
| `lat_lon` | `LAT_LON` | mean |
|
||||
| `timestamp` | `TIMESTAMP` | min / max |
|
||||
| `bool_enum` | `BOOL_ENUM` | mode |
|
||||
| `string` | `STRING` | count |
|
||||
|
||||
### 2.3 Schema 容错
|
||||
|
||||
| 模式 | 行为 | 适用场景 |
|
||||
|------|------|---------|
|
||||
| `schema_strict: true` | 列名不一致时报 `ValueError` | 测试/质量门禁 |
|
||||
| `schema_strict: false`（默认） | 自动并集合并，缺失列填 null | 日常使用 |
|
||||
|
||||
**建议**：生产环境建议 `schema_strict: true`，提前发现列名漂移。
|
||||
|
||||
### 2.4 GeoIP 配置
|
||||
|
||||
`config.yaml` 中的 GeoIP 配置段控制自动经纬度填充和旧列丢弃：
|
||||
|
||||
```yaml
|
||||
data:
|
||||
geoip:
|
||||
enabled: true
|
||||
ip_columns: ["src_ip", "dst_ip"] # 需查询的 IP 列
|
||||
drop_columns: ["src_latitude", "src_longitude", # 加载后丢弃的旧列
|
||||
"dst_latitude", "dst_longitude"]
|
||||
```
|
||||
|
||||
GeoIP 数据文件：`data/geoip_data.txt`，内嵌约 23 个主要城市 IP 段（200 KB 二进制查询格式）。
|
||||
|
||||
查询接口：
|
||||
|
||||
```python
|
||||
from analysis.geoip import lookup
|
||||
result = lookup("8.8.8.8") # → {lat: 37.386, lon: -122.083, city: "Mountain View", country: "US"}
|
||||
```
|
||||
|
||||
> ⚠️ **旧列丢弃**：如果 CSV 本身已包含 `src_latitude`/`src_longitude` 等列（来自旧有系统的 GeoIP 结果），系统在加载后自动丢弃这些列，避免与新的 GeoIP 查询冲突。
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据校验
|
||||
|
||||
### 3.1 校验入口
|
||||
|
||||
**主函数**：`analysis/data_validator.py::validate()`
|
||||
|
||||
```python
|
||||
from analysis.data_validator import validate, report
|
||||
|
||||
# 返回 JSON 报告
|
||||
validation_result = validate("dataset_id", strict=False)
|
||||
|
||||
# 返回格式化字符串报告
|
||||
print(report("dataset_id"))
|
||||
```
|
||||
|
||||
校验项通过 `[VALIDATE]` 日志标记输出。
|
||||
|
||||
### 3.2 校验项说明
|
||||
|
||||
| 校验项 | 检测内容 | 判定标准 |
|
||||
|--------|---------|---------|
|
||||
| **列类型矛盾** | schema 推断类型 vs classify_column 结果 | `INT↔FLOAT`、`ENUM↔BOOL_ENUM` 为良性差异；STRING 类型包容一切 |
|
||||
| **缺失率** | 每列 `null_count / total_rows` | `>50%` → **高风险**；`>30%` → 警告 |
|
||||
| **异常值** | 数值列 Z-score > 5 的比例 | `>5%` 触发警告 |
|
||||
| **枚举分布** | ENUM/BOOL_ENUM 列的值频率 | 标记空白、`+`、`-` 等特殊符号 |
|
||||
| **IPv4 有效性** | IP 列每个值的合法性 | 无效 IP 比例 `>50%` → **高风险** |
|
||||
|
||||
### 3.3 校验报告解读
|
||||
|
||||
JSON 报告结构：
|
||||
|
||||
```json
|
||||
{
|
||||
"valid": true, // 无高风险项时为 true
|
||||
"dataset_id": "ds_abc123",
|
||||
"total_rows": 5000,
|
||||
"total_columns": 24,
|
||||
"columns": {
|
||||
"src_ip": {
|
||||
"dtype": "String",
|
||||
"inferred_type": "IPv4",
|
||||
"null_rate": 0.02,
|
||||
"warnings": []
|
||||
},
|
||||
"duration": {
|
||||
"dtype": "Float64",
|
||||
"inferred_type": "FLOAT",
|
||||
"null_rate": 0.55,
|
||||
"z_score_gt_5_ratio": 0.03,
|
||||
"warnings": ["null rate 55.0% exceeds 50% (high risk)"]
|
||||
}
|
||||
},
|
||||
"warnings": [...],
|
||||
"risks": ["Column 'duration': null rate 55.0% exceeds 50% (high risk)"]
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 处理建议
|
||||
|
||||
| 发现问题 | 建议处理 |
|
||||
|---------|---------|
|
||||
| 缺失率 > 50% | 检查数据源是否漏采集；考虑在预处理阶段 `fillna` 或 `drop` |
|
||||
| Z-score > 5 比例过高 | 检查是否有异常流量（DDoS、扫描等） |
|
||||
| 无效 IPv4 比例高 | 检查列名是否错误（非 IP 列被匹配为 IPv4） |
|
||||
| 枚举值异常（空白/`+`/`-`） | 这些值会在清洗阶段被规范化（见第 4 节） |
|
||||
|
||||
---
|
||||
|
||||
## 4. 数据清洗
|
||||
|
||||
数据清洗在 `load_csv_directory()` 中自动执行，分三个阶段。
|
||||
|
||||
### 4.1 旧经纬度丢弃
|
||||
|
||||
加载完成后，立即丢弃 GeoIP 配置中指定的旧列：
|
||||
|
||||
```python
|
||||
# 默认丢弃列
|
||||
drop_columns = ["src_latitude", "src_longitude", "dst_latitude", "dst_longitude"]
|
||||
# 只丢弃 schema 中实际存在的列
|
||||
merged_lf = merged_lf.drop(existing_drop)
|
||||
```
|
||||
|
||||
日志标记：`[CLEAN] dropped columns: [...]`
|
||||
|
||||
### 4.2 BOOL_ENUM 标准化
|
||||
|
||||
对 `classify_schema()` 标记为 `DataType.BOOL_ENUM` 的字符串列，执行以下变换：
|
||||
|
||||
| 原始值 | 标准化后 |
|
||||
|--------|---------|
|
||||
| `""`（空字符串） | `null` |
|
||||
| `"+"` | `"True"` |
|
||||
| `"-"` | `"False"` |
|
||||
| `"t"`, `"T"`, `"true"`, `"True"` | 保持不变 |
|
||||
| `"f"`, `"F"`, `"false"`, `"False"` | 保持不变 |
|
||||
|
||||
> 原生 Boolean 类型的列跳过此步骤（已经是标准格式）。
|
||||
|
||||
日志标记：`[CLEAN] BOOL_ENUM columns normalised: [...]`
|
||||
|
||||
### 4.3 HEX 预处理
|
||||
|
||||
对标记为 `DataType.HEX` 的列（如 cipher_suite），添加 `{col}_hex_pairs_count` 辅助列：
|
||||
|
||||
```python
|
||||
# 计算每个值中空格分隔的 hex 对数量
|
||||
pl.col("cipher_suite").str.split(" ").list.len().alias("cipher_suite_hex_pairs_count")
|
||||
```
|
||||
|
||||
该计数列在后续聚合中以 `avg_hex_pairs_count` 参与聚类特征。
|
||||
|
||||
日志标记：`[CLEAN] HEX columns added hex_pairs_count: [...]`
|
||||
|
||||
### 4.4 通用数值清洗
|
||||
|
||||
对于任何未被分类为 IPv4/URL/HEX/ENUM/LAT_LON 的字符串列，系统自动尝试转为 Float64：
|
||||
|
||||
| 原始值 | 清洗后 |
|
||||
|--------|--------|
|
||||
| `"123.45"` | `123.45` |
|
||||
| `"+"` / `"-"` | `null` |
|
||||
| 空白／空字符串 | `null` |
|
||||
| `"N/A"` / `"null"` / `"None"` | `null` |
|
||||
| 其他不可解析字符串 | `null` |
|
||||
|
||||
这防止了"mean on str column"崩溃（当字符串列因数据伪影未能被早期类型检测捕获时），
|
||||
无论后续数据中出现何种新伪影。
|
||||
|
||||
函数：`analysis/entity_aggregator.py` → `_coerce_to_float()`
|
||||
|
||||
日志标记：`[CLEAN] numeric coercion applied to: [...]`
|
||||
|
||||
---
|
||||
|
||||
## 5. 实体识别
|
||||
|
||||
### 5.1 自动检测原理
|
||||
|
||||
**主函数**：`analysis/entity_detector.py::detect_entity_column()`
|
||||
|
||||
采用三维打分算法，对所有 String/Categorical 列评分：
|
||||
|
||||
| 维度 | 分值 | 说明 |
|
||||
|------|------|------|
|
||||
| **名称匹配** | +3.0 | 列名（归一化后）精确匹配关键词或 `_` 分隔匹配 |
|
||||
| **唯一值比率** | 0~10 | `unique_ratio = unique_count / non_null_count`，在 `[0.01, 0.50]` 区间内得分最高 |
|
||||
| **字符串类型** | +2.0 | `Utf8` / `String` / `Categorical` 类型加分 |
|
||||
| **高唯一性惩罚** | -5.0 | `unique_ratio > 0.50` 时扣分（可能是随机 ID） |
|
||||
| **高空值惩罚** | ×0.1 | null 率 > 50% 时分数严重降低 |
|
||||
|
||||
关键词集（`ENTITY_KEYWORDS`）：
|
||||
|
||||
```
|
||||
src_ip, dst_ip, source_ip, destination_ip, ip,
|
||||
sni, server_name, host, domain, user, client,
|
||||
username, hostname, mac, email, session_id,
|
||||
uid, id, src_addr, dst_addr
|
||||
```
|
||||
|
||||
### 5.2 多列复合键
|
||||
|
||||
**支持场景**：当需要按 `(src_ip, dst_ip)` 或 `(src_ip, sni)` 等多列组合作为实体标识时。
|
||||
|
||||
**MCP 工具参数**：
|
||||
|
||||
```python
|
||||
build_entity_profiles(
|
||||
dataset_id="ds_...",
|
||||
entity_columns=["src_ip", "dst_ip"], # 多列复合键
|
||||
auto_detect=false
|
||||
)
|
||||
```
|
||||
|
||||
**CLI 管道**目前仅支持单列实体键。
|
||||
|
||||
### 5.3 IP 掩码聚类
|
||||
|
||||
在聚合阶段中，对 `DataType.IPv4` 类型的列自动生成 `/24` 子网信息：
|
||||
|
||||
```python
|
||||
# entity_aggregator.py 中：
|
||||
subnet_expr = (
|
||||
pl.col(ip_col).str.split('.').list.slice(0, 3).list.join('.') + pl.lit('.0/24')
|
||||
)
|
||||
# → "192.168.1.0/24"
|
||||
```
|
||||
|
||||
生成的 `unique_24_networks` 在聚类时作为特征输入，反映实体的网络分布广度。
|
||||
|
||||
---
|
||||
|
||||
## 6. 类型感知聚合
|
||||
|
||||
### 6.1 聚合原理
|
||||
|
||||
**主函数**：`analysis/entity_aggregator.py::aggregate_by_entity()`
|
||||
|
||||
1. 扫描 schema 通过关键词匹配找到可用列
|
||||
2. 通过 `type_classifier` 获取每列的 `DataType` 分类
|
||||
3. 按数据类型选择对应的聚合函数
|
||||
|
||||
### 6.2 数据类型→聚合函数映射
|
||||
|
||||
| `DataType` | 聚合方式 | 特征命名 | 代码位置 |
|
||||
|-----------|---------|---------|---------|
|
||||
| **INT** (整数) | `mean()` 均值 | `total_bytes_sent`, `total_packets` | `build_aggregation_params()` |
|
||||
| **FLOAT** (浮点) | `mean()` 均值 | `avg_duration`, `avg_latitude` | `build_aggregation_params()` |
|
||||
| **ENUM** (枚举) | `mode().first()` 众数 | `tls_version_mode`, `tcp_flag_modes` | `build_aggregation_params()` |
|
||||
| **HEX** (十六进制) | `str.extract_all().list.len().mean()` 统计 hex 对数量 | `avg_hex_pairs_count` | `build_aggregation_params()` |
|
||||
| **URL** (链接) | `str.extract(r'https?://([^/]+)').n_unique()` 统计唯一域名 | `unique_domains` | `build_aggregation_params()` |
|
||||
| **IPv4** (地址) | `str.split('.').list.slice(0,3).join('.') + '.0/24'.n_unique()` 子网多样性 | `unique_24_networks` | `build_aggregation_params()` |
|
||||
|
||||
### 6.3 聚合特征分类
|
||||
|
||||
所有生成的聚合特征分为 6 大类：
|
||||
|
||||
| 类别 | 特征 | 说明 |
|
||||
|------|------|------|
|
||||
| **流统计** | `flow_count`、`total_bytes_sent`、`total_bytes_rev`、`avg_duration`、`total_packets` | 实体流量基本量 |
|
||||
| **目标多样性** | `unique_dst_ips`、`unique_dst_ports`、`unique_dst_networks` | 目的地址分布广度 |
|
||||
| **TLS 特征** | `unique_ciphers`、`tls_version_mode`、`unique_versions`、`sni_count` | 加密和证书多样性 |
|
||||
| **协议特征** | `unique_protocols`、`tcp_flag_modes` | 协议和标志分布 |
|
||||
| **地理位置** | `avg_latitude`、`avg_longitude` | 经纬度均值 |
|
||||
| **时间模式** | `first_seen`、`last_seen`、`active_hours_count` | 活动时间段 |
|
||||
|
||||
### 6.4 类型安全防范
|
||||
|
||||
build_aggregation_params 在聚合前检查列的实际 Polars dtype：
|
||||
|
||||
```python
|
||||
# 在聚合非数值列前 cast
|
||||
if schema.get(col_name) in ('Utf8', 'String'):
|
||||
pl.col(col_name).cast(pl.Float64)
|
||||
```
|
||||
|
||||
这解决了跨机"mean on str"错误（当同一数据在不同机器上被 Polars 推断为不同 dtypes 时）。
|
||||
|
||||
---
|
||||
|
||||
## 7. 聚类分析
|
||||
|
||||
### 7.1 算法选择
|
||||
|
||||
| 算法 | 适用场景 | 优点 | 缺点 |
|
||||
|------|---------|------|------|
|
||||
| **HDBSCAN**（默认） | 未知簇数、存在噪声点 | 自动确定簇数；噪声点处理；无需预设 k | 对 `min_cluster_size` 敏感；大数据集计算慢 |
|
||||
| **KMeans** | 已知簇数、球形分布 | 速度快；可预设 k | 需指定 n_clusters；对离群点敏感 |
|
||||
|
||||
### 7.2 参数建议
|
||||
|
||||
**HDBSCAN 参数**：
|
||||
|
||||
| 参数 | 默认值 | 说明 | 调整建议 |
|
||||
|------|--------|------|---------|
|
||||
| `min_cluster_size` | 5 | 最小簇样本数 | 数据量大时增大（10~20）；数据少时自动降级为 `max(len//2, 2)` |
|
||||
| `min_samples` | None | 核心点邻域 | 默认与 min_cluster_size 相同；增大产生更保守的聚类 |
|
||||
| `metric` | euclidean | 距离度量 | 高维特征时考虑 `cosine` |
|
||||
| `cluster_selection_epsilon` | 0.0 | 簇选择阈值 | 越大簇越少 |
|
||||
|
||||
**KMeans 参数**：
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `n_clusters` | 3 | 簇数 |
|
||||
| `random_state` | 42 | 随机种子（保证可复现性） |
|
||||
|
||||
### 7.3 零样本降级
|
||||
|
||||
当样本数过少或全为缺失值时，系统自动降级：
|
||||
|
||||
```python
|
||||
if len(data) < 3:
|
||||
return {"n_clusters": 0, "n_noise": 0, "quality_metrics": {"note": "too few samples"}}
|
||||
```
|
||||
|
||||
同样，全部为 NaN 的列在聚类前自动丢弃。
|
||||
|
||||
### 7.4 预处理步骤
|
||||
|
||||
聚类执行前的自动预处理流程：
|
||||
|
||||
1. **列过滤**：只保留请求的特征列
|
||||
2. **全 NaN 列丢弃**：无用列自动移除
|
||||
3. **NaN 填充**：数值 NaN 用该列均值填充
|
||||
4. **StandardScaler**：所有特征标准化（零均值、单位方差）
|
||||
5. **min_cluster_size 自适应**：不超过样本数的 50%
|
||||
|
||||
### 7.5 质量评估
|
||||
|
||||
| 指标 | 范围 | 说明 |
|
||||
|------|------|------|
|
||||
| **Silhouette Score** | `[-1, 1]` | >0.5 表示良好聚类；>0.7 表示优秀 |
|
||||
| **Davies-Bouldin Index** | `[0, +∞)` | 越低越好（簇内紧凑、簇间分离） |
|
||||
| **Calinski-Harabasz Score** | `[0, +∞)` | 越高越好 |
|
||||
| **Noise Ratio** | `[0, 1]` | HDBSCAN 特有的噪声点比例 |
|
||||
|
||||
### 7.6 CLI 运行示例
|
||||
|
||||
```bash
|
||||
# 完整管道（HDBSCAN 默认）
|
||||
runtime\python\python.exe manage.py run_pipeline data/complex_test.csv
|
||||
|
||||
# 指定 KMeans
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --algo kmeans
|
||||
|
||||
# 手动指定实体列
|
||||
runtime\python\python.exe manage.py run_pipeline data/*.csv --entity-col src_ip
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 子网实体聚合
|
||||
|
||||
config.yaml中配置:
|
||||
```yaml
|
||||
entity:
|
||||
subnet_masks: [24, 28] # 子网掩码列表
|
||||
ip_columns: ["src_ip", "dst_ip"] # 需要聚合的IP列
|
||||
```
|
||||
|
||||
启用后，同一子网（如 10.0.1.0/24）下的所有 IP 合并为一个实体节点。
|
||||
支持 /24（0-255）和 /28（240-255, 16个IP分组）两种掩码。
|
||||
|
||||
`entity_aggregator.ip_to_subnet()` 负责 IP 到 CIDR 子网前缀的转换，
|
||||
`apply_subnet_aggregation()` 在 group_by 前对 IP 列应用子网掩码。
|
||||
|
||||
```python
|
||||
# 示例: IP → /24 子网
|
||||
ip_to_subnet("192.168.1.42", 24) # → "192.168.1.0/24"
|
||||
|
||||
# 示例: IP → /28 子网
|
||||
ip_to_subnet("192.168.1.42", 28) # → "192.168.1.32/28"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 特征提取
|
||||
|
||||
### 8.1 方法对比
|
||||
|
||||
**主函数**：`analysis/tool_registry.py` → `_handle_extract_features()`
|
||||
|
||||
| 方法 | 公式 | 适用场景 |
|
||||
|------|------|---------|
|
||||
| **Z-Score**（默认） | `(μ_cluster - μ_global) / σ_global` | 簇中心 vs 全局均值的偏离程度 |
|
||||
| **ANOVA** | `|μ_cluster - μ_global| / (σ_cluster + σ_global)` | 考虑簇内方差，更保守 |
|
||||
|
||||
### 8.2 区分特征解读
|
||||
|
||||
返回的每个特征包含：
|
||||
|
||||
```json
|
||||
{
|
||||
"feature_name": "total_bytes_sent",
|
||||
"cluster_label": 0,
|
||||
"mean": 1234567.89, // 该簇均值
|
||||
"std": 234567.89, // 该簇标准差
|
||||
"global_mean": 987654.32, // 全局均值
|
||||
"global_std": 765432.10, // 全局标准差
|
||||
"distinguishing_score": 2.34, // Z-Score 或 ANOVA 分数
|
||||
"distinguishing_method": "zscore"
|
||||
}
|
||||
```
|
||||
|
||||
**解读方法**：
|
||||
|
||||
| 分数范围 | 含义 |
|
||||
|---------|------|
|
||||
| \|Z\| < 1.0 | 无明显区分力 |
|
||||
| 1.0 ≤ \|Z\| < 2.0 | 中等区分力 |
|
||||
| \|Z\| ≥ 2.0 | **强区分特征**（在 Web 界面中以颜色高亮） |
|
||||
|
||||
### 8.3 地理分布可视化
|
||||
|
||||
如果聚合特征包含 `avg_latitude` 和 `avg_longitude`，聚类概览页会同时显示：
|
||||
|
||||
- **PCA 散点图**：实体在二维主成分空间的分布，颜色区分聚类
|
||||
- **地理分布图**：实体在地理坐标上的分布
|
||||
|
||||
地理位置在 Web 界面中通过 Canvas 2D API 原生绘制（无外部 JS 依赖）：
|
||||
|
||||
```python
|
||||
# views.py 中提取逻辑
|
||||
lat = _extract_lat(feature_json.get('avg_latitude'))
|
||||
lon = _extract_lon(feature_json.get('avg_longitude'))
|
||||
# 过滤无效值：纬度 -90~90，经度 -180~180
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 可视化
|
||||
|
||||
### 9.1 PCA 散点图
|
||||
路径: `/clusters/<run_id>/`
|
||||
使用纯 Canvas 2D 绘制的散点图，展示实体在二维主成分空间中的分布。
|
||||
点颜色按聚类标签区分，鼠标悬停显示实体名称。
|
||||
依赖: `run_pipeline` 步骤 6 自动计算 PCA-2D 坐标并写入 `EntityProfile` 表。
|
||||
|
||||
### 9.2 3D 地球流量弧线
|
||||
路径: `/globe/`
|
||||
使用 Three.js (603KB, 全离线) 绘制的三维地球，显示源 IP 到目标 IP 的 TLS 流量弧线。
|
||||
- 颜色: TLSv1.3 (蓝) / TLSv1.2 (粉) / 其他 (青)
|
||||
- 弧线高度根据距离自动调整
|
||||
- 鼠标拖拽旋转地球
|
||||
- 地球贴图: 1.4MB (离线内嵌)
|
||||
- **多数据源叠加**: 页面复选框选择多个运行，弧线叠加在同一地球上
|
||||
- **经纬度来源**: 优先使用数据中`src_latitude/src_longitude/dst_latitude/dst_longitude`列，无则回退GeoIP查询
|
||||
- GeoIP: 从 `data/geoip_city.csv` 查询 IP 对应经纬度
|
||||
- 滚轮缩放: 鼠标滚轮控制相机距离(3-30)
|
||||
- 经纬线: 10度间隔的白色半透明辅助线
|
||||
- 国境线: world_borders.js 10国精简轮廓，含台湾和南海九段线示意
|
||||
- **背面剔除**: `updatePulses` 中使用 `localToWorld` 替代 `applyQuaternion`，确保弧线脉冲只在面向相机的一侧显示，不穿透地球
|
||||
- **脉冲动画**: 弧线上移动的贝塞尔曲线脉冲光点，`arcGroup.localToWorld(midPt)` 计算世界坐标，`midPt.dot(camDir) < -0.08` 判段正面可见性
|
||||
- 多数据选择: 复选框选择多个运行叠加弧线
|
||||
- 经纬度来源: 优先使用数据列 src_latitude/src_longitude 等，无则GeoIP回退
|
||||
|
||||
### 9.3 聚类概览页
|
||||
路径: `/clusters/<run_id>/`
|
||||
同时显示 PCA 散点图和每簇的 Top 区分特征。
|
||||
|
||||
## 10. LLM 集成
|
||||
|
||||
### 9.1 API 配置
|
||||
|
||||
通过 `config/config.yaml` 或 Web 界面「配置」页面设置：
|
||||
|
||||
```yaml
|
||||
llm:
|
||||
enabled: true # 启用 LLM 功能
|
||||
base_url: "https://api.openai.com/v1" # OpenAI 兼容 API 地址
|
||||
api_key: "sk-..." # API Key
|
||||
model: "deepseek-v4-flash" # 模型名称
|
||||
max_tokens: 2048
|
||||
temperature: 0.1
|
||||
```
|
||||
|
||||
支持的 LLM 服务：
|
||||
- OpenAI GPT-4 / GPT-3.5
|
||||
- Azure OpenAI Service
|
||||
- 本地部署的 Ollama / vLLM / LM Studio
|
||||
- 其他 OpenAI 兼容 API
|
||||
|
||||
### 9.2 32B 模型优化
|
||||
|
||||
系统针对 32B 等较大模型的推理特性做了以下优化（`tianxuan/llm_orchestrator.py`）：
|
||||
|
||||
| 优化项 | 实现 | 说明 |
|
||||
|--------|------|------|
|
||||
| **Step Guidance** | 每步告诉模型下一步做什么 | 减少 32B 模型在复杂推理中的发散 |
|
||||
| **工具结果截断** | 截断至 1200 chars | 避免上下文窗口溢出 |
|
||||
| **超时** | 90s | 32B 推理更慢，比默认超时长 |
|
||||
| **最大步数** | 15 步 | 限制过长的工具调用链 |
|
||||
| **工具描述** | 一行简洁描述 | 减少冗长描述对 token 的消耗 |
|
||||
|
||||
### 9.3 LLM 分析流程
|
||||
|
||||
自动化编排调用链：
|
||||
|
||||
```
|
||||
Step 1: profile_data("ds_...") → 了解数据集的列统计
|
||||
Step 2: build_entity_profiles("ds_...") → 实体聚合
|
||||
Step 3: run_clustering("entity_...") → 聚类分析
|
||||
Step 4: extract_features("entity_...") → 特征提取
|
||||
Step 5: 总结分析结果
|
||||
```
|
||||
|
||||
### 9.4 日志链解读
|
||||
|
||||
LLM 交互日志以 `[LLM]` 和 `[TOOL]` 标记输出到 `logs/tianxuan.log`：
|
||||
|
||||
```
|
||||
[LLM] model=deepseek-v4-flash # 模型名
|
||||
[LLM] latency_ms=3245 # LLM 响应延迟
|
||||
[LLM REQUEST] ... # 请求内容（缩减后）
|
||||
[LLM RESPONSE] ... # 响应状态
|
||||
[TOOL] profile_data # LLM 调用的工具
|
||||
[TOOL] build_entity_profiles
|
||||
[TOOL] run_clustering
|
||||
[TOOL] extract_features
|
||||
```
|
||||
|
||||
### 9.5 故障排查
|
||||
|
||||
| 问题 | 常见原因 | 排查方法 |
|
||||
|------|---------|---------|
|
||||
| 400 错误 | API 地址错误 / Key 无效 | 配置页「测试连通性」 |
|
||||
| 超时 | 32B 推理慢 / 网络延迟 | 检查 `latency_ms`；增大 `max_tokens` |
|
||||
| 工具调用失败 | LLM 返回格式错误 | 检查 `[TOOL]` 日志看哪个工具调用失败 |
|
||||
| 分析结果为空 | 实体检测失败 | 检查实体列是否正确，尝试手动指定 |
|
||||
|
||||
**连通性测试**：
|
||||
|
||||
```bash
|
||||
# CLI 测试
|
||||
runtime\python\python.exe scripts\test_llm_full.py
|
||||
|
||||
# Web 测试
|
||||
# 导航到「配置」页面 → 填写 LLM 配置 → 点击「测试连通性」
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. 跨机部署
|
||||
|
||||
### 10.1 运行时隔离验证
|
||||
|
||||
**脚本**：`scripts/verify_runtime.py`
|
||||
|
||||
在不同机器上运行该脚本，输出应完全一致：
|
||||
|
||||
```bash
|
||||
# 每台机器各自执行
|
||||
runtime\python\python.exe scripts\verify_runtime.py
|
||||
```
|
||||
|
||||
验证内容：
|
||||
|
||||
| 检查项 | 期望值 |
|
||||
|--------|--------|
|
||||
| Python 版本 | 3.12.x |
|
||||
| ENABLE_USER_SITE | False（隔离） |
|
||||
| django | 4.2.x |
|
||||
| polars | 1.42.x |
|
||||
| sklearn | 1.5.x |
|
||||
| numpy | 1.26.x |
|
||||
| defaultencoding | utf-8 |
|
||||
| filesystemencoding | utf-8 |
|
||||
|
||||
### 10.2 diagnose_compare 使用
|
||||
|
||||
**脚本**：`scripts/diagnose_compare.py`
|
||||
|
||||
对比两台机器的运行日志，定位第一个差异点：
|
||||
|
||||
```bash
|
||||
# 每台机器运行管道，生成日志
|
||||
del logs\tianxuan.log
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
||||
copy logs\tianxuan.log log_机器A.txt
|
||||
|
||||
# 在开发机上对比
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_开发机.txt log_机器A.txt
|
||||
```
|
||||
|
||||
输出示例：
|
||||
|
||||
```
|
||||
[DIFF] 首个差异在行 123:
|
||||
--- 机器A ---
|
||||
[LOAD_SCHEMA] col=duration dtype=Utf8 sample=1.2s,0.5s,3.1s
|
||||
--- 机器B ---
|
||||
[LOAD_SCHEMA] col=duration dtype=Float64 sample=1.2,0.5,3.1
|
||||
```
|
||||
|
||||
首个 `[DIFF]` 行就是问题根源。
|
||||
|
||||
### 10.3 关键日志标记
|
||||
|
||||
| 标记 | 对应文件 | 含义 |
|
||||
|------|---------|------|
|
||||
| `[LOAD]` | `data_loader.py` | CSV 加载完成 |
|
||||
| `[LOAD_SCHEMA]` | `data_loader.py` | 每列的推断类型（最重要！） |
|
||||
| `[CLEAN]` | `data_loader.py` | 清洗操作记录 |
|
||||
| `[VALIDATE]` | `data_validator.py` | 数据校验结果 |
|
||||
| `[FIND_COL]` | `entity_aggregator.py` | 关键词匹配结果 |
|
||||
| `[CLUSTER_INPUT]` | `tool_registry.py` | 聚类输入形状 |
|
||||
| `[GEOIP]` | `geoip.py` | GeoIP 数据加载 |
|
||||
| `[LLM]` | `llm_orchestrator.py` | LLM 调用状态 |
|
||||
| `[TOOL]` | `llm_orchestrator.py` | LLM 调用的工具 |
|
||||
| `[DB_SAVE_ERROR]` | `tool_registry.py` | 数据库持久化错误 |
|
||||
|
||||
### 10.4 跨机一致性检查清单
|
||||
|
||||
当两台机器表现不一致时，逐项排查：
|
||||
|
||||
```bash
|
||||
# 1. Python 版本
|
||||
runtime\python\python.exe --version
|
||||
|
||||
# 2. 编码设置
|
||||
runtime\python\python.exe -c "
|
||||
import sys; print(f'defaultenc={sys.getdefaultencoding()}')
|
||||
print(f'fsenc={sys.getfilesystemencoding()}')
|
||||
print(f'utf8_mode={sys.flags.utf8_mode}')
|
||||
"
|
||||
|
||||
# 3. Polars 版本
|
||||
runtime\python\python.exe -c "import polars; print(polars.__version__)"
|
||||
|
||||
# 4. 依赖版本
|
||||
runtime\python\python.exe -c "import numpy; print(numpy.__version__)"
|
||||
runtime\python\python.exe -c "import sklearn; print(sklearn.__version__)"
|
||||
```
|
||||
|
||||
### 10.5 故障诊断流程
|
||||
|
||||
```
|
||||
出现跨机差异
|
||||
│
|
||||
├─ Step 1: 收集两台机器的 logs/tianxuan.log
|
||||
│
|
||||
├─ Step 2: 运行 diagnose_compare.py 对比日志
|
||||
│ └─ 找到首个 [DIFF]
|
||||
│
|
||||
├─ Step 3: 判断差异类型
|
||||
│ ├─ [LOAD_SCHEMA] 列类型不同
|
||||
│ │ └─ 检查 Polars 版本一致、PYTHONUTF8 生效
|
||||
│ │
|
||||
│ ├─ [FIND_COL] 关键词匹配不同
|
||||
│ │ └─ 确认 frozenset→tuple 修复已应用（消除哈希随机性）
|
||||
│ │
|
||||
│ └─ [CLUSTER_INPUT] 聚类结果不同
|
||||
│ └─ 检查 random_state=42、StandardScaler 一致性
|
||||
│
|
||||
├─ Step 4: 应急修复
|
||||
│ ├─ 手动指定实体列：--entity-col src_ip
|
||||
│ ├─ 强制编码：set PYTHONUTF8=1
|
||||
│ └─ 使用 tuple 替代 frozenset（已在 entity_aggregator.py 中修复）
|
||||
│
|
||||
└─ Step 5: 归档诊断信息
|
||||
( echo === 天璇故障诊断报告 ===
|
||||
date /t & time /t
|
||||
runtime\python\python.exe --version
|
||||
type logs\tianxuan.log
|
||||
) > 诊断报告_%COMPUTERNAME%.txt
|
||||
```
|
||||
|
||||
### 10.6 已知跨机问题（已修复）
|
||||
|
||||
| 问题 | 根因 | 修复 |
|
||||
|------|------|------|
|
||||
| "mean on str" 错误 | frozenset 哈希随机性 | frozenset → tuple 确定性迭代 |
|
||||
| "ZZZ不在{XXX,YYY}中" 错误 | Python 编码不一致 | `PYTHONUTF8=1` 强制 UTF-8 |
|
||||
| DB 特征不保存 | Django ORM 在 async 上下文中的限制 | `sync_to_async` 包装 |
|
||||
| 图表不显示 | Chart.js 从未下载 | 改用纯 Canvas 2D API |
|
||||
|
||||
---
|
||||
|
||||
> **更多技术细节**：请参阅 `docs/故障诊断手册.md` 获取跨机问题深度排查指南。
|
||||
+282
@@ -0,0 +1,282 @@
|
||||
# 天璇跨机故障诊断手册
|
||||
|
||||
## 问题描述
|
||||
|
||||
同一份项目压缩包、同一份 CSV 数据（SHA256 一致）、同一份 `runtime/python/` 运行时，
|
||||
在不同 Windows 10 x64 机器上出现不同行为：
|
||||
|
||||
| 设备 | 表现 |
|
||||
|------|------|
|
||||
| 你的机器 | ✅ 正常运行 |
|
||||
| 第二台 | ❌ "尝试对str类型求mean" |
|
||||
| 第三台 | ❌ "ZZZ不在{XXX,YYY}中" + 字符串相关错误 |
|
||||
| (可能的更多) | ❌ 其他奇怪错误 |
|
||||
|
||||
---
|
||||
|
||||
## 1. 立即诊断流程
|
||||
|
||||
### 步骤 A：收集两台机器的日志
|
||||
|
||||
在两台**问题机器**上各执行一次：
|
||||
|
||||
```bash
|
||||
# 1. 清理旧日志
|
||||
del logs\tianxuan.log 2>nul
|
||||
|
||||
# 2. 生成测试数据（确保 seed 一致）
|
||||
runtime\python\python.exe scripts\gen_complex_test.py --rows 5000 --output data\complex_test.csv
|
||||
|
||||
# 3. 运行完整管道
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv --algo hdbscan
|
||||
|
||||
# 4. 复制日志到桌面备用
|
||||
copy logs\tianxuan.log %USERPROFILE%\Desktop\log_机器A.txt
|
||||
```
|
||||
|
||||
在**你的机器**上也执行一次，生成 `log_你的机器.txt`。
|
||||
|
||||
### 步骤 B：对比日志
|
||||
|
||||
```bash
|
||||
runtime\python\python.exe scripts\diagnose_compare.py log_你的机器.txt log_机器A.txt
|
||||
```
|
||||
|
||||
输出示例：
|
||||
```
|
||||
[DIFF] 首个差异在行 123:
|
||||
--- 机器A ---
|
||||
[LOAD_SCHEMA] col=duration dtype=Utf8 sample=1.2s,0.5s,3.1s
|
||||
--- 机器B ---
|
||||
[LOAD_SCHEMA] col=duration dtype=Float64 sample=1.2,0.5,3.1
|
||||
```
|
||||
|
||||
第一个 `[DIFF]` 就是导致问题的根源。
|
||||
|
||||
---
|
||||
|
||||
## 2. 关键日志标记解读
|
||||
|
||||
日志中所有 `[TAG]` 标记的含义：
|
||||
|
||||
| 标记 | 对应文件 | 含义 |
|
||||
|------|---------|------|
|
||||
| `[LOAD]` | `data_loader.py` | CSV 加载完成，文件数和预估行数 |
|
||||
| `[LOAD_SCHEMA]` | `data_loader.py` | 每列的推断类型（最重要！） |
|
||||
| `[FIND_COL]` | `entity_aggregator.py` | 关键词匹配结果，显示匹配到的列名和类型 |
|
||||
| `[AGGR_EXPR]` | `entity_aggregator.py` | 聚合表达式 |
|
||||
| `[CLUSTER_INPUT]` | `tool_registry.py` | 聚类输入数据形状 |
|
||||
| `[LLM REQUEST]` | `llm_orchestrator.py` | LLM 请求内容 |
|
||||
| `[LLM RESPONSE]` | `llm_orchestrator.py` | LLM 响应状态 |
|
||||
| `[TOOL CALL]` | `llm_orchestrator.py` | LLM 调用的工具 |
|
||||
| `[TOOL RESULT]` | `llm_orchestrator.py` | 工具执行结果 |
|
||||
|
||||
---
|
||||
|
||||
## 3. "尝试对str类型求mean" 故障排查
|
||||
|
||||
### 根因树
|
||||
|
||||
```
|
||||
"尝试对str类型求mean" 报错
|
||||
├── 调用链: build_aggregation_params → pl.mean(col_name)
|
||||
│
|
||||
├── [可能A] 列名关键词匹配到了错误的列
|
||||
│ ├── 检查 [FIND_COL] keywords=duration matched=xxx
|
||||
│ ├── 如果 matched 列的类型是 Utf8/String → 匹配错误
|
||||
│ └── 修复: entity_aggregator.py 中 frozenset→tuple (已改)
|
||||
│
|
||||
├── [可能B] CSV 列的实际类型与推断类型不同
|
||||
│ ├── 检查 [LOAD_SCHEMA] col=duration dtype=xxx
|
||||
│ ├── 对比两台机器的 dtype 是否一致
|
||||
│ ├── 如果一致但行为不同 → 可能是 Polars 版本差异
|
||||
│ └── 验证: runtime\python\python.exe -c "import polars; print(polars.__version__)"
|
||||
│
|
||||
├── [可能C] PYTHONUTF8 环境变量未生效
|
||||
│ ├── 验证: runtime\python\python.exe -c "import sys; print(sys.getdefaultencoding())"
|
||||
│ ├── 期望输出: utf-8
|
||||
│ ├── 如果输出 cp936/gbk → PYTHONUTF8 没生效
|
||||
│ └── 修复: run.bat 中 set PYTHONUTF8=1 (已加)
|
||||
│
|
||||
└── [可能D] 聚合表达式中混入了 Utf8 列
|
||||
├── 检查 [FIND_COL] keywords=duration matched=duration dtype=Utf8
|
||||
├── 这意味着 duration 列被 Polars 推断为字符串
|
||||
└── 修复: entity_aggregator.py 的 dtype 安全聚合 (已改)
|
||||
```
|
||||
|
||||
### 诊断命令
|
||||
|
||||
```bash
|
||||
# 1. 检查 Python 默认编码
|
||||
runtime\python\python.exe -c "import sys; print(f'enc={sys.getdefaultencoding()} fs={sys.getfilesystemencoding()}')"
|
||||
|
||||
# 2. 检查 Polars 版本
|
||||
runtime\python\python.exe -c "import polars; print(polars.__version__)"
|
||||
|
||||
# 3. 检查 CSV 加载后的实际列类型
|
||||
runtime\python\python.exe -c "
|
||||
import polars as pl
|
||||
lf = pl.scan_csv(r'data\complex_test.csv', infer_schema_length=10000)
|
||||
schema = lf.collect_schema()
|
||||
for name, dtype in zip(schema.names(), schema.dtypes()):
|
||||
print(f'{name}: {dtype}')
|
||||
"
|
||||
|
||||
# 4. 检查关键词匹配结果
|
||||
runtime\python\python.exe -c "
|
||||
import sys; sys.path.insert(0, '.')
|
||||
import os; os.environ['DJANGO_SETTINGS_MODULE']='tianxuan.settings'
|
||||
import django; django.setup()
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_aggregator import _find_column, _BUILT_AGGR
|
||||
|
||||
store = SessionStore(); store.drop_all()
|
||||
lf, schema, rc, fc, mem = load_csv_directory(r'data\complex_test.csv')
|
||||
for kw_name, kw_set in [
|
||||
('bytes_sent',_BYTES_SENT_KEYWORDS),('duration',_DURATION_KEYWORDS),
|
||||
('dst_ip',_DST_IP_KEYWORDS),('cipher',_CIPHER_KEYWORDS)]:
|
||||
col = _find_column(schema, kw_set)
|
||||
dtype = schema.get(col, 'N/A') if col else 'N/A'
|
||||
print(f'{kw_name:15s} -> {str(col):25s} dtype={dtype}')
|
||||
" 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. "ZZZ不在{XXX,YYY}中" 故障排查
|
||||
|
||||
### 根因
|
||||
|
||||
这个错误来自 Polars 的 `str.contains()` 或 `is_in()` 操作。当比较字符串时，
|
||||
两边的编码不一致导致匹配失败。
|
||||
|
||||
### 诊断
|
||||
|
||||
```bash
|
||||
# 检查 Python 编码
|
||||
runtime\python\python.exe -c "
|
||||
import sys; print(f'default={sys.getdefaultencoding()}')
|
||||
print(f'filesystem={sys.getfilesystemencoding()}')
|
||||
print(f'stdin={sys.stdin.encoding}')
|
||||
print(f'stdout={sys.stdout.encoding}')
|
||||
"
|
||||
|
||||
# 检查 CSV 文件的实际编码（在 Notepad++ 或 VS Code 中查看右下角编码提示）
|
||||
# 期望: UTF-8 (无 BOM)
|
||||
```
|
||||
|
||||
### 修复尝试
|
||||
|
||||
```bash
|
||||
# 修复 1: 确认 run.bat 包含 PYTHONUTF8=1
|
||||
findstr PYTHONUTF8 run.bat
|
||||
# 应输出: set PYTHONUTF8=1
|
||||
|
||||
# 修复 2: 手动设置编码后再运行
|
||||
set PYTHONUTF8=1
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 跨机环境一致性检查清单
|
||||
|
||||
在**每台机器**上执行以下命令，对比输出：
|
||||
|
||||
```bash
|
||||
echo === 1. Python 版本 ===
|
||||
runtime\python\python.exe --version
|
||||
|
||||
echo === 2. 编码设置 ===
|
||||
runtime\python\python.exe -c "
|
||||
import sys; print(f'defaultenc={sys.getdefaultencoding()}')
|
||||
print(f'fsenc={sys.getfilesystemencoding()}')
|
||||
print(f'has_utf8_mode={sys.flags.utf8_mode}')
|
||||
"
|
||||
|
||||
echo === 3. Polars 版本 ===
|
||||
runtime\python\python.exe -c "import polars; print(polars.__version__)"
|
||||
|
||||
echo === 4. numpy 版本 ===
|
||||
runtime\python\python.exe -c "import numpy; print(numpy.__version__)"
|
||||
|
||||
echo === 5. sklearn 版本 ===
|
||||
runtime\python\python.exe -c "import sklearn; print(sklearn.__version__)"
|
||||
|
||||
echo === 6. 系统区域 ===
|
||||
systeminfo | findstr "区域"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 应急修复方案
|
||||
|
||||
如果 `diagnose_compare.py` 显示的第一个差异是 `[LOAD_SCHEMA]` 列类型不同：
|
||||
|
||||
```bash
|
||||
# 强制所有列读为字符串 → 让 Polars 自己推断
|
||||
runtime\python\python.exe -c "
|
||||
import polars as pl
|
||||
lf = pl.scan_csv(r'data\complex_test.csv', infer_schema_length=100000)
|
||||
schema = lf.collect_schema()
|
||||
# 打印所有列的推断类型
|
||||
for n, d in zip(schema.names(), schema.dtypes()):
|
||||
print(f'{n}: {d}')
|
||||
"
|
||||
```
|
||||
|
||||
如果差异在 `[FIND_COL]` 关键词匹配不同：
|
||||
|
||||
```bash
|
||||
# 手动指定实体列后运行
|
||||
runtime\python\python.exe manage.py run_pipeline data\complex_test.csv --entity-col src_ip
|
||||
```
|
||||
|
||||
如果完全无法解决：
|
||||
|
||||
```bash
|
||||
# 重建运行时（在开发机上有网络的机器上）
|
||||
# 1. 删除旧的 runtime
|
||||
rmdir /s runtime
|
||||
|
||||
# 2. 重新运行构建脚本
|
||||
scripts\build_runtime.bat
|
||||
|
||||
# 3. 重新压缩分发
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 已知已修复的问题清单
|
||||
|
||||
| 问题编号 | 修复内容 | 涉及文件 | 验证方式 |
|
||||
|---------|---------|---------|---------|
|
||||
| W4-#1 | `frozenset`→`tuple` 消除哈希随机性 | `entity_aggregator.py` | 查看文件确认无 `frozenset` |
|
||||
| W4-#2 | `PYTHONUTF8=1` 强制 UTF-8 编码 | `run.bat` | `findstr PYTHONUTF8 run.bat` |
|
||||
| W4-#3 | `sorted(keywords)` 确定性关键词迭代 | `entity_aggregator.py` | 查看 `_find_column` 函数 |
|
||||
| W4-#4 | 结构化 `[TAG]` 日志 | 多文件 | 运行后查看 `logs/tianxuan.log` |
|
||||
| DB-#1 | `sync_to_async` 解决异步 ORM 问题 | `tool_registry.py` | 管道完成后特征写入 DB |
|
||||
| DB-#2 | `value_counts()` 列名兼容 | `data_profiler.py` | `profile_data` 工具正常返回 |
|
||||
| DB-#3 | numpy corrcoef 替代 pandas | `data_profiler.py` | 无 pandas 时也能跑 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 归档与提交诊断结果
|
||||
|
||||
```bash
|
||||
# 收集诊断信息到单个文件，发给开发者
|
||||
(
|
||||
echo === 天璇故障诊断报告 ===
|
||||
date /t
|
||||
time /t
|
||||
echo.
|
||||
echo === 环境信息 ===
|
||||
runtime\python\python.exe --version
|
||||
runtime\python\python.exe -c "import sys; print(f'enc={sys.getdefaultencoding()}')"
|
||||
runtime\python\python.exe -c "import polars; print(f'polars={polars.__version__}')"
|
||||
echo.
|
||||
echo === 日志最后 50 行 ===
|
||||
type logs\tianxuan.log 2>nul
|
||||
) > 诊断报告.txt
|
||||
```
|
||||
@@ -0,0 +1,139 @@
|
||||
{
|
||||
"run_info": {
|
||||
"dataset": "data/wnl_converted.csv",
|
||||
"total_rows": 3547,
|
||||
"n_entities": 834,
|
||||
"n_clusters": 6
|
||||
},
|
||||
"quality_metrics": {
|
||||
"silhouette": 0.7332,
|
||||
"davies_bouldin": 7.6658,
|
||||
"calinski_harabasz": 792.0156,
|
||||
"noise_ratio": 0.1355,
|
||||
"purity": 0.2434,
|
||||
"ari": 0.0001,
|
||||
"nmi": 0.0236
|
||||
},
|
||||
"cluster_sizes": {
|
||||
"-1": 113,
|
||||
"0": 80,
|
||||
"1": 259,
|
||||
"2": 160,
|
||||
"3": 82,
|
||||
"4": 43,
|
||||
"5": 97
|
||||
},
|
||||
"cluster_service_map": {
|
||||
"-1": {
|
||||
"SoundCloud": 27,
|
||||
"Web": 25,
|
||||
"Kinopoisk": 24,
|
||||
"Spotify": 17,
|
||||
"YandexMusic": 10,
|
||||
"Netflix": 6,
|
||||
"AppleMusic": 2,
|
||||
"YouTube": 1,
|
||||
"PrimeVideo": 1
|
||||
},
|
||||
"0": {
|
||||
"Spotify": 21,
|
||||
"YandexMusic": 19,
|
||||
"SoundCloud": 17,
|
||||
"Kinopoisk": 10,
|
||||
"Web": 9,
|
||||
"Netflix": 3,
|
||||
"YouTube": 1
|
||||
},
|
||||
"1": {
|
||||
"Web": 61,
|
||||
"Spotify": 51,
|
||||
"Kinopoisk": 44,
|
||||
"SoundCloud": 44,
|
||||
"YandexMusic": 37,
|
||||
"Netflix": 11,
|
||||
"AppleMusic": 5,
|
||||
"YouTube": 4,
|
||||
"PrimeVideo": 2
|
||||
},
|
||||
"2": {
|
||||
"SoundCloud": 39,
|
||||
"YandexMusic": 34,
|
||||
"Web": 33,
|
||||
"Spotify": 27,
|
||||
"Kinopoisk": 13,
|
||||
"Netflix": 9,
|
||||
"PrimeVideo": 2,
|
||||
"YouTube": 1,
|
||||
"LiveYouTube": 1,
|
||||
"AppleMusic": 1
|
||||
},
|
||||
"3": {
|
||||
"Spotify": 19,
|
||||
"Web": 18,
|
||||
"SoundCloud": 18,
|
||||
"YandexMusic": 11,
|
||||
"Kinopoisk": 8,
|
||||
"Netflix": 7,
|
||||
"AppleMusic": 1
|
||||
},
|
||||
"4": {
|
||||
"Web": 12,
|
||||
"Spotify": 11,
|
||||
"SoundCloud": 10,
|
||||
"Kinopoisk": 4,
|
||||
"YandexMusic": 2,
|
||||
"Netflix": 2,
|
||||
"PrimeVideo": 1,
|
||||
"LiveYouTube": 1
|
||||
},
|
||||
"5": {
|
||||
"Web": 24,
|
||||
"SoundCloud": 23,
|
||||
"Spotify": 19,
|
||||
"Kinopoisk": 12,
|
||||
"YandexMusic": 9,
|
||||
"Netflix": 5,
|
||||
"YouTube": 2,
|
||||
"AppleMusic": 1,
|
||||
"LiveFacebook": 1,
|
||||
"PrimeVideo": 1
|
||||
}
|
||||
},
|
||||
"purity_breakdown": {
|
||||
"-1": {
|
||||
"dominant_label": "SoundCloud",
|
||||
"dominant_count": 27,
|
||||
"size": 113
|
||||
},
|
||||
"0": {
|
||||
"dominant_label": "Spotify",
|
||||
"dominant_count": 21,
|
||||
"size": 80
|
||||
},
|
||||
"1": {
|
||||
"dominant_label": "Web",
|
||||
"dominant_count": 61,
|
||||
"size": 259
|
||||
},
|
||||
"2": {
|
||||
"dominant_label": "SoundCloud",
|
||||
"dominant_count": 39,
|
||||
"size": 160
|
||||
},
|
||||
"3": {
|
||||
"dominant_label": "Spotify",
|
||||
"dominant_count": 19,
|
||||
"size": 82
|
||||
},
|
||||
"4": {
|
||||
"dominant_label": "Web",
|
||||
"dominant_count": 12,
|
||||
"size": 43
|
||||
},
|
||||
"5": {
|
||||
"dominant_label": "Web",
|
||||
"dominant_count": 24,
|
||||
"size": 97
|
||||
}
|
||||
}
|
||||
}
|
||||
+8338
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,6 @@
|
||||
def main():
|
||||
print("Hello from tls-analyzer!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env python
|
||||
"""Django's command-line utility for administrative tasks."""
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
"""Run administrative tasks."""
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
|
||||
try:
|
||||
from django.core.management import execute_from_command_line
|
||||
except ImportError as exc:
|
||||
raise ImportError(
|
||||
"Couldn't import Django. Are you sure it's installed and "
|
||||
"available on your PYTHONPATH environment variable? Did you "
|
||||
"forget to activate a virtual environment?"
|
||||
) from exc
|
||||
execute_from_command_line(sys.argv)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,31 @@
|
||||
[project]
|
||||
name = "tls-analyzer"
|
||||
version = "0.1.0"
|
||||
description = "Add your description here"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"django==4.2.30",
|
||||
"mcp==1.28.1",
|
||||
"polars==1.42.1",
|
||||
"scikit-learn==1.5.2",
|
||||
"numpy==1.26.4",
|
||||
"pyarrow==25.0.0",
|
||||
"pyyaml==6.0.3",
|
||||
"pydantic==2.13.4",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"ruff>=0.4.0",
|
||||
"mypy>=1.10.0",
|
||||
"pytest>=8.0.0",
|
||||
"pytest-mock>=3.12.0",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
filterwarnings = [
|
||||
"ignore::DeprecationWarning",
|
||||
]
|
||||
@@ -0,0 +1,12 @@
|
||||
# 天璇 (TianXuan) - Locked Runtime Dependencies
|
||||
# Generated from runtime Python 3.12.2
|
||||
# Use: runtime\python\python.exe -m pip install -r requirements.txt
|
||||
|
||||
django==4.2.30
|
||||
mcp==1.28.1
|
||||
polars==1.42.1
|
||||
scikit-learn==1.5.2
|
||||
numpy==1.26.4
|
||||
pyarrow==25.0.0
|
||||
pyyaml==6.0.3
|
||||
pydantic==2.13.4
|
||||
@@ -0,0 +1,8 @@
|
||||
@echo off
|
||||
set PYTHONUTF8=1
|
||||
cd /d "%~dp0"
|
||||
echo Starting TianXuan...
|
||||
start /B runtime\python\python.exe manage.py runserver 127.0.0.1:8000 --noreload
|
||||
timeout /t 3 >nul
|
||||
start http://127.0.0.1:8000/
|
||||
timeout /t 5 >nul
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze CSV columns and output a concise summary.
|
||||
|
||||
Usage:
|
||||
runtime/python/python.exe scripts/column_survey.py --csv "data/*.csv"
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import sys
|
||||
from collections import OrderedDict
|
||||
|
||||
import polars as pl
|
||||
|
||||
|
||||
def _analyze_columns(df: pl.DataFrame) -> list[OrderedDict]:
|
||||
"""Return per-column stats: name, dtype, unique_count, null_pct, samples."""
|
||||
rows: list[OrderedDict] = []
|
||||
for col in df.columns:
|
||||
series = df[col]
|
||||
total = len(series)
|
||||
null_cnt = series.null_count()
|
||||
null_pct = (null_cnt / total * 100) if total > 0 else 0.0
|
||||
unique_cnt = series.n_unique()
|
||||
|
||||
non_null = series.drop_nulls()
|
||||
samples = non_null[:5].to_list() if len(non_null) > 0 else []
|
||||
|
||||
rows.append(OrderedDict([
|
||||
("name", col),
|
||||
("dtype", series.dtype),
|
||||
("unique", unique_cnt),
|
||||
("null_pct", null_pct),
|
||||
("samples", samples),
|
||||
("total_non_null", len(non_null)),
|
||||
]))
|
||||
return rows
|
||||
|
||||
|
||||
def _fmt_samples(samples: list, total_non_null: int) -> str:
|
||||
items = ", ".join(repr(s) for s in samples)
|
||||
if 0 < len(samples) < total_non_null:
|
||||
items += ", ..."
|
||||
return f"[{items}]"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Survey CSV columns")
|
||||
parser.add_argument("--csv", required=True, help="Glob pattern for CSV files")
|
||||
args = parser.parse_args()
|
||||
|
||||
files = sorted(glob.glob(args.csv, recursive=True))
|
||||
if not files:
|
||||
print(f"No files match pattern: {args.csv}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Load each file independently, group by schema
|
||||
schema_groups: OrderedDict[str, list[pl.DataFrame]] = OrderedDict()
|
||||
for f in files:
|
||||
try:
|
||||
lf = pl.scan_csv(f).head(5000)
|
||||
df = lf.collect()
|
||||
if df.is_empty():
|
||||
print(f"Skip empty: {f}", file=sys.stderr)
|
||||
continue
|
||||
# Use schema string as grouping key
|
||||
key = str(sorted(df.schema.items()))
|
||||
schema_groups.setdefault(key, []).append(df)
|
||||
except Exception as e:
|
||||
print(f"Skip {f}: {e}", file=sys.stderr)
|
||||
|
||||
if not schema_groups:
|
||||
print("No readable CSV data", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Report each schema group
|
||||
for idx, (schema_key, dfs) in enumerate(schema_groups.items()):
|
||||
merged = pl.concat(dfs)
|
||||
stats = _analyze_columns(merged)
|
||||
|
||||
if len(schema_groups) > 1:
|
||||
group_label = f" (group {idx + 1}, {len(dfs)} file(s))"
|
||||
else:
|
||||
group_label = f" ({sum(len(v) for v in schema_groups.values())} file(s))"
|
||||
|
||||
print(f"=== Column Survey{group_label} ===")
|
||||
for s in stats:
|
||||
print(
|
||||
f"{s['name']}: {s['dtype']}, "
|
||||
f"unique={s['unique']}, "
|
||||
f"null={s['null_pct']:.1f}%, "
|
||||
f"sample={_fmt_samples(s['samples'], s['total_non_null'])}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Debug database persistence"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
|
||||
import django
|
||||
django.setup()
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
import asyncio
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features, _save_features_to_db
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
|
||||
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
|
||||
|
||||
entity_col = detect_entity_column('ds')['recommended']
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
|
||||
metadata={'row_count': len(df_agg), 'csv_glob': csv_path})
|
||||
|
||||
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity',
|
||||
cluster_columns=[c for c in features if not c.startswith('_')][:8],
|
||||
algorithm='hdbscan', params={}, random_state=42))
|
||||
cid = clust_result['cluster_result_id']
|
||||
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id='entity', cluster_result_id=cid,
|
||||
top_k=10, method='zscore', save_to_db=True
|
||||
))
|
||||
|
||||
feats = feat_result.get('features', [])
|
||||
print(f'n_features={feat_result.get("n_features")}')
|
||||
labels = set(f.get('cluster_label') for f in feats)
|
||||
print(f'unique_labels={labels}')
|
||||
if feats:
|
||||
print(f'sample feature: {feats[0]}')
|
||||
print(f'sample keys: {list(feats[0].keys())}')
|
||||
|
||||
print(f'db_saved={feat_result.get("db_saved")}')
|
||||
|
||||
from analysis.models import ClusterResult, ClusterFeature, EntityProfile
|
||||
print(f'ClusterResult: {ClusterResult.objects.count()}')
|
||||
for cr in ClusterResult.objects.all():
|
||||
print(f' label={cr.cluster_label} size={cr.size} feats={cr.features.count()}')
|
||||
|
||||
print(f'ClusterFeature: {ClusterFeature.objects.count()}')
|
||||
print(f'EntityProfile: {EntityProfile.objects.count()}')
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Debug: test _save_features_to_db step by step"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
import asyncio
|
||||
from analysis.tool_registry import _handle_run_clustering
|
||||
from analysis import tool_registry
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
print('1. Store cleared')
|
||||
|
||||
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
|
||||
entity_col = detect_entity_column('ds')['recommended']
|
||||
agg_lf, features_list = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect()
|
||||
entity_schema = dict(zip(features_list, ['']*len(features_list)))
|
||||
store.store_dataset('entity_data', agg_lf, schema=entity_schema, metadata={'row_count': len(df_agg), 'csv_glob': csv_path})
|
||||
print('2. Dataset stored')
|
||||
|
||||
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
|
||||
cluster_columns=[c for c in features_list if not c.startswith('_')][:8],
|
||||
algorithm='hdbscan', params={}, random_state=42))
|
||||
cid = clust_result['cluster_result_id']
|
||||
print(f'3. Clustering done: {cid}, n_clusters={clust_result.get("n_clusters")}')
|
||||
|
||||
# Manual check
|
||||
ce = store.get_cluster_result(cid)
|
||||
if ce:
|
||||
pid = ce.get('parent_dataset_id', '')
|
||||
print(f'4. Cluster entry: parent_dataset_id={pid}')
|
||||
de = store.get_dataset(pid)
|
||||
if de:
|
||||
cg = de.get('metadata', {}).get('csv_glob', 'MISSING')
|
||||
print(f'5. Dataset entry exists, csv_glob={cg}')
|
||||
else:
|
||||
print('5. FAIL: dataset entry not found!')
|
||||
# List all datasets
|
||||
print(f' Available datasets: {store.list_datasets()}')
|
||||
else:
|
||||
print('4. FAIL: cluster entry not found!')
|
||||
|
||||
# Direct test
|
||||
print('6. Test features...')
|
||||
from analysis.models import AnalysisRun
|
||||
runs_before = AnalysisRun.objects.count()
|
||||
print(f' Runs before: {runs_before}')
|
||||
result = tool_registry._save_features_to_db(cid, [{
|
||||
'feature_name': 'test', 'cluster_label': 0,
|
||||
'mean': 1.0, 'std': 0.5, 'global_mean': 0.5, 'global_std': 0.3,
|
||||
'distinguishing_score': 2.0, 'distinguishing_method': 'zscore'
|
||||
}])
|
||||
print(f' Result: {result}')
|
||||
print(f' Runs after: {AnalysisRun.objects.count()}')
|
||||
|
||||
from analysis.models import ClusterResult, ClusterFeature
|
||||
print(f' ClusterResult: {ClusterResult.objects.count()}')
|
||||
print(f' ClusterFeature: {ClusterFeature.objects.count()}')
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Debug: test _save_features_to_db with existing store data"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.tool_registry import _save_features_to_db
|
||||
|
||||
store = SessionStore()
|
||||
print(f'Store entries: {len(store._stores)}')
|
||||
for k, v in list(store._stores.items())[:5]:
|
||||
entry_type = v.get('type')
|
||||
if entry_type == 'cluster_result':
|
||||
cid = k
|
||||
print(f' cluster_result: {cid}, n_clusters={v.get("n_clusters")}')
|
||||
print(f' parent_dataset_id={v.get("parent_dataset_id")}')
|
||||
|
||||
# Try saving
|
||||
result = _save_features_to_db(cid, [{
|
||||
'feature_name': 'test_feature', 'cluster_label': 0,
|
||||
'mean': 1.0, 'std': 0.5, 'global_mean': 0.5, 'global_std': 0.3,
|
||||
'distinguishing_score': 2.0, 'distinguishing_method': 'zscore'
|
||||
}])
|
||||
print(f' _save_features_to_db returned: {result}')
|
||||
@@ -0,0 +1,68 @@
|
||||
"""
|
||||
Debug: test pipeline data persistence
|
||||
"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
|
||||
import django
|
||||
django.setup()
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
import asyncio
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
|
||||
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'test_flows.csv')
|
||||
|
||||
# Step 1: Load
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
|
||||
print(f'1. Loaded {rc} rows')
|
||||
|
||||
# Step 2: Detect entity
|
||||
result = detect_entity_column('ds')
|
||||
entity_col = result['recommended']
|
||||
print(f'2. Entity col: {entity_col}')
|
||||
|
||||
# Step 3: Aggregate
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))), metadata={'row_count': len(df_agg)})
|
||||
print(f'3. Aggregated {len(df_agg)} entities')
|
||||
|
||||
# Step 4: Cluster
|
||||
feature_cols = [c for c in features if not c.startswith('_')][:8]
|
||||
clust_result = asyncio.run(_handle_run_clustering(
|
||||
dataset_id='entity', cluster_columns=feature_cols,
|
||||
algorithm='hdbscan', params={}, random_state=42
|
||||
))
|
||||
print(f'4. Clustering: n_clusters={clust_result.get("n_clusters")}')
|
||||
if 'error' in clust_result:
|
||||
print(f' ERROR: {clust_result["error"]}')
|
||||
sys.exit(1)
|
||||
|
||||
cid = clust_result.get('cluster_result_id', '')
|
||||
print(f' cluster_result_id={cid}')
|
||||
entry = store.get_cluster_result(cid)
|
||||
print(f' store entry exists={entry is not None}')
|
||||
if entry:
|
||||
print(f' n_clusters in store={entry.get("n_clusters")}')
|
||||
|
||||
# Step 5: Extract features + save to DB
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id='entity', cluster_result_id=cid,
|
||||
top_k=10, method='zscore', save_to_db=True
|
||||
))
|
||||
print(f'5. Features: n={feat_result.get("n_features")} db_saved={feat_result.get("db_saved")}')
|
||||
if 'error' in feat_result:
|
||||
print(f' ERROR: {feat_result["error"]}')
|
||||
|
||||
# Step 6: Check DB
|
||||
from analysis.models import AnalysisRun, ClusterResult, ClusterFeature, EntityProfile
|
||||
print(f'6. DB: ClusterResult={ClusterResult.objects.count()} ClusterFeature={ClusterFeature.objects.count()} EntityProfile={EntityProfile.objects.count()}')
|
||||
@@ -0,0 +1,56 @@
|
||||
"""
|
||||
对比两台机器的日志文件，标记第一个差异点。
|
||||
用法: runtime\python\python.exe scripts\diagnose_compare.py log_a.txt log_b.txt
|
||||
"""
|
||||
import sys, re
|
||||
|
||||
# Pattern for log timestamps: [2026-07-14 20:00:00,123]
|
||||
_TS_PATTERN = re.compile(r'^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]')
|
||||
|
||||
def strip_timestamp(line):
|
||||
"""Remove leading timestamp from a log line for comparison."""
|
||||
return _TS_PATTERN.sub('', line, count=1).strip()
|
||||
|
||||
def extract_key(line):
|
||||
"""Extract [TAG] key=value from a log line for comparison."""
|
||||
m = re.match(r'\[(\w+)\]', line)
|
||||
if m:
|
||||
return m.group(0) # return the [TAG]
|
||||
return line[:50]
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print('用法: diagnose_compare.py <机器A日志> <机器B日志>')
|
||||
sys.exit(1)
|
||||
|
||||
with open(sys.argv[1], 'r', encoding='utf-8') as f:
|
||||
lines_a = f.readlines()
|
||||
with open(sys.argv[2], 'r', encoding='utf-8') as f:
|
||||
lines_b = f.readlines()
|
||||
|
||||
max_lines = min(len(lines_a), len(lines_b))
|
||||
first_diff = None
|
||||
|
||||
for i in range(max_lines):
|
||||
a_stripped = strip_timestamp(lines_a[i])
|
||||
b_stripped = strip_timestamp(lines_b[i])
|
||||
|
||||
if a_stripped != b_stripped:
|
||||
if first_diff is None:
|
||||
first_diff = i
|
||||
|
||||
if first_diff is None:
|
||||
if len(lines_a) != len(lines_b):
|
||||
print(f'[MATCH] 前 {max_lines} 行一致, 但行数不同 (A={len(lines_a)}, B={len(lines_b)})')
|
||||
else:
|
||||
print('[MATCH] 两份日志完全一致')
|
||||
else:
|
||||
start = max(0, first_diff - 3)
|
||||
print(f'[DIFF] 首个差异在行 {first_diff + 1}:')
|
||||
print('--- 机器A ---')
|
||||
print(''.join(lines_a[start:first_diff + 3]))
|
||||
print('--- 机器B ---')
|
||||
print(''.join(lines_b[start:first_diff + 3]))
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,433 @@
|
||||
"""Evaluate clustering quality by comparing against ground-truth labels.
|
||||
|
||||
Usage:
|
||||
set DJANGO_ALLOW_ASYNC_UNSAFE=true
|
||||
runtime\python\python.exe scripts\eval_clustering.py
|
||||
"""
|
||||
import os, sys, json, asyncio, traceback
|
||||
os.environ['DJANGO_ALLOW_ASYNC_UNSAFE'] = 'true'
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tianxuan.settings')
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
import django; django.setup()
|
||||
|
||||
import numpy as np
|
||||
import polars as pl
|
||||
from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
from analysis.models import AnalysisRun, ClusterResult, EntityProfile, ClusterFeature
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
|
||||
|
||||
CSV_PATH = 'data/wnl_converted.csv'
|
||||
|
||||
def main():
|
||||
print("=" * 70)
|
||||
print("CLUSTERING QUALITY EVALUATION — WNL Converted Dataset")
|
||||
print("=" * 70)
|
||||
|
||||
# ── Step 1: Load CSV ────────────────────────────────────────────────
|
||||
print("\n[1] Loading CSV...")
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(CSV_PATH)
|
||||
print(f" Rows: {row_count}, Files: {file_count}, Memory: {memory_mb:.1f} MB")
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
ds_id = 'eval_ds'
|
||||
store.store_dataset(ds_id, lf, schema=schema, metadata={
|
||||
'row_count': row_count, 'file_count': file_count, 'csv_glob': CSV_PATH,
|
||||
})
|
||||
|
||||
# ── Step 2: Detect entity column ────────────────────────────────────
|
||||
print("\n[2] Detecting entity column...")
|
||||
result = detect_entity_column(ds_id)
|
||||
used_col = result.get('recommended')
|
||||
candidates = result.get('candidates', [])
|
||||
print(f" Detected: {used_col} (from {len(candidates)} candidates)")
|
||||
|
||||
# ── Step 3: Aggregate by entity ──────────────────────────────────────
|
||||
print("\n[3] Aggregating by entity...")
|
||||
agg_lf, feature_columns = aggregate_by_entity(lf, used_col, schema)
|
||||
df_agg = agg_lf.collect(streaming=True)
|
||||
entity_ds_id = 'eval_entity'
|
||||
store.store_dataset(entity_ds_id, agg_lf, schema=dict(zip(feature_columns, ['']*len(feature_columns))),
|
||||
metadata={'row_count': len(df_agg), 'entity_column': used_col, 'csv_glob': CSV_PATH})
|
||||
print(f" Entities: {len(df_agg)}, Features: {len(feature_columns)}")
|
||||
|
||||
# ── Step 4: Clustering ──────────────────────────────────────────────
|
||||
print("\n[4] Running clustering (HDBSCAN)...")
|
||||
user_features = [
|
||||
c for c in feature_columns
|
||||
if not c.startswith('_') and c not in ('first_seen', 'last_seen', 'src_ip')
|
||||
][:10]
|
||||
print(f" Features: {user_features}")
|
||||
|
||||
clust_result = asyncio.run(_handle_run_clustering(
|
||||
dataset_id=entity_ds_id,
|
||||
cluster_columns=user_features,
|
||||
algorithm='hdbscan', params={}, random_state=42,
|
||||
))
|
||||
|
||||
if 'error' in clust_result:
|
||||
print(f" ERROR: {clust_result['error']}")
|
||||
return
|
||||
|
||||
cluster_id = clust_result.get('cluster_result_id', '')
|
||||
n_clusters = clust_result.get('n_clusters', 0)
|
||||
quality = clust_result.get('quality_metrics', {})
|
||||
print(f" Clusters: {n_clusters}")
|
||||
print(f" Noise ratio: {quality.get('noise_ratio', 'N/A')}")
|
||||
print(f" Silhouette: {quality.get('silhouette_score', 'N/A')}")
|
||||
print(f" Davies-Bouldin: {quality.get('davies_bouldin_score', 'N/A')}")
|
||||
print(f" CH Score: {quality.get('calinski_harabasz_score', 'N/A')}")
|
||||
print(f" Cluster sizes: {quality.get('cluster_sizes', {})}")
|
||||
|
||||
# ── Step 5: Feature extraction ──────────────────────────────────────
|
||||
print("\n[5] Extracting distinguishing features...")
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id=entity_ds_id, cluster_result_id=cluster_id,
|
||||
top_k=10, method='zscore', save_to_db=True,
|
||||
))
|
||||
if 'error' in feat_result:
|
||||
print(f" Feature extraction error: {feat_result['error']}")
|
||||
else:
|
||||
print(f" Features extracted: {feat_result.get('n_features', 0)}")
|
||||
|
||||
# ── Get cluster labels from SessionStore ───────────────────────────
|
||||
print("\n[6] Reading cluster labels from session store...")
|
||||
cluster_entry = store.get_cluster_result(cluster_id)
|
||||
if cluster_entry is None:
|
||||
print(" ERROR: Cluster result not found in SessionStore!")
|
||||
return
|
||||
|
||||
labels_array = np.array(cluster_entry['labels'])
|
||||
n_entities = len(labels_array)
|
||||
print(f" Labels array size: {n_entities}")
|
||||
|
||||
# ── Map entities to ground-truth labels ────────────────────────────
|
||||
print("\n[7] Mapping entities to ground-truth labels...")
|
||||
# The entity column is :ipd (dst IP)
|
||||
# The original CSV has labels per-flow. After aggregation by :ipd,
|
||||
# each entity may have multiple flows with different labels.
|
||||
# We need the majority label per entity.
|
||||
|
||||
# Read original data
|
||||
df_raw = pl.read_csv(CSV_PATH, schema_overrides={'label': pl.Utf8})
|
||||
|
||||
# Get entity values from aggregated data
|
||||
entity_values = df_agg[used_col].to_list()
|
||||
|
||||
# Build entity -> label mapping from raw data (majority label per entity)
|
||||
raw_entities = df_raw[used_col].to_list()
|
||||
raw_labels = df_raw['label'].to_list()
|
||||
|
||||
entity_label_counter = defaultdict(Counter)
|
||||
for ent, lbl in zip(raw_entities, raw_labels):
|
||||
if lbl and str(lbl).strip():
|
||||
entity_label_counter[ent][str(lbl).strip()] += 1
|
||||
|
||||
# Get majority label for each entity in the aggregated set
|
||||
true_labels = []
|
||||
for ev in entity_values:
|
||||
if ev in entity_label_counter:
|
||||
true_labels.append(entity_label_counter[ev].most_common(1)[0][0])
|
||||
else:
|
||||
true_labels.append('Unknown')
|
||||
|
||||
true_labels = np.array(true_labels)
|
||||
print(f" Mapped {len(true_labels)} entities to ground-truth labels")
|
||||
|
||||
# Label distribution (ground truth among entities)
|
||||
print("\n Ground truth label distribution among entities:")
|
||||
gt_counts = Counter(true_labels)
|
||||
for lbl, cnt in gt_counts.most_common():
|
||||
print(f" {lbl:>20s}: {cnt:4d} ({cnt/len(true_labels)*100:5.1f}%)")
|
||||
|
||||
# ── Purity Score ────────────────────────────────────────────────────
|
||||
print("\n[8] Computing purity score...")
|
||||
# For each cluster, find the most common true label
|
||||
total_correct = 0
|
||||
cluster_label_map = {}
|
||||
for label in sorted(set(labels_array)):
|
||||
mask = labels_array == label
|
||||
cluster_entities = true_labels[mask]
|
||||
if len(cluster_entities) == 0:
|
||||
continue
|
||||
most_common = Counter(cluster_entities).most_common(1)[0]
|
||||
cluster_label_map[int(label)] = {
|
||||
'dominant_label': most_common[0],
|
||||
'dominant_count': most_common[1],
|
||||
'size': len(cluster_entities),
|
||||
}
|
||||
total_correct += most_common[1]
|
||||
|
||||
purity = total_correct / len(labels_array)
|
||||
print(f" Purity: {purity:.4f} ({total_correct}/{len(labels_array)} correct)")
|
||||
|
||||
# ── Adjusted Rand Index ─────────────────────────────────────────────
|
||||
print("\n[9] Computing Adjusted Rand Index (ARI)...")
|
||||
# Need numeric labels for ARI - map truth labels to integers
|
||||
unique_true = sorted(set(true_labels))
|
||||
true_to_int = {lbl: i for i, lbl in enumerate(unique_true)}
|
||||
true_numeric = np.array([true_to_int[lbl] for lbl in true_labels])
|
||||
|
||||
# Noise points (-1) cause issues for some metrics, filter them
|
||||
valid_mask = labels_array >= 0
|
||||
if valid_mask.sum() > 1:
|
||||
ari = adjusted_rand_score(true_numeric[valid_mask], labels_array[valid_mask])
|
||||
nmi = normalized_mutual_info_score(true_numeric[valid_mask], labels_array[valid_mask])
|
||||
print(f" ARI (excluding noise): {ari:.4f}")
|
||||
print(f" NMI (excluding noise): {nmi:.4f}")
|
||||
else:
|
||||
print(" (Not enough non-noise points to compute ARI/NMI)")
|
||||
ari = float('nan')
|
||||
|
||||
# With noise (-1) - some implementations treat -1 as just another label
|
||||
ari_with_noise = adjusted_rand_score(true_numeric, labels_array)
|
||||
print(f" ARI (with noise): {ari_with_noise:.4f}")
|
||||
|
||||
# ── Detailed per-cluster analysis ───────────────────────────────────
|
||||
print("\n" + "=" * 70)
|
||||
print("PER-CLUSTER PROFILES")
|
||||
print("=" * 70)
|
||||
|
||||
# Get feature stats (z-scores) from feature extraction
|
||||
features_list = feat_result.get('features', [])
|
||||
|
||||
# Also compute per-cluster feature means and z-scores from scratch
|
||||
numeric_cols = [c for c in df_agg.columns
|
||||
if df_agg[c].dtype in (pl.Float32, pl.Float64, pl.Int32, pl.Int64,
|
||||
pl.UInt32, pl.UInt64)
|
||||
and not c.startswith('_')]
|
||||
|
||||
# Global stats
|
||||
global_means = {}
|
||||
global_stds = {}
|
||||
for col in numeric_cols:
|
||||
s = df_agg[col].drop_nulls()
|
||||
if len(s) > 0:
|
||||
global_means[col] = float(s.mean())
|
||||
global_stds[col] = float(s.std()) if s.std() is not None and s.std() > 1e-10 else 1.0
|
||||
|
||||
# Build feature index for z-scores
|
||||
feature_by_cluster = defaultdict(dict)
|
||||
for f in features_list:
|
||||
feature_by_cluster[f['cluster_label']][f['feature_name']] = f
|
||||
|
||||
# Per cluster report
|
||||
# First build cross-tab: cluster x service
|
||||
cluster_service = defaultdict(Counter)
|
||||
for lbl, true_lbl in zip(labels_array, true_labels):
|
||||
cluster_service[int(lbl)][true_lbl] += 1
|
||||
|
||||
# Sort clusters by size
|
||||
cluster_sizes = Counter(labels_array)
|
||||
sorted_clusters = sorted(cluster_sizes.keys())
|
||||
|
||||
for label in sorted_clusters:
|
||||
size = cluster_sizes[label]
|
||||
pct = size / n_entities * 100
|
||||
is_noise = label == -1
|
||||
|
||||
print(f"\n Cluster {label}: {size} entities ({pct:.1f}%){' [NOISE]' if is_noise else ''}")
|
||||
|
||||
# Feature z-scores (top 5)
|
||||
cluster_feats = feature_by_cluster.get(label, {})
|
||||
sorted_feats = sorted(cluster_feats.items(),
|
||||
key=lambda x: abs(x[1].get('distinguishing_score', 0)), reverse=True)
|
||||
if sorted_feats:
|
||||
for feat_name, feat_data in sorted_feats[:5]:
|
||||
score = feat_data.get('distinguishing_score', 0)
|
||||
mean_val = feat_data.get('mean', 0)
|
||||
arrow = '+' if score >= 0 else ''
|
||||
print(f" {feat_name:35s} {mean_val:>10.4f} ({arrow}{score:.2f}σ)")
|
||||
else:
|
||||
# Compute z-scores from scratch
|
||||
label_mask = labels_array == label
|
||||
for col in numeric_cols[:8]:
|
||||
cluster_vals = df_agg[col].filter(pl.Series('_mask', label_mask)).drop_nulls()
|
||||
if len(cluster_vals) > 0 and col in global_means and col in global_stds:
|
||||
c_mean = float(cluster_vals.mean())
|
||||
z = (c_mean - global_means[col]) / max(global_stds[col], 1e-10)
|
||||
arrow = '+' if z >= 0 else ''
|
||||
print(f" {col:35s} {c_mean:>10.4f} ({arrow}{z:.2f}σ)")
|
||||
|
||||
# Dominant services
|
||||
print(f" {'':─<50s}")
|
||||
svc_counts = cluster_service.get(label, Counter())
|
||||
if svc_counts:
|
||||
total = sum(svc_counts.values())
|
||||
dominant_services = svc_counts.most_common(5)
|
||||
svc_str = ', '.join(f"{svc}({cnt/total*100:.0f}%)" for svc, cnt in dominant_services)
|
||||
print(f" Services: {svc_str}")
|
||||
else:
|
||||
print(f" Services: (none)")
|
||||
|
||||
# Classification
|
||||
print(f" {'':─<50s}")
|
||||
if is_noise:
|
||||
classification = "NOISE — unassigned by HDBSCAN"
|
||||
else:
|
||||
classification = classify_cluster(label, cluster_feats, svc_counts)
|
||||
print(f" → {classification}")
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────────
|
||||
print("\n" + "=" * 70)
|
||||
print("SUMMARY")
|
||||
print("=" * 70)
|
||||
print(f" Total entities: {n_entities}")
|
||||
print(f" Number of clusters (excl. noise): {n_clusters}")
|
||||
print(f" Noise count: {cluster_sizes.get(-1, 0)}")
|
||||
print(f" Noise ratio: {quality.get('noise_ratio', 'N/A')}")
|
||||
print(f" Silhouette score: {quality.get('silhouette_score', 'N/A')}")
|
||||
print(f" Purity: {purity:.4f}")
|
||||
print(f" ARI (excl. noise): {ari:.4f}" if not np.isnan(ari) else " ARI: N/A")
|
||||
print(f" NMI (excl. noise): {nmi:.4f}" if not np.isnan(ari) else " NMI: N/A")
|
||||
print()
|
||||
|
||||
# Confusion matrix: cluster x service
|
||||
print(" Confusion Matrix (Cluster × Service):")
|
||||
all_services = sorted(set(true_labels))
|
||||
header = f" {'Cluster':>8s}" + ''.join(f"{s:>14s}" for s in all_services[:8])
|
||||
if len(all_services) > 8:
|
||||
header += f" ... ({len(all_services)-8} more)"
|
||||
print(header)
|
||||
for label in sorted_clusters:
|
||||
row = cluster_service.get(label, Counter())
|
||||
row_str = f" {label:>8d}" + ''.join(f"{row.get(s, 0):>14d}" for s in all_services[:8])
|
||||
if len(all_services) > 8:
|
||||
extras = sum(row.get(s, 0) for s in all_services[8:])
|
||||
row_str += f" (+{extras})"
|
||||
print(row_str)
|
||||
|
||||
# ── Save report ─────────────────────────────────────────────────────
|
||||
report = {
|
||||
'run_info': {
|
||||
'dataset': CSV_PATH,
|
||||
'total_rows': row_count,
|
||||
'n_entities': n_entities,
|
||||
'n_clusters': n_clusters,
|
||||
},
|
||||
'quality_metrics': {
|
||||
'silhouette': quality.get('silhouette_score'),
|
||||
'davies_bouldin': quality.get('davies_bouldin_score'),
|
||||
'calinski_harabasz': quality.get('calinski_harabasz_score'),
|
||||
'noise_ratio': quality.get('noise_ratio'),
|
||||
'purity': round(purity, 4),
|
||||
'ari': round(ari, 4) if not np.isnan(ari) else None,
|
||||
'nmi': round(nmi, 4) if not np.isnan(ari) else None,
|
||||
},
|
||||
'cluster_sizes': quality.get('cluster_sizes', {}),
|
||||
'cluster_service_map': {
|
||||
str(k): dict(v.most_common()) for k, v in sorted(cluster_service.items())
|
||||
},
|
||||
'purity_breakdown': cluster_label_map,
|
||||
}
|
||||
|
||||
report_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
|
||||
'logs', 'clustering_eval_report.json')
|
||||
os.makedirs(os.path.dirname(report_path), exist_ok=True)
|
||||
with open(report_path, 'w') as f:
|
||||
json.dump(report, f, indent=2, default=str)
|
||||
print(f"\n Report saved to: {report_path}")
|
||||
|
||||
# ── Also save DB results properly ───────────────────────────────────
|
||||
print("\n Saving results to DB...")
|
||||
run = AnalysisRun.objects.create(
|
||||
csv_glob=CSV_PATH, status='completed',
|
||||
total_flows=row_count, entity_count=n_entities,
|
||||
cluster_count=n_clusters, entity_column=used_col,
|
||||
)
|
||||
|
||||
# Save ClusterResult records
|
||||
for label in sorted_clusters:
|
||||
size = cluster_sizes[label]
|
||||
proportion = size / n_entities if n_entities > 0 else 0
|
||||
ClusterResult.objects.create(
|
||||
run=run, cluster_label=int(label),
|
||||
size=size, proportion=proportion,
|
||||
noise_ratio=quality.get('noise_ratio'),
|
||||
silhouette_score=quality.get('silhouette_score'),
|
||||
)
|
||||
print(f" Saved {len(sorted_clusters)} ClusterResult records")
|
||||
|
||||
# Save EntityProfile records with correct cluster labels
|
||||
saved_ep = 0
|
||||
for i, ev in enumerate(entity_values):
|
||||
if i >= len(labels_array):
|
||||
break
|
||||
lbl = labels_array[i]
|
||||
cr = ClusterResult.objects.filter(run=run, cluster_label=int(lbl)).first()
|
||||
EntityProfile.objects.create(
|
||||
run=run, entity_value=str(ev),
|
||||
cluster_label=int(lbl), cluster=cr,
|
||||
feature_json={},
|
||||
)
|
||||
saved_ep += 1
|
||||
print(f" Saved {saved_ep} EntityProfile records")
|
||||
print(f" Run ID: #{run.id}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("EVALUATION COMPLETE")
|
||||
print("=" * 70)
|
||||
|
||||
|
||||
def classify_cluster(label, features, services):
|
||||
"""Classify a cluster as NORMAL, SUSPICIOUS, or MALICIOUS based on TLS features and services."""
|
||||
if not services:
|
||||
return "UNKNOWN"
|
||||
|
||||
top_services = services.most_common(3)
|
||||
service_names = [s[0] for s in top_services]
|
||||
|
||||
# Check for suspicious TLS signals in feature z-scores
|
||||
high_sni_missing = False
|
||||
low_tls_modern = False
|
||||
high_non_std_port = False
|
||||
low_recoverable = False
|
||||
|
||||
for feat_name, feat_data in features.items():
|
||||
score = feat_data.get('distinguishing_score', 0)
|
||||
if 'sni_missing' in feat_name and score > 1.5:
|
||||
high_sni_missing = True
|
||||
elif 'tls_modern' in feat_name and score < -1.5:
|
||||
low_tls_modern = True
|
||||
elif 'non_standard_port' in feat_name and score > 1.5:
|
||||
high_non_std_port = True
|
||||
elif 'recoverable' in feat_name and score < -1.0:
|
||||
low_recoverable = True
|
||||
|
||||
# Also look at services
|
||||
suspicious_services = {'Unknown'}
|
||||
|
||||
# If cluster has high concentration of one service, that's normal behavior
|
||||
dominant_pct = services.most_common(1)[0][1] / sum(services.values())
|
||||
|
||||
if dominant_pct > 0.7:
|
||||
return f"NORMAL — mostly {services.most_common(1)[0][0]} ({dominant_pct:.0%})"
|
||||
|
||||
# Check for suspicious feature patterns
|
||||
suspicious_signals = []
|
||||
if high_sni_missing:
|
||||
suspicious_signals.append('high sni_missing')
|
||||
if low_tls_modern:
|
||||
suspicious_signals.append('low tls_modern')
|
||||
if high_non_std_port:
|
||||
suspicious_signals.append('non_std_ports')
|
||||
if low_recoverable:
|
||||
suspicious_signals.append('low_recoverability')
|
||||
|
||||
if len(suspicious_signals) >= 2:
|
||||
return f"SUSPICIOUS — {', '.join(suspicious_signals)}; services: {', '.join(service_names)}"
|
||||
elif len(suspicious_signals) >= 1:
|
||||
return f"QUERY — {', '.join(suspicious_signals)}; dominant: {services.most_common(1)[0][0]}"
|
||||
else:
|
||||
return f"NORMAL — services: {', '.join(service_names)}"
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,86 @@
|
||||
"""生成复杂 TLS 合成测试数据（含经纬度、MAC地址、大量空值）"""
|
||||
import polars as pl
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
RNG = np.random.RandomState(42)
|
||||
|
||||
CLIENTS = [f'192.168.{RNG.randint(1,255)}.{RNG.randint(1,255)}' for _ in range(80)]
|
||||
SERVERS = (
|
||||
[f'10.0.{RNG.randint(1,255)}.{RNG.randint(1,255)}' for _ in range(80)]
|
||||
+ [f'203.0.113.{RNG.randint(1,255)}' for _ in range(30)]
|
||||
)
|
||||
DOMAINS = ['mail.example.com', 'api.example.org', 'cdn.example.net', 'login.example.org',
|
||||
'storage.example.net', 'video.example.com', 'auth.example.com', 'pay.example.org']
|
||||
TLS_VERSIONS = ['TLSv1.2', 'TLSv1.3', 'TLSv1.2', 'TLSv1.3', 'TLSv1.1', None]
|
||||
CIPHERS = ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384',
|
||||
'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', None]
|
||||
CERT_STATUS = ['valid', 'valid', 'valid', 'expired', 'self_signed', None]
|
||||
|
||||
def gen_mac():
|
||||
return ':'.join(f'{RNG.randint(0,255):02x}' for _ in range(6))
|
||||
|
||||
CITIES = [
|
||||
('Beijing', 39.9, 116.4), ('Shanghai', 31.2, 121.5), ('Tokyo', 35.7, 139.7),
|
||||
('Singapore', 1.35, 103.8), ('London', 51.5, -0.13), ('Frankfurt', 50.1, 8.68),
|
||||
('NewYork', 40.7, -74.0), ('SanFrancisco', 37.8, -122.4),
|
||||
('Sydney', -33.9, 151.2), ('Mumbai', 19.1, 72.9),
|
||||
]
|
||||
|
||||
def gen_flows(n_rows=5000):
|
||||
from datetime import datetime, timedelta
|
||||
base_ts = datetime(2025, 1, 1)
|
||||
rows = []
|
||||
for _ in range(n_rows):
|
||||
src_ip = RNG.choice(CLIENTS)
|
||||
dst_ip = RNG.choice(SERVERS)
|
||||
si = RNG.randint(0, len(CITIES))
|
||||
di = RNG.randint(0, len(CITIES))
|
||||
src_city, src_lat, src_lon = CITIES[si]
|
||||
dst_city, dst_lat, dst_lon = CITIES[di]
|
||||
same_city = src_city == dst_city
|
||||
|
||||
bs = int(max(0, RNG.lognormal(8, 2))) if RNG.random() > 0.05 else None
|
||||
br = int(max(0, bs * RNG.uniform(0.1, 2))) if bs and RNG.random() > 0.05 else None
|
||||
dur = round(RNG.exponential(20), 2) if RNG.random() > 0.08 else None
|
||||
hok = RNG.choice([True, True, True, False, None])
|
||||
trusted = hok and RNG.choice([True, True, False, None]) if hok else None
|
||||
tmo = int(RNG.exponential(500)) + 100 if hok else RNG.choice([None, RNG.randint(5000, 30000)])
|
||||
|
||||
rows.append({
|
||||
'timestamp': (base_ts + timedelta(seconds=RNG.randint(0, 31536000))).strftime('%Y-%m-%d %H:%M:%S'),
|
||||
'src_ip': src_ip, 'dst_ip': dst_ip,
|
||||
'src_mac': gen_mac(), 'dst_mac': gen_mac(),
|
||||
'src_url': f'https://{RNG.choice(DOMAINS)}{RNG.choice(["/","/login","/api/v1/users","/api/v1/data","/favicon.ico"])}',
|
||||
'dst_url': f'https://{RNG.choice(DOMAINS)}{RNG.choice(["/","/login","/api/v1/users","/api/v1/data"])}',
|
||||
'sni': RNG.choice(DOMAINS + [None, None]),
|
||||
'src_city': src_city, 'src_latitude': src_lat, 'src_longitude': src_lon,
|
||||
'dst_city': dst_city if not same_city else None,
|
||||
'dst_latitude': dst_lat if not same_city else None,
|
||||
'dst_longitude': dst_lon if not same_city else None,
|
||||
'bytes_sent': bs, 'bytes_rev': br, 'duration': dur,
|
||||
'tls_version': RNG.choice(TLS_VERSIONS),
|
||||
'cipher': RNG.choice(CIPHERS),
|
||||
'dst_port': RNG.choice([80, 443, 8080, 8443, 53, 993, 5223, None]),
|
||||
'cert_status': RNG.choice(CERT_STATUS),
|
||||
'handshake_ok': hok, 'trusted': trusted, 'timeout_ms': tmo,
|
||||
})
|
||||
df = pl.DataFrame(rows)
|
||||
nulls = df.null_count()
|
||||
for col in df.columns:
|
||||
n = int(nulls[col].item())
|
||||
if n > 0:
|
||||
print(f' {col}: {n}/{len(df)} 缺失 ({n*100//len(df)}%)')
|
||||
print(f'总行数: {len(df)}, 列数: {len(df.columns)}')
|
||||
return df
|
||||
|
||||
if __name__ == '__main__':
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--rows', type=int, default=5000)
|
||||
parser.add_argument('--output', type=str, default='data/complex_test.csv')
|
||||
args = parser.parse_args()
|
||||
df = gen_flows(args.rows)
|
||||
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
|
||||
df.write_csv(args.output, null_value='')
|
||||
print(f'已生成: {args.output} ({Path(args.output).stat().st_size/1024/1024:.1f} MB)')
|
||||
@@ -0,0 +1,872 @@
|
||||
"""Generate synthetic TLS flow test data with 3 real-world traffic profiles.
|
||||
|
||||
Usage:
|
||||
runtime\\python\\python.exe scripts\\gen_test_data.py [--rows N] [--output PATH]
|
||||
runtime\\python\\python.exe scripts\\gen_test_data.py --globe [--rows N] [--output PATH]
|
||||
"""
|
||||
import sys
|
||||
import polars as pl
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
# ===================================================================
|
||||
# TLS Constants
|
||||
# ===================================================================
|
||||
|
||||
CIPHER_HEX_MAP: dict[str, str] = {
|
||||
'TLS_AES_128_GCM_SHA256': '13 01',
|
||||
'TLS_AES_256_GCM_SHA384': '13 02',
|
||||
'TLS_ECDHE_ECDSA_AES128_GCM_SHA256': 'c0 2b',
|
||||
'TLS_ECDHE_RSA_AES128_GCM_SHA256': 'c0 2f',
|
||||
'TLS_ECDHE_RSA_AES256_GCM_SHA384': 'c0 30',
|
||||
'TLS_RSA_AES128_GCM_SHA256': '00 9c',
|
||||
'TLS_RSA_AES_128_CBC_SHA': '00 3c',
|
||||
'TLS_ECDHE_RSA_AES128_CBC_SHA': 'c0 09',
|
||||
'TLS_RSA_WITH_RC4_128_SHA': '00 05',
|
||||
'TLS_CHACHA20_POLY1305_SHA256': 'cc a9',
|
||||
}
|
||||
|
||||
CURVE_MAP: dict[str, str] = {
|
||||
'00 1d': 'x25519',
|
||||
'00 17': 'secp256r1',
|
||||
'00 18': 'secp384r1',
|
||||
'00 19': 'secp521r1',
|
||||
}
|
||||
|
||||
COUNTRY_CODES: list[str] = [
|
||||
'US', 'JP', 'DE', 'GB', 'SG', 'KR', 'FR', 'AU', 'IN', 'CA', 'BR',
|
||||
'RU', 'CN', 'NG', 'IR', 'NL', 'SE', 'HK', 'TW', 'ZA',
|
||||
]
|
||||
|
||||
COUNTRY_A: list[str] = ['US', 'JP', 'DE', 'GB', 'SG', 'KR', 'FR', 'AU', 'NL', 'CA', 'SE']
|
||||
COUNTRY_B: list[str] = ['US', 'NL', 'DE', 'SG', 'HK', 'JP', 'GB']
|
||||
COUNTRY_C: list[str] = ['RU', 'CN', 'BR', 'NG', 'IR', 'KR', 'US', 'HK', 'TW', 'ZA']
|
||||
|
||||
TABLE_NAMES: list[str] = [
|
||||
'tls_flows', 'network_traffic', 'encrypted_streams', 'ssl_logs', 'packet_capture',
|
||||
]
|
||||
LINK_NAMES: list[str] = [
|
||||
'internet-edge', 'mpls-backbone', 'vpn-tunnel', 'wan-link',
|
||||
'cloud-connect', 'dc-interconnect', 'peering-link',
|
||||
]
|
||||
NODE_NAMES: list[str] = (
|
||||
[f'gw-{i:02d}' for i in range(1, 11)]
|
||||
+ [f'core-{i:02d}' for i in range(1, 7)]
|
||||
)
|
||||
|
||||
ISP_NAMES: list[str] = [
|
||||
'China Telecom', 'China Unicom', 'China Mobile', 'AT&T', 'Verizon',
|
||||
'Comcast', 'Deutsche Telekom', 'BT Group', 'NTT', 'KDDI',
|
||||
'Singtel', 'Telstra', 'Reliance Jio', 'T-Mobile', 'Orange',
|
||||
'Rostelecom', 'Vivo', 'MTN Group', 'Etisalat',
|
||||
]
|
||||
ORG_NAMES: list[str] = [
|
||||
'Alibaba Inc.', 'Tencent Holdings', 'Baidu Inc.', 'Google LLC',
|
||||
'Amazon.com Inc.', 'Microsoft Corp.', 'Meta Platforms Inc.', 'Apple Inc.',
|
||||
'Netflix Inc.', 'Cloudflare Inc.', 'Akamai Technologies',
|
||||
'Yandex', 'Kaspersky Lab', 'Mail.ru Group',
|
||||
]
|
||||
CITY_NAMES: list[str] = [
|
||||
'Beijing', 'Shanghai', 'Tokyo', 'Seoul', 'New York', 'London',
|
||||
'Paris', 'Berlin', 'Singapore', 'Sydney', 'Mumbai', 'Dubai',
|
||||
'Toronto', 'Moscow', 'Sao Paulo', 'San Francisco', 'Los Angeles',
|
||||
'Lagos', 'Tehran', 'Hong Kong',
|
||||
]
|
||||
|
||||
# ===================================================================
|
||||
# Traffic Profile Definitions
|
||||
# ===================================================================
|
||||
|
||||
# Each profile defines distribution parameters for row generation.
|
||||
# We encode TLS version, cipher, curve as weighted-choice lists.
|
||||
|
||||
TLS_PROFILES: dict[str, list[tuple[str, float]]] = {
|
||||
'A': [('03 04', 0.80), ('03 03', 0.18), ('02 00', 0.01), ('03 00', 0.01)],
|
||||
'B': [('03 04', 0.50), ('03 03', 0.40), ('03 01', 0.05), ('03 02', 0.05)],
|
||||
'C': [('03 01', 0.30), ('03 02', 0.20), ('03 03', 0.20), ('03 04', 0.10),
|
||||
('02 00', 0.10), ('03 00', 0.10)],
|
||||
}
|
||||
|
||||
CIPHER_PROFILES: dict[str, list[tuple[str, float]]] = {
|
||||
'A': [('13 01', 0.50), ('c0 2b', 0.20), ('c0 2f', 0.15), ('13 02', 0.10), ('cc a9', 0.05)],
|
||||
'B': [('c0 2f', 0.30), ('13 01', 0.30), ('c0 30', 0.15), ('00 9c', 0.15), ('13 02', 0.10)],
|
||||
'C': [('00 3c', 0.25), ('c0 09', 0.20), ('00 05', 0.10), ('13 01', 0.15),
|
||||
('c0 2f', 0.10), ('c0 2b', 0.10), ('00 9c', 0.10)],
|
||||
}
|
||||
|
||||
CURVE_PROFILES: dict[str, list[tuple[str, float]]] = {
|
||||
'A': [('00 1d', 0.70), ('00 17', 0.25), ('00 18', 0.05)],
|
||||
'B': [('00 17', 0.50), ('00 1d', 0.40), ('00 18', 0.10)],
|
||||
'C': [('00 17', 0.25), ('00 1d', 0.20), ('00 18', 0.15),
|
||||
('00 19', 0.10), ('', 0.30)], # 30% no curve
|
||||
}
|
||||
|
||||
# Port profiles: (port, weight)
|
||||
PORT_PROFILES: dict[str, list[tuple[int, float]]] = {
|
||||
'A': [(443, 0.90), (80, 0.05), (8080, 0.02), (8443, 0.01), (993, 0.01), (465, 0.01)],
|
||||
'B': [(443, 0.60), (8443, 0.15), (8080, 0.10), (3128, 0.05), (1080, 0.05), (8888, 0.05)],
|
||||
'C': [(443, 0.60), (80, 0.05), (8080, 0.05), (8443, 0.05), (0, 0.25)], # 0 = random non-standard
|
||||
}
|
||||
|
||||
# Duration (seconds) profiles: used with exponential + clip
|
||||
DUR_PROFILES: dict[str, tuple[float, float, float]] = {
|
||||
'A': (2.0, 0.5, 8.0), # (mean exp, min, max)
|
||||
'B': (60.0, 10.0, 300.0),
|
||||
'C': (1.0, 0.3, 600.0), # bimodal: will mix short & long
|
||||
}
|
||||
|
||||
# ===================================================================
|
||||
# IP Pools (deterministic via numpy seed)
|
||||
# ===================================================================
|
||||
|
||||
def _ip_pool(n: int, seed_base: int, prefix: str | None = None) -> list[str]:
|
||||
"""Generate *n* reproducible IPs with optional *prefix* (e.g. '10.0.1')."""
|
||||
rng = np.random.RandomState(seed_base)
|
||||
ips: list[str] = []
|
||||
for _ in range(n):
|
||||
if prefix:
|
||||
parts = prefix.split('.')
|
||||
ip = (f'{parts[0]}.{parts[1]}.{parts[2]}.'
|
||||
f'{rng.randint(1, 255)}')
|
||||
else:
|
||||
a = rng.randint(1, 223)
|
||||
b = rng.randint(0, 256)
|
||||
c = rng.randint(0, 256)
|
||||
d = rng.randint(1, 255)
|
||||
ip = f'{a}.{b}.{c}.{d}'
|
||||
ips.append(ip)
|
||||
return ips
|
||||
|
||||
NORMAL_SRC_IPS: list[str] = _ip_pool(150, 42) # residential-like
|
||||
NORMAL_DST_IPS: list[str] = _ip_pool(120, 142) # CDN/cloud-like
|
||||
PROXY_SRC_IPS: list[str] = _ip_pool(8, 242) # proxy exit nodes
|
||||
PROXY_DST_IPS: list[str] = _ip_pool(15, 342) # limited targets
|
||||
MAL_SRC_IPS: list[str] = _ip_pool(180, 442) # botnet-like diverse
|
||||
MAL_DST_IPS: list[str] = _ip_pool(60, 542) # malicious targets
|
||||
|
||||
# ===================================================================
|
||||
# SNI / Domain pools
|
||||
# ===================================================================
|
||||
|
||||
LEGIT_DOMAINS: list[str] = [
|
||||
'www.google.com', 'api.github.com', 'facebook.com', 'youtube.com',
|
||||
'amazon.com', 'netflix.com', 'apple.com', 'microsoft.com',
|
||||
'cloudflare.com', 'stackoverflow.com', 'reddit.com', 'twitter.com',
|
||||
'linkedin.com', 'instagram.com', 'office.com', 'live.com',
|
||||
'bing.com', 'adobe.com', 'salesforce.com', 'zoom.us',
|
||||
'maps.google.com', 'mail.google.com', 'drive.google.com',
|
||||
'docs.google.com', 'play.google.com', 'support.microsoft.com',
|
||||
'portal.office.com', 'login.live.com', 'cdn.cloudflare.com',
|
||||
]
|
||||
|
||||
PROXY_DOMAINS: list[str] = [
|
||||
'proxy.example.com', 'vpn-gateway.example.net', 'tunnel.example.org',
|
||||
'exit-node.example.com', 'relay.example.net', 'gateway.example.org',
|
||||
'', '',
|
||||
]
|
||||
|
||||
DGA_DOMAINS: list[str] = [
|
||||
'gkj3hj2k3.sdf2k3j.net', '8sdf7h3k.xyz', 'j2k3h4k5h7.ru',
|
||||
'asdf9082k.biz', 'mnbv34k2.info', '2kj3h5k7jh9.com',
|
||||
'xcnv98sdf2.top', 'q3kj4h5k7.work', 'vbsd78sdf.xyz',
|
||||
'm2k3j5h6k8.ru', '9s8d7f6g5h.net', 'k2j3h5k7j9.org',
|
||||
'fasdf2342.biz', 'poiuy234k.info', 'zxcv7890m.top',
|
||||
'hk3j5k7h9.net', 'sdf2k3j4h5.ru', 'vb9s8d7f6.xyz',
|
||||
'n2k3j5h7k9.com', 'x98s7d6f5.info',
|
||||
]
|
||||
|
||||
FAKE_DOMAINS: list[str] = [
|
||||
'secure-login-app.com', 'update-manager.net', 'cdn-cache-service.com',
|
||||
'analytics-tracker.net', 'content-delivery-system.org',
|
||||
'download-center.net', 'account-verify.com', 'system-update.org',
|
||||
'cloud-sync-service.net', 'notification-hub.com',
|
||||
]
|
||||
|
||||
# ===================================================================
|
||||
# Helpers
|
||||
# ===================================================================
|
||||
|
||||
def _weighted_choice(options: list[tuple[str, float]]) -> str:
|
||||
"""Pick one from [(value, weight), ...] according to weight."""
|
||||
items, raw_w = zip(*options)
|
||||
total = sum(raw_w)
|
||||
return str(np.random.choice(items, p=[w / total for w in raw_w]))
|
||||
|
||||
|
||||
def _weighted_choice_int(options: list[tuple[int, float]]) -> int:
|
||||
"""Pick int from [(value, weight), ...]."""
|
||||
items, raw_w = zip(*options)
|
||||
total = sum(raw_w)
|
||||
return int(np.random.choice(items, p=[w / total for w in raw_w]))
|
||||
|
||||
|
||||
def _random_hex_pairs(n_pairs: int = 2) -> str:
|
||||
"""Generate space-separated hex pairs, e.g. 'c0 2b'."""
|
||||
pairs = [f'{np.random.randint(0, 256):02x}' for _ in range(n_pairs)]
|
||||
return ' '.join(pairs)
|
||||
|
||||
|
||||
def _random_32_bytes_hex() -> str:
|
||||
"""Generate 32 random hex bytes as a space-separated string."""
|
||||
pairs = [f'{np.random.randint(0, 256):02x}' for _ in range(32)]
|
||||
return ' '.join(pairs)
|
||||
|
||||
|
||||
def _low_entropy_hex() -> str:
|
||||
"""Generate repeating low-entropy TLS random (Profile C: beaconing).
|
||||
|
||||
Pattern: '00 01 00 01 00 01 ...' (repeating 2-byte cycle).
|
||||
"""
|
||||
pairs = ['00', '01'] * 16
|
||||
return ' '.join(pairs)
|
||||
|
||||
|
||||
def _maybe(rate: float) -> bool:
|
||||
"""Return True with probability *rate* (value should be blanked)."""
|
||||
return np.random.random() < rate
|
||||
|
||||
|
||||
def _assign_profile() -> str:
|
||||
"""Assign profile A (60%), B (20%), or C (20%)."""
|
||||
r = np.random.random()
|
||||
if r < 0.60:
|
||||
return 'A'
|
||||
elif r < 0.80:
|
||||
return 'B'
|
||||
else:
|
||||
return 'C'
|
||||
|
||||
|
||||
def _pick_sni(profile: str) -> tuple[str, str]:
|
||||
"""Generate (snam, cnam) for the given profile.
|
||||
|
||||
Returns (sni_name, common_name).
|
||||
"""
|
||||
if profile == 'A':
|
||||
if _maybe(0.05):
|
||||
return ('', '')
|
||||
domain = str(np.random.choice(LEGIT_DOMAINS))
|
||||
return (domain, domain) # always match
|
||||
|
||||
elif profile == 'B':
|
||||
if _maybe(0.20):
|
||||
return ('', '')
|
||||
domain = str(np.random.choice(PROXY_DOMAINS))
|
||||
if domain == '':
|
||||
return ('', '')
|
||||
return (domain, domain)
|
||||
|
||||
else: # C
|
||||
if _maybe(0.30):
|
||||
return ('', '')
|
||||
r = np.random.random()
|
||||
if r < 0.20:
|
||||
domain = str(np.random.choice(DGA_DOMAINS))
|
||||
elif r < 0.70:
|
||||
domain = str(np.random.choice(FAKE_DOMAINS))
|
||||
else:
|
||||
domain = str(np.random.choice(LEGIT_DOMAINS))
|
||||
|
||||
# 30% mismatch snam != cnam
|
||||
if _maybe(0.30):
|
||||
if _maybe(0.5):
|
||||
cnam = str(np.random.choice(FAKE_DOMAINS))
|
||||
else:
|
||||
cnam = str(np.random.choice(LEGIT_DOMAINS))
|
||||
return (domain, cnam)
|
||||
return (domain, domain)
|
||||
|
||||
|
||||
def _pick_country(profile: str) -> tuple[str, str]:
|
||||
"""Pick (scnt, dcnt) for the given profile."""
|
||||
if profile == 'A':
|
||||
scnt = str(np.random.choice(COUNTRY_A))
|
||||
dcnt = str(np.random.choice(COUNTRY_A))
|
||||
elif profile == 'B':
|
||||
scnt = str(np.random.choice(COUNTRY_B))
|
||||
dcnt = str(np.random.choice(COUNTRY_B))
|
||||
else:
|
||||
scnt = str(np.random.choice(COUNTRY_C))
|
||||
dcnt = str(np.random.choice(COUNTRY_C))
|
||||
return scnt, dcnt
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# Main generator — 3 profiles, 44 columns
|
||||
# ===================================================================
|
||||
|
||||
def generate_tls_flows(n_rows: int = 1000, seed: int = 42) -> pl.DataFrame:
|
||||
"""Generate synthetic TLS flow data with 3 real-world traffic profiles.
|
||||
|
||||
Profile A (60%): Normal Browsing
|
||||
Profile B (20%): Proxy/VPN Traffic
|
||||
Profile C (20%): Malicious/Suspicious
|
||||
|
||||
Returns a Polars DataFrame with all 44 columns matching the user's
|
||||
exact column naming convention.
|
||||
"""
|
||||
np.random.seed(seed)
|
||||
base_ts = datetime(2024, 1, 1, 0, 0, 0)
|
||||
day_seconds = 86400 * 90 # 90-day window
|
||||
|
||||
rows: list[dict] = []
|
||||
total = n_rows
|
||||
next_pct = 10
|
||||
|
||||
for i in range(total):
|
||||
# Progress indicator
|
||||
pct = (i + 1) * 100 // total
|
||||
if pct >= next_pct:
|
||||
sys.stdout.write('+')
|
||||
sys.stdout.flush()
|
||||
next_pct += 10
|
||||
|
||||
profile = _assign_profile()
|
||||
|
||||
# --- IPs ---
|
||||
if profile == 'A':
|
||||
src_ip = str(np.random.choice(NORMAL_SRC_IPS))
|
||||
dst_ip = str(np.random.choice(NORMAL_DST_IPS))
|
||||
elif profile == 'B':
|
||||
src_ip = str(np.random.choice(PROXY_SRC_IPS))
|
||||
dst_ip = str(np.random.choice(PROXY_DST_IPS))
|
||||
else:
|
||||
src_ip = str(np.random.choice(MAL_SRC_IPS))
|
||||
dst_ip = str(np.random.choice(MAL_DST_IPS))
|
||||
|
||||
# --- Ports ---
|
||||
src_port = np.random.randint(1024, 65536)
|
||||
dst_port = _weighted_choice_int(PORT_PROFILES[profile])
|
||||
if dst_port == 0: # random non-standard
|
||||
dst_port = int(np.random.choice([p for p in range(1025, 65536) if p != 443]))
|
||||
|
||||
# --- scnt, dcnt ---
|
||||
scnt, dcnt = _pick_country(profile)
|
||||
|
||||
# --- TLS version ---
|
||||
tls_ver = _weighted_choice(TLS_PROFILES[profile])
|
||||
|
||||
# --- SNI / CN ---
|
||||
snam, cnam = _pick_sni(profile)
|
||||
|
||||
# --- Duration ---
|
||||
mean_dur, min_dur, max_dur = DUR_PROFILES[profile]
|
||||
if profile == 'C':
|
||||
# Bimodal: beaconing (<1s) or C2 (long, >60s)
|
||||
if _maybe(0.50):
|
||||
dur = round(np.random.uniform(0.3, 1.0), 2)
|
||||
else:
|
||||
dur = round(min(max(np.random.exponential(120), 60), 600), 2)
|
||||
else:
|
||||
dur = round(min(max(np.random.exponential(mean_dur), min_dur), max_dur), 2)
|
||||
|
||||
# --- Session / timeout / key size ---
|
||||
ses = round(np.random.uniform(0, 1000), 4)
|
||||
tmo = round(np.random.uniform(0, 300), 4)
|
||||
ksz = int(np.random.choice([128, 256, 384, 521]))
|
||||
|
||||
# --- cnrs / isrs (blank="no", +="yes") ---
|
||||
if profile == 'A':
|
||||
cnrs_val = '+' if _maybe(0.30) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.10 else ''
|
||||
elif profile == 'B':
|
||||
cnrs_val = '+' if _maybe(0.60) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.40 else ''
|
||||
else:
|
||||
cnrs_val = '+' if _maybe(0.20) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.50 else ''
|
||||
|
||||
# --- Bytes / packets ---
|
||||
if profile == 'A':
|
||||
ack_val = int(max(1, np.random.exponential(3000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(200, 1500)))
|
||||
elif profile == 'B':
|
||||
ack_val = int(max(1, np.random.exponential(50000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
|
||||
else:
|
||||
if _maybe(0.50):
|
||||
# Beaconing: very small, regular packets
|
||||
ack_val = int(np.random.uniform(30, 100)) * np.random.randint(1, 20)
|
||||
ppk_val = max(1, ack_val // int(np.random.uniform(30, 100)))
|
||||
else:
|
||||
# Data exfil: large transfer
|
||||
ack_val = int(max(1, np.random.exponential(100000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
|
||||
|
||||
# --- Timestamps ---
|
||||
ts_offset = np.random.randint(0, day_seconds)
|
||||
ts_full = base_ts + timedelta(seconds=ts_offset)
|
||||
ts_str = ts_full.strftime('%Y-%m-%d %H:%M:%S')
|
||||
tm_str = ts_full.strftime('%H:%M:%S')
|
||||
dbd_str = (base_ts + timedelta(
|
||||
seconds=np.random.randint(0, day_seconds)
|
||||
)).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
# --- Cipher ---
|
||||
cipher_hex = _weighted_choice(CIPHER_PROFILES[profile])
|
||||
cipher_name = CIPHER_HEX_MAP.get(cipher_hex, '')
|
||||
|
||||
# --- Curve ---
|
||||
curve_hex = _weighted_choice(CURVE_PROFILES[profile])
|
||||
curve_name = CURVE_MAP.get(curve_hex, '')
|
||||
|
||||
# --- Other ---
|
||||
ipp_val = int(np.random.choice([6, 17], p=[0.85, 0.15]))
|
||||
dbn_val = np.random.randint(1, 10)
|
||||
tabl = str(np.random.choice(TABLE_NAMES))
|
||||
link_name = str(np.random.choice(LINK_NAMES))
|
||||
node = str(np.random.choice(NODE_NAMES))
|
||||
|
||||
# --- TLS random (low entropy for Profile C) ---
|
||||
if profile == 'C':
|
||||
rnd = _low_entropy_hex()
|
||||
else:
|
||||
rnd = _random_32_bytes_hex()
|
||||
rnt_val = int(base_ts.timestamp()) + np.random.randint(0, day_seconds)
|
||||
|
||||
# --- GeoIP dot-notation columns (non-globe: mostly blank) ---
|
||||
def _geo_val(val, blank_rate=0.70):
|
||||
return val if not _maybe(blank_rate) else ''
|
||||
|
||||
lat_src = '' if _maybe(0.70) else round(np.random.uniform(-90, 90), 6)
|
||||
lon_src = '' if _maybe(0.70) else round(np.random.uniform(-180, 180), 6)
|
||||
lat_dst = '' if _maybe(0.70) else round(np.random.uniform(-90, 90), 6)
|
||||
lon_dst = '' if _maybe(0.70) else round(np.random.uniform(-180, 180), 6)
|
||||
ispn_src = '' if _maybe(0.70) else str(np.random.choice(ISP_NAMES))
|
||||
ispn_dst = '' if _maybe(0.70) else str(np.random.choice(ISP_NAMES))
|
||||
orgn_src = '' if _maybe(0.70) else str(np.random.choice(ORG_NAMES))
|
||||
orgn_dst = '' if _maybe(0.70) else str(np.random.choice(ORG_NAMES))
|
||||
city_src = '' if _maybe(0.70) else str(np.random.choice(CITY_NAMES))
|
||||
city_dst = '' if _maybe(0.70) else str(np.random.choice(CITY_NAMES))
|
||||
|
||||
# --- Assemble row dict ---
|
||||
row = {
|
||||
':ips': src_ip,
|
||||
':ipd': dst_ip,
|
||||
':prs': '' if _maybe(0.30) else src_port,
|
||||
':prd': '' if _maybe(0.30) else dst_port,
|
||||
'scnt': scnt if not _maybe(0.30) else '',
|
||||
'dcnt': dcnt if not _maybe(0.30) else '',
|
||||
'server-ip': dst_ip,
|
||||
'client-ip': src_ip,
|
||||
'0ver': tls_ver if not _maybe(0.05) else '',
|
||||
'snam': snam,
|
||||
'cnam': cnam if not _maybe(0.05) else '',
|
||||
'4dur': '' if _maybe(0.30) else dur,
|
||||
'8ses': '' if _maybe(0.30) else ses,
|
||||
'2tmo': '' if _maybe(0.30) else tmo,
|
||||
'4ksz': '' if _maybe(0.30) else ksz,
|
||||
'cnrs': cnrs_val,
|
||||
'isrs': isrs_val,
|
||||
'8ack': ack_val if not _maybe(0.05) else '',
|
||||
'8ppk': ppk_val if not _maybe(0.05) else '',
|
||||
'8dbd': dbd_str if not _maybe(0.30) else '',
|
||||
'1ipp': ipp_val if not _maybe(0.30) else '',
|
||||
'4dbn': '' if _maybe(0.30) else dbn_val,
|
||||
'tabl': tabl if not _maybe(0.30) else '',
|
||||
'name': link_name if not _maybe(0.30) else '',
|
||||
'source-node': node if not _maybe(0.30) else '',
|
||||
'cipher-suite': cipher_name if not _maybe(0.30) else '',
|
||||
'ecdhe-named-curve': curve_name if not _maybe(0.30) else '',
|
||||
'0cph': cipher_hex if not _maybe(0.30) else '',
|
||||
'0crv': curve_hex if not _maybe(0.30) else '',
|
||||
'0rnd': rnd if not _maybe(0.30) else '',
|
||||
'0rnt': '' if _maybe(0.30) else rnt_val,
|
||||
'row': i + 1,
|
||||
'time': tm_str if not _maybe(0.30) else '',
|
||||
'timestamp': ts_str if not _maybe(0.10) else '',
|
||||
':ips.latd': lat_src,
|
||||
':ips.lond': lon_src,
|
||||
':ipd.latd': lat_dst,
|
||||
':ipd.lond': lon_dst,
|
||||
':ips.ispn': ispn_src,
|
||||
':ipd.ispn': ispn_dst,
|
||||
':ips.orgn': orgn_src,
|
||||
':ipd.orgn': orgn_dst,
|
||||
':ips.city': city_src,
|
||||
':ipd.city': city_dst,
|
||||
}
|
||||
rows.append(row)
|
||||
|
||||
sys.stdout.write('\n')
|
||||
sys.stdout.flush()
|
||||
return pl.DataFrame(rows)
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# GeoIP loader (shared by globe mode)
|
||||
# ===================================================================
|
||||
|
||||
def _load_geoip_ranges() -> dict[str, list[dict]]:
|
||||
"""Load GeoIP range data from ``data/geoip_data.txt``."""
|
||||
data_path = Path(__file__).resolve().parent.parent / 'data' / 'geoip_data.txt'
|
||||
cities: dict[str, list[dict]] = {}
|
||||
with open(data_path, 'r', encoding='utf-8') as f:
|
||||
for raw in f:
|
||||
stripped = raw.strip()
|
||||
if not stripped or stripped.startswith('#'):
|
||||
continue
|
||||
parts = [p.strip() for p in stripped.split(',')]
|
||||
if len(parts) != 6:
|
||||
continue
|
||||
start_str, end_str, lat_str, lon_str, city, _country = parts
|
||||
start_int = _ip_to_int(start_str)
|
||||
end_int = _ip_to_int(end_str)
|
||||
if start_int is None or end_int is None:
|
||||
continue
|
||||
try:
|
||||
lat = float(lat_str)
|
||||
lon = float(lon_str)
|
||||
except ValueError:
|
||||
continue
|
||||
city_key = city.replace(' ', '')
|
||||
if city_key not in cities:
|
||||
cities[city_key] = []
|
||||
cities[city_key].append({
|
||||
'start': start_int,
|
||||
'end': end_int,
|
||||
'lat': lat,
|
||||
'lon': lon,
|
||||
})
|
||||
return cities
|
||||
|
||||
|
||||
def _ip_to_int(ip_str: str) -> int | None:
|
||||
"""Convert dotted IPv4 string to 32-bit integer."""
|
||||
parts = ip_str.strip().split('.')
|
||||
if len(parts) != 4:
|
||||
return None
|
||||
try:
|
||||
octets = [int(p) for p in parts]
|
||||
if any(o < 0 or o > 255 for o in octets):
|
||||
return None
|
||||
return (octets[0] << 24) | (octets[1] << 16) | (octets[2] << 8) | octets[3]
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _int_to_ip(n: int) -> str:
|
||||
"""Convert 32-bit integer to dotted IPv4 string."""
|
||||
return f'{n >> 24 & 255}.{n >> 16 & 255}.{n >> 8 & 255}.{n & 255}'
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Globe mode city → (ISP, Org) mapping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_GLOBE_ORG_MAP: dict[str, tuple[str, str]] = {
|
||||
'Beijing': ('China Unicom', 'Tencent Holdings'),
|
||||
'Shanghai': ('China Telecom', 'Alibaba Inc.'),
|
||||
'Taipei': ('Chunghwa Telecom', 'Hon Hai Group'),
|
||||
'Tokyo': ('NTT', 'SoftBank Corp.'),
|
||||
'Seoul': ('KT Corp.', 'Samsung SDS'),
|
||||
'Singapore': ('Singtel', 'Grab Holdings'),
|
||||
'Bangkok': ('True Corp.', 'G-Able'),
|
||||
'Mumbai': ('Reliance Jio', 'Tata Consultancy'),
|
||||
'Sydney': ('Telstra', 'Atlassian'),
|
||||
'MountainView': ('Google Fiber', 'Google LLC'),
|
||||
'NewYork': ('Verizon', 'Amazon.com Inc.'),
|
||||
'LosAngeles': ('AT&T', 'Meta Platforms Inc.'),
|
||||
'Toronto': ('Rogers', 'Shopify Inc.'),
|
||||
'London': ('BT Group', 'British Telecom'),
|
||||
'Paris': ('Orange', 'TotalEnergies'),
|
||||
'Frankfurt': ('Deutsche Telekom', 'SAP SE'),
|
||||
'Amsterdam': ('KPN', 'ING Group'),
|
||||
'Dublin': ('Eir', 'Accenture'),
|
||||
'Stockholm': ('Telia', 'Ericsson'),
|
||||
'Moscow': ('Rostelecom', 'Yandex'),
|
||||
'Dubai': ('Etisalat', 'Emirates Group'),
|
||||
'SaoPaulo': ('Vivo', 'Itau Unibanco'),
|
||||
'HongKong': ('PCCW', 'HSBC Holdings'),
|
||||
'Lagos': ('MTN Group', 'Access Bank'),
|
||||
'Tehran': ('TCI', 'Iran Khodro'),
|
||||
}
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# Globe mode — generates data suitable for 3D globe visualisation
|
||||
# ===================================================================
|
||||
|
||||
def generate_globe_test_data(n_rows: int = 1000, seed: int = 42) -> pl.DataFrame:
|
||||
"""Generate test data for 3D globe visualisation using GeoIP-resolvable IPs.
|
||||
|
||||
Uses the same 3 traffic profiles as the normal mode, but with
|
||||
real GeoIP-resolvable source/destination IPs so lat/lon columns
|
||||
correspond to actual geographic locations.
|
||||
|
||||
Data corruption patterns (for testing globe view robustness):
|
||||
- ~30% of ``:ips.latd`` values are pure ``+``
|
||||
- ~10% of ``:ips.lond`` values are empty ``""``
|
||||
"""
|
||||
np.random.seed(seed)
|
||||
city_ranges = _load_geoip_ranges()
|
||||
|
||||
# Source cities (Asia / Pacific)
|
||||
src_cities = [
|
||||
'Beijing', 'Shanghai', 'Taipei', 'Tokyo',
|
||||
'Seoul', 'Singapore', 'Bangkok', 'Mumbai', 'Sydney',
|
||||
]
|
||||
# Destination cities (global rest)
|
||||
dst_cities = [
|
||||
'MountainView', 'NewYork', 'LosAngeles', 'Toronto',
|
||||
'London', 'Paris', 'Frankfurt', 'Amsterdam', 'Dublin',
|
||||
'Stockholm', 'Moscow', 'Dubai', 'SaoPaulo',
|
||||
]
|
||||
|
||||
all_cities = set(city_ranges.keys())
|
||||
src_cities = [c for c in src_cities if c in all_cities]
|
||||
dst_cities = [c for c in dst_cities if c in all_cities]
|
||||
if not src_cities or not dst_cities:
|
||||
raise RuntimeError('No usable GeoIP city ranges — cannot generate globe data.')
|
||||
|
||||
def _rand_ip_from_city(city: str) -> tuple[str, float, float]:
|
||||
rng_list = city_ranges[city]
|
||||
rng_entry = rng_list[np.random.randint(len(rng_list))]
|
||||
ip_int = rng_entry['start'] + np.random.randint(0, rng_entry['end'] - rng_entry['start'] + 1)
|
||||
return _int_to_ip(ip_int), rng_entry['lat'], rng_entry['lon']
|
||||
|
||||
base_ts = int(datetime(2023, 11, 1).timestamp())
|
||||
thirty_days_sec = 86400 * 60
|
||||
|
||||
rows: list[dict] = []
|
||||
total = n_rows
|
||||
next_pct = 10
|
||||
|
||||
for i in range(total):
|
||||
# Progress indicator
|
||||
pct = (i + 1) * 100 // total
|
||||
if pct >= next_pct:
|
||||
sys.stdout.write('+')
|
||||
sys.stdout.flush()
|
||||
next_pct += 10
|
||||
|
||||
profile = _assign_profile()
|
||||
|
||||
# GeoIP-resolvable IPs + lat/lon
|
||||
src_city = str(np.random.choice(src_cities))
|
||||
src_ip_raw, src_lat_raw, src_lon_raw = _rand_ip_from_city(src_city)
|
||||
|
||||
dst_city = str(np.random.choice(dst_cities))
|
||||
dst_ip_raw, dst_lat, dst_lon = _rand_ip_from_city(dst_city)
|
||||
|
||||
# Globe-specific corruption patterns
|
||||
src_lat_out: str | float = '+' if np.random.random() < 0.30 else round(src_lat_raw, 6)
|
||||
src_lon_out: str | float = '' if np.random.random() < 0.10 else round(src_lon_raw, 6)
|
||||
|
||||
# --- Ports ---
|
||||
src_port = np.random.randint(1024, 65536)
|
||||
dst_port = _weighted_choice_int(PORT_PROFILES[profile])
|
||||
if dst_port == 0:
|
||||
dst_port = int(np.random.choice([p for p in range(1025, 65536) if p != 443]))
|
||||
|
||||
# --- scnt, dcnt ---
|
||||
scnt, dcnt = _pick_country(profile)
|
||||
|
||||
# --- TLS version ---
|
||||
tls_ver = _weighted_choice(TLS_PROFILES[profile])
|
||||
|
||||
# --- SNI ---
|
||||
snam, cnam = _pick_sni(profile)
|
||||
|
||||
# --- Duration ---
|
||||
mean_dur, min_dur, max_dur = DUR_PROFILES[profile]
|
||||
if profile == 'C':
|
||||
if _maybe(0.50):
|
||||
dur = round(np.random.uniform(0.3, 1.0), 2)
|
||||
else:
|
||||
dur = round(min(max(np.random.exponential(120), 60), 600), 2)
|
||||
else:
|
||||
dur = round(min(max(np.random.exponential(mean_dur), min_dur), max_dur), 2)
|
||||
|
||||
# --- Session / timeout / key size ---
|
||||
ses = round(np.random.uniform(0, 1000), 4)
|
||||
tmo = round(np.random.uniform(0, 300), 4)
|
||||
ksz = int(np.random.choice([128, 256, 384, 521]))
|
||||
|
||||
# --- cnrs / isrs ---
|
||||
if profile == 'A':
|
||||
cnrs_val = '+' if _maybe(0.30) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.10 else ''
|
||||
elif profile == 'B':
|
||||
cnrs_val = '+' if _maybe(0.60) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.40 else ''
|
||||
else:
|
||||
cnrs_val = '+' if _maybe(0.20) else ''
|
||||
isrs_val = '+' if np.random.random() < 0.50 else ''
|
||||
|
||||
# --- Bytes / packets ---
|
||||
if profile == 'A':
|
||||
ack_val = int(max(1, np.random.exponential(3000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(200, 1500)))
|
||||
elif profile == 'B':
|
||||
ack_val = int(max(1, np.random.exponential(50000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
|
||||
else:
|
||||
if _maybe(0.50):
|
||||
ack_val = int(np.random.uniform(30, 100)) * np.random.randint(1, 20)
|
||||
ppk_val = max(1, ack_val // int(np.random.uniform(30, 100)))
|
||||
else:
|
||||
ack_val = int(max(1, np.random.exponential(100000)))
|
||||
ppk_val = max(1, int(ack_val / np.random.uniform(100, 1400)))
|
||||
|
||||
# --- Timestamps ---
|
||||
ts = base_ts + np.random.randint(0, thirty_days_sec)
|
||||
ts_str = datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
|
||||
tm_str = datetime.fromtimestamp(ts).strftime('%H:%M:%S')
|
||||
dbd_str = datetime.fromtimestamp(
|
||||
base_ts + np.random.randint(0, thirty_days_sec)
|
||||
).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
# --- Cipher ---
|
||||
cipher_hex = _weighted_choice(CIPHER_PROFILES[profile])
|
||||
cipher_name = CIPHER_HEX_MAP.get(cipher_hex, '')
|
||||
|
||||
# --- Curve ---
|
||||
curve_hex = _weighted_choice(CURVE_PROFILES[profile])
|
||||
curve_name = CURVE_MAP.get(curve_hex, '')
|
||||
|
||||
# --- Other ---
|
||||
ipp_val = int(np.random.choice([6, 17], p=[0.85, 0.15]))
|
||||
dbn_val = np.random.randint(1, 10)
|
||||
tabl = str(np.random.choice(TABLE_NAMES))
|
||||
link_name = str(np.random.choice(LINK_NAMES))
|
||||
node = str(np.random.choice(NODE_NAMES))
|
||||
|
||||
# --- TLS random ---
|
||||
if profile == 'C':
|
||||
rnd = _low_entropy_hex()
|
||||
else:
|
||||
rnd = _random_32_bytes_hex()
|
||||
rnt_val = base_ts + np.random.randint(0, thirty_days_sec)
|
||||
|
||||
# --- GeoIP ISP/org from globe map ---
|
||||
src_isp, src_org = _GLOBE_ORG_MAP.get(
|
||||
src_city, (str(np.random.choice(ISP_NAMES)), str(np.random.choice(ORG_NAMES))),
|
||||
)
|
||||
dst_isp, dst_org = _GLOBE_ORG_MAP.get(
|
||||
dst_city, (str(np.random.choice(ISP_NAMES)), str(np.random.choice(ORG_NAMES))),
|
||||
)
|
||||
|
||||
rows.append({
|
||||
':ips': src_ip_raw,
|
||||
':ipd': dst_ip_raw,
|
||||
':prs': '' if _maybe(0.30) else src_port,
|
||||
':prd': '' if _maybe(0.30) else dst_port,
|
||||
'scnt': scnt if not _maybe(0.30) else '',
|
||||
'dcnt': dcnt if not _maybe(0.30) else '',
|
||||
'server-ip': dst_ip_raw,
|
||||
'client-ip': src_ip_raw,
|
||||
'0ver': tls_ver if not _maybe(0.05) else '',
|
||||
'snam': snam,
|
||||
'cnam': cnam if not _maybe(0.05) else '',
|
||||
'4dur': '' if _maybe(0.30) else dur,
|
||||
'8ses': '' if _maybe(0.30) else ses,
|
||||
'2tmo': '' if _maybe(0.30) else tmo,
|
||||
'4ksz': '' if _maybe(0.30) else ksz,
|
||||
'cnrs': cnrs_val,
|
||||
'isrs': isrs_val,
|
||||
'8ack': ack_val if not _maybe(0.05) else '',
|
||||
'8ppk': ppk_val if not _maybe(0.05) else '',
|
||||
'8dbd': dbd_str if not _maybe(0.30) else '',
|
||||
'1ipp': ipp_val if not _maybe(0.30) else '',
|
||||
'4dbn': '' if _maybe(0.30) else dbn_val,
|
||||
'tabl': tabl if not _maybe(0.30) else '',
|
||||
'name': link_name if not _maybe(0.30) else '',
|
||||
'source-node': node if not _maybe(0.30) else '',
|
||||
'cipher-suite': cipher_name if not _maybe(0.30) else '',
|
||||
'ecdhe-named-curve': curve_name if not _maybe(0.30) else '',
|
||||
'0cph': cipher_hex if not _maybe(0.30) else '',
|
||||
'0crv': curve_hex if not _maybe(0.30) else '',
|
||||
'0rnd': rnd if not _maybe(0.30) else '',
|
||||
'0rnt': '' if _maybe(0.30) else rnt_val,
|
||||
'row': i + 1,
|
||||
'time': tm_str if not _maybe(0.30) else '',
|
||||
'timestamp': ts_str if not _maybe(0.10) else '',
|
||||
':ips.latd': src_lat_out,
|
||||
':ips.lond': src_lon_out,
|
||||
':ipd.latd': round(dst_lat, 6),
|
||||
':ipd.lond': round(dst_lon, 6),
|
||||
':ips.ispn': src_isp,
|
||||
':ipd.ispn': dst_isp,
|
||||
':ips.orgn': src_org,
|
||||
':ipd.orgn': dst_org,
|
||||
':ips.city': src_city,
|
||||
':ipd.city': dst_city,
|
||||
})
|
||||
|
||||
sys.stdout.write('\n')
|
||||
sys.stdout.flush()
|
||||
return pl.DataFrame(rows)
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# CSV save & CLI
|
||||
# ===================================================================
|
||||
|
||||
EXPECTED_COLUMNS: list[str] = [
|
||||
':ips', ':ipd', ':prs', ':prd', 'scnt', 'dcnt', 'server-ip', 'client-ip',
|
||||
'0ver', 'snam', 'cnam', '4dur', '8ses', '2tmo', '4ksz', 'cnrs', 'isrs',
|
||||
'8ack', '8ppk', '8dbd', '1ipp', '4dbn', 'tabl', 'name', 'source-node',
|
||||
'cipher-suite', 'ecdhe-named-curve', '0cph', '0crv', '0rnd', '0rnt',
|
||||
'row', 'time', 'timestamp',
|
||||
':ips.latd', ':ips.lond', ':ipd.latd', ':ipd.lond',
|
||||
':ips.ispn', ':ipd.ispn', ':ips.orgn', ':ipd.orgn', ':ips.city', ':ipd.city',
|
||||
]
|
||||
|
||||
|
||||
def save_csv(df: pl.DataFrame, path: str):
|
||||
"""Write DataFrame to CSV with full 44-column validation."""
|
||||
out_path = Path(path)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Validate column count
|
||||
actual_cols = df.columns
|
||||
if len(actual_cols) != 44:
|
||||
print(f'WARNING: Expected 44 columns, got {len(actual_cols)}')
|
||||
missing = set(EXPECTED_COLUMNS) - set(actual_cols)
|
||||
extra = set(actual_cols) - set(EXPECTED_COLUMNS)
|
||||
if missing:
|
||||
print(f' Missing: {sorted(missing)}')
|
||||
if extra:
|
||||
print(f' Extra: {sorted(extra)}')
|
||||
|
||||
df.write_csv(out_path)
|
||||
size_kb = out_path.stat().st_size / 1024
|
||||
print(f'Generated: {path} ({len(df)} rows, {size_kb:.0f} KB, {len(actual_cols)} cols)')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description='Generate synthetic TLS flow test data (3 traffic profiles)')
|
||||
parser.add_argument('--rows', type=int, default=1000, help='Number of rows (default: 1000)')
|
||||
parser.add_argument('--output', type=str, default=None, help='Output CSV path')
|
||||
parser.add_argument('--globe', action='store_true',
|
||||
help='Generate 3D globe visualisation test data (GeoIP-resolvable IPs)')
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.globe:
|
||||
output = args.output or 'data/globe_test.csv'
|
||||
print(f'Generating {args.rows} globe rows...')
|
||||
print('Progress: ', end='', flush=True)
|
||||
df = generate_globe_test_data(args.rows)
|
||||
save_csv(df, output)
|
||||
|
||||
# Globe-specific stats
|
||||
plus_cnt = df.filter(pl.col(':ips.latd') == '+').height
|
||||
blank_cnt = df.filter(pl.col(':ips.lond') == '').height
|
||||
print(f'\nGlobe data stats:')
|
||||
print(f' :ips.latd "+": {plus_cnt}/{len(df)} ({plus_cnt/len(df)*100:.1f}%)')
|
||||
print(f' :ips.lond "": {blank_cnt}/{len(df)} ({blank_cnt/len(df)*100:.1f}%)')
|
||||
else:
|
||||
output = args.output or 'data/test_flows.csv'
|
||||
print(f'Generating {args.rows} rows (A:60% Normal, B:20% Proxy, C:20% Malicious)...')
|
||||
print('Progress: ', end='', flush=True)
|
||||
df = generate_tls_flows(args.rows)
|
||||
save_csv(df, output)
|
||||
|
||||
# Profile distribution
|
||||
print(f'\nProfile distribution:')
|
||||
for prefix, label in [('A', 'A-Normal'), ('B', 'B-Proxy'), ('C', 'C-Malicious')]:
|
||||
# Profile isn't stored as a column, so estimate from IP patterns
|
||||
pass
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Test 32B-optimized LLM orchestrator"""
|
||||
import sys, os, json
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
|
||||
cfg = LLMConfig(base_url='https://api.deepseek.com',
|
||||
api_key='sk-360ef76d59674d6b8bc2eb160327dd39',
|
||||
model='deepseek-v4-flash')
|
||||
|
||||
result = run_llm_pipeline('ds', config=cfg)
|
||||
print(f'Steps: {result.get("steps")}')
|
||||
out = result.get("result", result.get("error", ""))
|
||||
print(f'Result: {str(out)[:200]}')
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Test all functionality under Chinese directory path"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
|
||||
print('1. Django import:', 'OK')
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.session_store import SessionStore
|
||||
print('2. All modules imported:', 'OK')
|
||||
|
||||
import asyncio
|
||||
from analysis.tool_registry import handle_call
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
|
||||
# Load from Chinese path
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', '复杂测试.csv')
|
||||
print(f'Loading: {csv}')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
print(f'3. Loaded {rc} rows from Chinese path CSV:', 'OK')
|
||||
|
||||
# Profile
|
||||
prof = asyncio.run(handle_call('profile_data', {'dataset_id': 'ds'}))
|
||||
print(f'4. Profile: {len(prof.get("column_stats",{}))} columns profiled:', 'OK' if 'error' not in prof else 'FAIL')
|
||||
|
||||
# Entity detection (direct call)
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
detect_result = detect_entity_column('ds')
|
||||
entity_col = detect_result.get('recommended', '')
|
||||
print(f'5. Entity detection: {entity_col}:', 'OK' if entity_col else 'FAIL')
|
||||
|
||||
# Aggregate
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect()
|
||||
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
|
||||
metadata={'row_count': len(df_agg), 'csv_glob': csv})
|
||||
print(f'6. Aggregation: {len(df_agg)} entities, {len(features)} features:', 'OK')
|
||||
|
||||
# Cluster
|
||||
clust = asyncio.run(handle_call('run_clustering', {
|
||||
'dataset_id': 'entity',
|
||||
'cluster_columns': [c for c in features if not c.startswith('_')][:8],
|
||||
'algorithm': 'hdbscan',
|
||||
'params': {'min_cluster_size': 5},
|
||||
'random_state': 42,
|
||||
}))
|
||||
print(f'7. Clustering: {clust.get("n_clusters")} clusters:', 'OK' if 'error' not in clust else f'FAIL: {clust.get("error")}')
|
||||
|
||||
# Extract features
|
||||
cid = clust.get('cluster_result_id', '')
|
||||
if cid and 'error' not in clust:
|
||||
feats = asyncio.run(handle_call('extract_features', {
|
||||
'dataset_id': 'entity',
|
||||
'cluster_result_id': cid,
|
||||
'top_k': 10, 'method': 'zscore', 'save_to_db': True,
|
||||
}))
|
||||
print(f'8. Feature extraction: {feats.get("n_features")} features db_saved={feats.get("db_saved")}:', 'OK' if feats.get('db_saved') else 'FAIL')
|
||||
|
||||
print('\n=== ALL TESTS PASSED UNDER CHINESE PATH ===')
|
||||
@@ -0,0 +1,128 @@
|
||||
"""天璇完整流程端到端测试"""
|
||||
import sys, os, importlib
|
||||
# 项目根目录 = 脚本目录的父目录
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
|
||||
OK = '[OK]'
|
||||
FAIL = '[!!]'
|
||||
|
||||
results = []
|
||||
|
||||
def check(name, ok, detail=""):
|
||||
icon = OK if ok else FAIL
|
||||
results.append((name, ok))
|
||||
print(f' {icon} {name} {detail}')
|
||||
|
||||
print('=' * 60)
|
||||
print('天璇 完整流程端到端测试')
|
||||
print('=' * 60)
|
||||
|
||||
# ── 1. 生成复杂测试数据 ──────────────────────────────────
|
||||
print('\n[1/7] 生成测试数据')
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location('gen_complex_test',
|
||||
os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'scripts', 'gen_complex_test.py'))
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
df = mod.gen_flows(n_rows=2000)
|
||||
csv_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'e2e_full.csv')
|
||||
df.write_csv(csv_path, null_value='')
|
||||
check('gen_complex_test', os.path.exists(csv_path), f'rows={len(df)} cols={len(df.columns)}')
|
||||
|
||||
# ── 2. 加载 + 清洗 ────────────────────────────────────────
|
||||
print('\n[2/7] 加载 CSV + 自动清洗')
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.session_store import SessionStore
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv_path)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv_path})
|
||||
|
||||
# 检查旧经纬度列是否被丢弃
|
||||
dropped_lat = not any('latitude' in c.lower() or 'longitude' in c.lower() for c in schema)
|
||||
check('丢弃旧经纬度列', dropped_lat, f'columns={len(schema)}')
|
||||
|
||||
# 检查 HEX 列是否被预处理
|
||||
schema_names = list(schema.keys())
|
||||
check('schema 有效', len(schema_names) > 10, f'{len(schema_names)} columns')
|
||||
|
||||
# ── 3. 数据校验 ────────────────────────────────────────────
|
||||
print('\n[3/7] 数据校验')
|
||||
from analysis.data_validator import validate
|
||||
v_result = validate('ds')
|
||||
n_risks = len(v_result.get('risks', []))
|
||||
check('数据校验完成', v_result.get('valid') is not None, f'valid={v_result["valid"]} risks={n_risks}')
|
||||
|
||||
# ── 4. 实体检测 ────────────────────────────────────────────
|
||||
print('\n[4/7] 实体检测')
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
detect_result = detect_entity_column('ds')
|
||||
entity_col = detect_result.get('recommended', '')
|
||||
n_candidates = len(detect_result.get('candidates', []))
|
||||
check('实体检测', bool(entity_col), f'entity={entity_col} candidates={n_candidates}')
|
||||
|
||||
# ── 5. 类型分类 + 聚合 ─────────────────────────────────────
|
||||
print('\n[5/7] 类型感知聚合')
|
||||
from analysis.type_classifier import classify_schema, DataType
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
|
||||
# 类型分类
|
||||
type_map = classify_schema(lf.head(500))
|
||||
types_found = set(t.name for t in type_map.values())
|
||||
check('类型分类', len(type_map) > 0, f'types={types_found}')
|
||||
|
||||
# 聚合
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect()
|
||||
entity_count = len(df_agg)
|
||||
store.store_dataset('entity', agg_lf, schema=dict(zip(features, ['']*len(features))),
|
||||
metadata={'row_count': entity_count, 'entity_column': entity_col})
|
||||
check('实体聚合', entity_count > 0, f'{entity_count} entities, {len(features)} features')
|
||||
|
||||
# 检查 GeoIP 和类型特征
|
||||
check('有特征列', len(features) > 5, f'features={features[:8]}')
|
||||
|
||||
# ── 6. 聚类 + 特征提取 ────────────────────────────────────
|
||||
print('\n[6/7] 聚类 + 特征提取')
|
||||
import asyncio
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
|
||||
|
||||
feature_cols = [c for c in features if not c.startswith('_')][:10]
|
||||
clust_result = asyncio.run(_handle_run_clustering(
|
||||
dataset_id='entity', cluster_columns=feature_cols,
|
||||
algorithm='hdbscan', params={}, random_state=42
|
||||
))
|
||||
n_clusters = clust_result.get('n_clusters', 0)
|
||||
check('HDBSCAN 聚类', n_clusters >= 0, f'n_clusters={n_clusters}')
|
||||
if 'error' not in clust_result:
|
||||
quality = clust_result.get('quality_metrics', {})
|
||||
check('聚类质量', quality.get('silhouette_score') is not None, f'silhouette={quality.get("silhouette_score")}')
|
||||
|
||||
cid = clust_result.get('cluster_result_id', '')
|
||||
if cid and 'error' not in clust_result:
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id='entity', cluster_result_id=cid,
|
||||
top_k=10, method='zscore', save_to_db=True
|
||||
))
|
||||
n_feats = feat_result.get('n_features', 0)
|
||||
check('特征提取 + DB持久化', feat_result.get('db_saved'), f'{n_feats} features')
|
||||
|
||||
# ── 7. 验证 ORM 数据 ──────────────────────────────────────
|
||||
print('\n[7/7] ORM 数据验证')
|
||||
from analysis.models import ClusterResult, ClusterFeature, EntityProfile
|
||||
n_clusters_db = ClusterResult.objects.count()
|
||||
n_features_db = ClusterFeature.objects.count()
|
||||
n_entities_db = EntityProfile.objects.count()
|
||||
check('ClusterResult 表', n_clusters_db > 0, f'{n_clusters_db} records')
|
||||
check('ClusterFeature 表', n_features_db > 0, f'{n_features_db} records')
|
||||
check('EntityProfile 表', n_entities_db >= 0, f'{n_entities_db} records')
|
||||
|
||||
# ── 结论 ──────────────────────────────────────────────────
|
||||
print('\n' + '=' * 60)
|
||||
passed = sum(1 for _, ok in results if ok)
|
||||
total = len(results)
|
||||
print(f'\n结果: {passed}/{total} 通过')
|
||||
print('=' * 60)
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Test LLM basic chat and orchestrator"""
|
||||
import sys, os, json, urllib.request
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
# Test 1: Basic chat
|
||||
print('=== Test 1: Basic chat ===')
|
||||
payload = json.dumps({
|
||||
'model': 'deepseek-chat',
|
||||
'messages': [
|
||||
{'role': 'system', 'content': 'You are a data analysis assistant.'},
|
||||
{'role': 'user', 'content': 'Profile dataset ds_001.'}
|
||||
],
|
||||
'max_tokens': 200,
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
|
||||
headers={
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39',
|
||||
},
|
||||
method='POST')
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
data = json.loads(resp.read())
|
||||
print('Basic chat OK')
|
||||
print('Response:', data['choices'][0]['message']['content'][:200])
|
||||
|
||||
# Test 2: LLM orchestrator
|
||||
print('\n=== Test 2: LLM orchestrator ===')
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
from tianxuan.llm_orchestrator import run_llm_pipeline, LLMConfig
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
import asyncio
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
|
||||
|
||||
store = SessionStore(); store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
|
||||
cfg = LLMConfig(
|
||||
base_url='https://api.deepseek.com',
|
||||
api_key='sk-360ef76d59674d6b8bc2eb160327dd39',
|
||||
model='deepseek-chat',
|
||||
)
|
||||
|
||||
result = run_llm_pipeline('ds', config=cfg)
|
||||
print('Orchestrator result:', json.dumps(result, ensure_ascii=False, indent=2)[:500])
|
||||
|
||||
# Test 3: Manual step-by-step (fallback if LLM fails)
|
||||
if 'error' in result:
|
||||
print('\n=== Test 3: Manual pipeline (fallback) ===')
|
||||
entity_col = detect_entity_column('ds')['recommended']
|
||||
print(f'Entity col: {entity_col}')
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect()
|
||||
print(f'Aggregated: {len(df_agg)} entities, {len(features)} features')
|
||||
store.store_dataset('entity_data', agg_lf, schema=dict(zip(features, ['']*len(features))),
|
||||
metadata={'row_count': len(df_agg), 'csv_glob': csv})
|
||||
clust_result = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
|
||||
cluster_columns=[c for c in features if not c.startswith('_')][:8],
|
||||
algorithm='hdbscan', params={}, random_state=42))
|
||||
print(f'Clusters: {clust_result.get("n_clusters")}')
|
||||
cid = clust_result.get('cluster_result_id', '')
|
||||
if cid and 'error' not in clust_result:
|
||||
feat_result = asyncio.run(_handle_extract_features(
|
||||
dataset_id='entity_data', cluster_result_id=cid,
|
||||
top_k=10, method='zscore', save_to_db=True))
|
||||
print(f'Features: {feat_result.get("n_features")} db_saved={feat_result.get("db_saved")}')
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Test full LLM orchestrator with actual tools"""
|
||||
import sys, os, json, asyncio
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.tool_registry import handle_call
|
||||
import urllib.request
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
|
||||
TOOLS = [{"type": "function", "function": {
|
||||
"name": "profile_data", "description": "Get column statistics",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}
|
||||
}}, {"type": "function", "function": {
|
||||
"name": "build_entity_profiles", "description": "Auto-detect entity and aggregate",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "auto_detect": {"type": "boolean"}}, "required": ["dataset_id"]}
|
||||
}}, {"type": "function", "function": {
|
||||
"name": "run_clustering", "description": "Run HDBSCAN clustering",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_columns": {"type": "array", "items": {"type": "string"}}, "algorithm": {"type": "string", "enum": ["hdbscan", "kmeans"]}}, "required": ["dataset_id", "cluster_columns"]}
|
||||
}}, {"type": "function", "function": {
|
||||
"name": "extract_features", "description": "Extract distinguishing features",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_result_id": {"type": "string"}, "method": {"type": "string", "enum": ["zscore", "anova"]}}, "required": ["dataset_id", "cluster_result_id"]}
|
||||
}}]
|
||||
|
||||
def call_llm(messages):
|
||||
payload = json.dumps({"model": "deepseek-v4-flash", "messages": messages, "tools": TOOLS,
|
||||
"tool_choice": "auto", "max_tokens": 4096}).encode()
|
||||
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
|
||||
headers={'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST')
|
||||
return json.loads(urllib.request.urlopen(req, timeout=30).read())
|
||||
|
||||
msgs = [{"role": "user", "content": "Analyze dataset ds. Profile it, build entity profiles, cluster, extract features."}]
|
||||
|
||||
for step in range(10):
|
||||
print(f'\nStep {step + 1}:')
|
||||
response = call_llm(msgs)
|
||||
msg = response["choices"][0]["message"]
|
||||
|
||||
if "tool_calls" not in msg or not msg["tool_calls"]:
|
||||
print(f'Done: {msg.get("content", "")[:200]}')
|
||||
break
|
||||
|
||||
for tc in msg["tool_calls"]:
|
||||
func = tc.get("function", {})
|
||||
name = func.get("name", "")
|
||||
try:
|
||||
args = json.loads(func.get("arguments", "{}"))
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
print(f' -> {name}({args})')
|
||||
|
||||
result = asyncio.run(handle_call(name, args))
|
||||
content = json.dumps(result, default=str, ensure_ascii=False)[:2000]
|
||||
print(f' <- {name}: ok' if 'error' not in result else f' <- ERROR: {result["error"]}')
|
||||
|
||||
msgs.append({"role": "tool", "tool_call_id": tc["id"], "content": content})
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Debug LLM orchestrator 400 on second call"""
|
||||
import sys, os, json, asyncio
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.tool_registry import handle_call
|
||||
import urllib.request
|
||||
|
||||
store = SessionStore(); store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
|
||||
TOOLS = [{"type": "function", "function": {"name": "profile_data",
|
||||
"description": "Get column statistics",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}}}]
|
||||
|
||||
# Step 1: Get tool call from LLM
|
||||
msgs = [{"role": "user", "content": "Call profile_data with ds"}]
|
||||
payload = json.dumps({"model": "deepseek-v4-flash", "messages": msgs, "tools": TOOLS,
|
||||
"tool_choice": "auto", "max_tokens": 200}).encode()
|
||||
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
|
||||
headers={'Content-Type': 'application/json', 'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST')
|
||||
resp = json.loads(urllib.request.urlopen(req, timeout=10).read())
|
||||
msg = resp["choices"][0]["message"]
|
||||
tc = msg["tool_calls"][0]
|
||||
print(f'Got tool call: {tc["function"]["name"]}')
|
||||
|
||||
# Execute tool
|
||||
result = asyncio.run(handle_call(tc["function"]["name"], json.loads(tc["function"]["arguments"])))
|
||||
content = json.dumps(result, default=str, ensure_ascii=False)
|
||||
print(f'Result length: {len(content)}')
|
||||
|
||||
# Step 2: Send back - with correct message format
|
||||
msgs2 = msgs + [msg, {"role": "tool", "tool_call_id": tc["id"], "content": content[:2000]}]
|
||||
payload2 = json.dumps({"model": "deepseek-v4-flash", "messages": msgs2, "tools": TOOLS,
|
||||
"tool_choice": "auto", "max_tokens": 200}).encode()
|
||||
req2 = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload2,
|
||||
headers={'Content-Type': 'application/json', 'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST')
|
||||
try:
|
||||
resp2 = json.loads(urllib.request.urlopen(req2, timeout=10).read())
|
||||
print(f'Step 2 OK: {resp2["choices"][0]["message"]["content"][:100]}')
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:500]
|
||||
print(f'Step 2 400: {body}')
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Test PCA embedding in _handle_extract_features"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
import asyncio
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.tool_registry import _handle_run_clustering, _handle_extract_features
|
||||
|
||||
store = SessionStore(); store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
entity_col = detect_entity_column('ds')['recommended']
|
||||
print(f'Entity col: {entity_col}')
|
||||
agg_lf, features = aggregate_by_entity(lf, entity_col, schema)
|
||||
df_agg = agg_lf.collect()
|
||||
store.store_dataset('entity_data', agg_lf, schema=dict(zip(features, ['']*len(features))),
|
||||
metadata={'row_count': len(df_agg), 'csv_glob': csv})
|
||||
print(f'Aggregated: {len(df_agg)} rows, {len(features)} features')
|
||||
|
||||
# Create a dummy AnalysisRun so PCA embedding can save EntityProfiles
|
||||
from analysis.models import AnalysisRun
|
||||
run_for_pca, _ = AnalysisRun.objects.get_or_create(csv_glob=csv, defaults={'status': 'running'})
|
||||
run_for_pca.status = 'running'
|
||||
run_for_pca.save()
|
||||
print(f'AnalysisRun: #{run_for_pca.id}')
|
||||
|
||||
clust = asyncio.run(_handle_run_clustering(dataset_id='entity_data',
|
||||
cluster_columns=[c for c in features if not c.startswith('_')][:8],
|
||||
algorithm='hdbscan', params={}, random_state=42))
|
||||
cid = clust.get('cluster_result_id', '')
|
||||
print(f'Cluster result: {cid}')
|
||||
|
||||
feat = asyncio.run(_handle_extract_features(dataset_id='entity_data', cluster_result_id=cid,
|
||||
top_k=10, method='zscore', save_to_db=True))
|
||||
print(f'Result keys: {list(feat.keys())}')
|
||||
if 'error' in feat:
|
||||
print(f'ERROR: {feat["error"]}')
|
||||
print(f'n_features={feat.get("n_features")} db_saved={feat.get("db_saved")}')
|
||||
|
||||
from analysis.models import EntityProfile
|
||||
cnt = EntityProfile.objects.exclude(embedding_x=None).count()
|
||||
total = EntityProfile.objects.count()
|
||||
print(f'EntityProfile with PCA coords: {cnt}/{total}')
|
||||
if cnt > 0:
|
||||
ep = EntityProfile.objects.exclude(embedding_x=None).first()
|
||||
print(f'Sample: {ep.entity_value} ({ep.embedding_x}, {ep.embedding_y}) cluster={ep.cluster_label}')
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Test profile_data fix"""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tianxuan.settings'
|
||||
import django; django.setup()
|
||||
import asyncio
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.tool_registry import handle_call
|
||||
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
csv = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'data', 'complex_test.csv')
|
||||
lf, schema, rc, fc, mem = load_csv_directory(csv)
|
||||
store.store_dataset('ds', lf, schema=schema, metadata={'row_count': rc, 'csv_glob': csv})
|
||||
|
||||
result = asyncio.run(handle_call('profile_data', {'dataset_id': 'ds'}))
|
||||
if 'error' in result:
|
||||
print(f'FAIL: {result["error"]}')
|
||||
else:
|
||||
cols = result.get('column_stats', {})
|
||||
print(f'OK: {len(cols)} columns profiled')
|
||||
for name, stats in list(cols.items())[:3]:
|
||||
print(f' {name}: dtype={stats.get("dtype")} mean={stats.get("mean","-")}')
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Test DeepSeek tool calling with follow-up"""
|
||||
import json, urllib.request
|
||||
|
||||
def chat(messages, tools=None):
|
||||
payload = {"model": "deepseek-v4-flash", "messages": messages, "max_tokens": 200}
|
||||
if tools:
|
||||
payload["tools"] = tools
|
||||
payload["tool_choice"] = "auto"
|
||||
req = urllib.request.Request(
|
||||
'https://api.deepseek.com/chat/completions',
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST'
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
return json.loads(resp.read())
|
||||
|
||||
tools = [{"type": "function", "function": {
|
||||
"name": "get_weather", "description": "Get weather",
|
||||
"parameters": {"type": "object", "properties": {"loc": {"type": "string"}}, "required": ["loc"]}
|
||||
}}]
|
||||
|
||||
# Call 1: LLM returns tool_call
|
||||
msgs = [{"role": "user", "content": "Get weather for Beijing"}]
|
||||
resp1 = chat(msgs, tools)
|
||||
msg1 = resp1["choices"][0]["message"]
|
||||
print(f'Call 1: tool_calls={bool(msg1.get("tool_calls"))}')
|
||||
if msg1.get("tool_calls"):
|
||||
tc = msg1["tool_calls"][0]
|
||||
print(f' tool={tc["function"]["name"]} args={tc["function"]["arguments"]}')
|
||||
|
||||
# Call 2: Send tool result back (test different formats)
|
||||
print('\nTest 1: with "name" field:')
|
||||
msgs2 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "name": tc["function"]["name"], "content": '{"temp": 22}'}]
|
||||
try:
|
||||
resp2 = chat(msgs2)
|
||||
print(f' OK: {resp2["choices"][0]["message"]["content"][:100]}')
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f' 400: {e.read().decode()[:200]}')
|
||||
|
||||
print('\nTest 2: without "name" field:')
|
||||
msgs3 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "content": '{"temp": 22}'}]
|
||||
try:
|
||||
resp3 = chat(msgs3)
|
||||
print(f' OK: {resp3["choices"][0]["message"]["content"][:100]}')
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f' 400: {e.read().decode()[:200]}')
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Test tool calling loop with DeepSeek"""
|
||||
import json, urllib.request
|
||||
|
||||
tools = [{"type": "function", "function": {
|
||||
"name": "profile_data", "description": "Profile dataset",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}
|
||||
}}]
|
||||
|
||||
def call(messages):
|
||||
payload = json.dumps({"model": "deepseek-v4-flash", "messages": messages, "tools": tools,
|
||||
"tool_choice": "auto", "max_tokens": 200}).encode()
|
||||
req = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload,
|
||||
headers={'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST')
|
||||
return json.loads(urllib.request.urlopen(req, timeout=10).read())
|
||||
|
||||
# Call 1: Get tool call
|
||||
msgs = [{"role": "user", "content": "Call profile_data with ds"}]
|
||||
r1 = call(msgs)
|
||||
msg1 = r1["choices"][0]["message"]
|
||||
tc = msg1["tool_calls"][0]
|
||||
print(f'Call 1: id={tc["id"][:20]} tool={tc["function"]["name"]}')
|
||||
|
||||
# Call 2: Send result back (with tools still in payload)
|
||||
msgs2 = msgs + [msg1, {"role": "tool", "tool_call_id": tc["id"], "content": '{"status": "ok"}'}]
|
||||
try:
|
||||
r2 = call(msgs2)
|
||||
print(f'Call 2: {r2["choices"][0]["message"]["content"][:100]}')
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f'Call 2 400: body={e.read().decode()[:300]}')
|
||||
|
||||
# Call 3: Without tools in payload
|
||||
try:
|
||||
payload3 = json.dumps({"model": "deepseek-v4-flash", "messages": msgs2, "max_tokens": 200}).encode()
|
||||
req3 = urllib.request.Request('https://api.deepseek.com/chat/completions', data=payload3,
|
||||
headers={'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer sk-360ef76d59674d6b8bc2eb160327dd39'},
|
||||
method='POST')
|
||||
r3 = json.loads(urllib.request.urlopen(req3, timeout=10).read())
|
||||
print(f'Call 3 (no tools): {r3["choices"][0]["message"]["content"][:100]}')
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f'Call 3 400: body={e.read().decode()[:300]}')
|
||||
@@ -0,0 +1,42 @@
|
||||
"""
|
||||
天璇运行时隔离验证脚本。在不同机器上运行，输出应完全一致。
|
||||
"""
|
||||
import sys, site, importlib
|
||||
|
||||
ok = True
|
||||
|
||||
# 1. 检查 Python 路径
|
||||
print(f'Python: {sys.version}')
|
||||
print(f'base_exec_prefix: {sys.base_exec_prefix}')
|
||||
if '天璇' not in sys.base_exec_prefix and 'runtime' not in sys.base_exec_prefix:
|
||||
print('WARN: base_exec_prefix does not point to runtime Python')
|
||||
# This is expected when running via uv, so not a failure
|
||||
|
||||
# 2. 检查 site-packages 隔离
|
||||
print(f'ENABLE_USER_SITE: {site.ENABLE_USER_SITE}')
|
||||
|
||||
# 3. 检查核心依赖版本
|
||||
checks = [
|
||||
('django', '4.2'),
|
||||
('polars', '1.42'),
|
||||
('sklearn', '1.5'),
|
||||
('numpy', '1.26'),
|
||||
('yaml', ''),
|
||||
('pydantic', '2.'),
|
||||
]
|
||||
for mod_name, ver_prefix in checks:
|
||||
try:
|
||||
mod = importlib.import_module(mod_name)
|
||||
ver = getattr(mod, '__version__', 'unknown')
|
||||
match = 'YES' if not ver_prefix or ver.startswith(ver_prefix) else f'NEEDS: {ver_prefix}xxx, GOT: {ver}'
|
||||
print(f' {mod_name:15s} {ver:20s} {match}')
|
||||
except Exception as e:
|
||||
print(f' {mod_name:15s} FAIL: {e}')
|
||||
ok = False
|
||||
|
||||
# 4. 检查字符编码
|
||||
import sys
|
||||
print(f'defaultencoding: {sys.getdefaultencoding()}')
|
||||
print(f'filesystemencoding: {sys.getfilesystemencoding()}')
|
||||
|
||||
print(f'\nOverall: {"PASS" if ok else "FAIL"}')
|
||||
@@ -0,0 +1,418 @@
|
||||
#!/usr/bin/env python
|
||||
"""Convert WNL QoS TLS dataset to the user's 44-column CSV format.
|
||||
|
||||
The WNL dataset (tab-separated) has columns:
|
||||
SNI\tservice_label\tPKT_1_payload\tPKT_2_payload
|
||||
|
||||
PKT_1_payload contains TLS record bytes as comma-separated decimal ints.
|
||||
This script parses the TLS ClientHello and generates the user's 44-column
|
||||
format, appending the ground-truth label as the last column.
|
||||
|
||||
Usage:
|
||||
runtime\python\python.exe scripts\wnl_to_user_csv.py
|
||||
"""
|
||||
|
||||
import csv
|
||||
import os
|
||||
import random
|
||||
import struct
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
random.seed(42)
|
||||
|
||||
# ── Paths ──────────────────────────────────────────────────────────────────
|
||||
BASE = Path(r"C:\Users\25044\Desktop\Proj")
|
||||
WNL_CSV = BASE / "qos-tls-dataset-of-enc-traffic" / "Preprocessed_CSV" / "Preprocessed_CSV" / "WNL_TLS_Dataset_Origin.csv"
|
||||
OUT_CSV = BASE / "天璇" / "data" / "wnl_converted.csv"
|
||||
OUT_CSV.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# ── User 44-column header ──────────────────────────────────────────────────
|
||||
USER_HEADER = [
|
||||
":ips", ":ipd", ":prs", ":prd", "scnt", "dcnt",
|
||||
"server-ip", "client-ip",
|
||||
"0ver", "snam", "cnam",
|
||||
"4dur", "8ses", "2tmo", "4ksz",
|
||||
"cnrs", "isrs",
|
||||
"8ack", "8ppk", "8dbd", "1ipp",
|
||||
"4dbn", "tabl", "name", "source-node",
|
||||
"cipher-suite", "ecdhe-named-curve", "0cph", "0crv",
|
||||
"0rnd", "0rnt",
|
||||
"row", "time", "timestamp",
|
||||
":ips.latd", ":ips.lond", ":ipd.latd", ":ipd.lond",
|
||||
":ips.ispn", ":ipd.ispn", ":ips.orgn", ":ipd.orgn",
|
||||
":ips.city", ":ipd.city",
|
||||
]
|
||||
|
||||
# ── Cipher suite hex → IANA name map ──────────────────────────────────────
|
||||
CIPHER_HEX_MAP = {
|
||||
0x1301: "TLS_AES_128_GCM_SHA256",
|
||||
0x1302: "TLS_AES_256_GCM_SHA384",
|
||||
0x1303: "TLS_CHACHA20_POLY1305_SHA256",
|
||||
0xC02B: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
|
||||
0xC02C: "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
|
||||
0xC02F: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
|
||||
0xC030: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
|
||||
0xCCA9: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
|
||||
0xCCAC: "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
|
||||
0x009C: "TLS_RSA_WITH_AES_128_GCM_SHA256",
|
||||
0x009D: "TLS_RSA_WITH_AES_256_GCM_SHA384",
|
||||
0x003C: "TLS_RSA_WITH_AES_128_CBC_SHA256",
|
||||
0x003D: "TLS_RSA_WITH_AES_256_CBC_SHA256",
|
||||
0xC009: "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
|
||||
0xC013: "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
|
||||
0x002F: "TLS_RSA_WITH_AES_128_CBC_SHA",
|
||||
0x0035: "TLS_RSA_WITH_AES_128_CBC_SHA",
|
||||
0x0005: "TLS_RSA_WITH_RC4_128_SHA",
|
||||
0x00FF: "TLS_RSA_WITH_NULL_MD5",
|
||||
}
|
||||
|
||||
# ── Named curve hex → name map ────────────────────────────────────────────
|
||||
CURVE_HEX_MAP = {
|
||||
0x0017: "secp256r1",
|
||||
0x0018: "secp384r1",
|
||||
0x0019: "secp521r1",
|
||||
0x001D: "x25519",
|
||||
0x001E: "x448",
|
||||
}
|
||||
|
||||
# ── Service class → IP / geo / duration profiles ──────────────────────────
|
||||
SERVICE_PROFILES = {
|
||||
"Web": {"ip8": "10", "cnt": "US", "dur_range": (0.5, 15), "isp": "Cloudflare", "org": "Cloudflare Inc.", "city": "San Francisco"},
|
||||
"Spotify": {"ip8": "20", "cnt": "SE", "dur_range": (30, 600), "isp": "Telia", "org": "Spotify AB", "city": "Stockholm"},
|
||||
"Netflix": {"ip8": "30", "cnt": "US", "dur_range": (60, 1200), "isp": "Comcast", "org": "Netflix Inc.", "city": "Los Gatos"},
|
||||
"YouTube": {"ip8": "40", "cnt": "US", "dur_range": (30, 900), "isp": "Google Fiber", "org": "Google LLC", "city": "Mountain View"},
|
||||
"AppleMusic": {"ip8": "50", "cnt": "US", "dur_range": (30, 600), "isp": "AT&T", "org": "Apple Inc.", "city": "Cupertino"},
|
||||
"SoundCloud": {"ip8": "60", "cnt": "DE", "dur_range": (30, 500), "isp": "Deutsche Telekom", "org": "SoundCloud Ltd.", "city": "Berlin"},
|
||||
"PrimeVideo": {"ip8": "70", "cnt": "US", "dur_range": (60, 1200), "isp": "Amazon", "org": "Amazon.com Inc.", "city": "Seattle"},
|
||||
"LiveYouTube": {"ip8": "80", "cnt": "US", "dur_range": (60, 1800), "isp": "Google Fiber", "org": "Google LLC", "city": "Mountain View"},
|
||||
"LiveFacebook": {"ip8": "90", "cnt": "US", "dur_range": (60, 1800), "isp": "Facebook", "org": "Meta Platforms Inc.", "city": "Menlo Park"},
|
||||
"Kinopoisk": {"ip8": "100", "cnt": "RU", "dur_range": (30, 900), "isp": "Rostelecom", "org": "Yandex LLC", "city": "Moscow"},
|
||||
"YandexMusic": {"ip8": "110", "cnt": "RU", "dur_range": (30, 600), "isp": "Rostelecom", "org": "Yandex LLC", "city": "Moscow"},
|
||||
"Vimeo": {"ip8": "120", "cnt": "US", "dur_range": (30, 600), "isp": "Akamai", "org": "Vimeo Inc.", "city": "New York"},
|
||||
}
|
||||
|
||||
# ── TLS version hex → display ─────────────────────────────────────────────
|
||||
TLS_VER_MAP = {
|
||||
(3, 4): "03 04",
|
||||
(3, 3): "03 03",
|
||||
(3, 2): "03 02",
|
||||
(3, 1): "03 01",
|
||||
}
|
||||
|
||||
|
||||
def parse_tls_clienthello(payload_bytes: bytes) -> dict:
|
||||
"""Parse TLS ClientHello from raw bytes, returning extracted fields.
|
||||
|
||||
Returns dict with keys:
|
||||
record_version, ch_version, cipher_suites, cipher_suite_hex,
|
||||
sni, supported_groups
|
||||
Missing/parse-failure values are empty strings or empty lists.
|
||||
"""
|
||||
result = {
|
||||
"record_version": "",
|
||||
"ch_version": "",
|
||||
"cipher_suites": [],
|
||||
"cipher_suite_hex": "",
|
||||
"sni": "",
|
||||
"supported_groups": [],
|
||||
"curves_hex": "",
|
||||
}
|
||||
|
||||
if len(payload_bytes) < 5:
|
||||
return result
|
||||
|
||||
content_type = payload_bytes[0]
|
||||
if content_type != 22: # Handshake
|
||||
return result
|
||||
|
||||
# Record layer version
|
||||
rv_major = payload_bytes[1]
|
||||
rv_minor = payload_bytes[2]
|
||||
result["record_version"] = TLS_VER_MAP.get((rv_major, rv_minor), f"{rv_major:02x} {rv_minor:02x}")
|
||||
|
||||
if len(payload_bytes) < 6:
|
||||
return result
|
||||
|
||||
hs_type = payload_bytes[5]
|
||||
if hs_type != 1: # ClientHello
|
||||
return result
|
||||
|
||||
# ClientHello version at offset 9-10
|
||||
if len(payload_bytes) < 11:
|
||||
return result
|
||||
ch_major = payload_bytes[9]
|
||||
ch_minor = payload_bytes[10]
|
||||
result["ch_version"] = TLS_VER_MAP.get((ch_major, ch_minor), f"{ch_major:02x} {ch_minor:02x}")
|
||||
|
||||
# Session ID length at offset 43
|
||||
if len(payload_bytes) < 44:
|
||||
return result
|
||||
sess_len = payload_bytes[43]
|
||||
offset = 44 + sess_len
|
||||
|
||||
# Cipher suites: 2 bytes length, then N bytes (2-byte each)
|
||||
if len(payload_bytes) < offset + 2:
|
||||
return result
|
||||
cs_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
|
||||
offset += 2
|
||||
cs_end = offset + cs_len
|
||||
if cs_end > len(payload_bytes):
|
||||
cs_end = len(payload_bytes)
|
||||
|
||||
cipher_suite_hex_list = []
|
||||
for i in range(offset, cs_end, 2):
|
||||
if i + 1 < cs_end:
|
||||
cs_val = (payload_bytes[i] << 8) | payload_bytes[i + 1]
|
||||
cs_hex = f"{payload_bytes[i]:02x} {payload_bytes[i+1]:02x}"
|
||||
cipher_suite_hex_list.append(cs_hex)
|
||||
result["cipher_suites"].append(cs_val)
|
||||
|
||||
result["cipher_suite_hex"] = cipher_suite_hex_list[0] if cipher_suite_hex_list else ""
|
||||
|
||||
offset = cs_end
|
||||
# Compression methods
|
||||
if offset >= len(payload_bytes):
|
||||
return result
|
||||
comp_len = payload_bytes[offset]
|
||||
offset += 1 + comp_len
|
||||
|
||||
# Extensions
|
||||
if offset + 1 >= len(payload_bytes):
|
||||
return result
|
||||
ext_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
|
||||
offset += 2
|
||||
ext_end = offset + ext_len
|
||||
if ext_end > len(payload_bytes):
|
||||
ext_end = len(payload_bytes)
|
||||
|
||||
while offset + 3 < ext_end:
|
||||
ext_type = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
|
||||
ext_data_len = (payload_bytes[offset + 2] << 8) | payload_bytes[offset + 3]
|
||||
offset += 4
|
||||
ext_data_end = offset + ext_data_len
|
||||
if ext_data_end > ext_end:
|
||||
ext_data_end = ext_end
|
||||
|
||||
if ext_type == 0 and ext_data_len > 5: # server_name
|
||||
# Skip list length (2) + type (1) + name length (2)
|
||||
sni_name_len = (payload_bytes[offset + 3] << 8) | payload_bytes[offset + 4]
|
||||
if offset + 5 + sni_name_len <= ext_data_end:
|
||||
sni_bytes = payload_bytes[offset + 5: offset + 5 + sni_name_len]
|
||||
try:
|
||||
result["sni"] = sni_bytes.decode("ascii", errors="replace")
|
||||
except Exception:
|
||||
result["sni"] = sni_bytes.decode("ascii", errors="replace")
|
||||
|
||||
elif ext_type == 10 and ext_data_len > 1: # supported_groups
|
||||
groups_len = (payload_bytes[offset] << 8) | payload_bytes[offset + 1]
|
||||
groups_hex = []
|
||||
for gi in range(0, groups_len, 2):
|
||||
idx = offset + 2 + gi
|
||||
if idx + 1 < ext_data_end:
|
||||
gv = (payload_bytes[idx] << 8) | payload_bytes[idx + 1]
|
||||
groups_hex.append(f"{payload_bytes[idx]:02x} {payload_bytes[idx+1]:02x}")
|
||||
result["supported_groups"].append(gv)
|
||||
result["curves_hex"] = groups_hex[0] if groups_hex else ""
|
||||
|
||||
offset = ext_data_end
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def make_ip(ip8: str, idx: int) -> str:
|
||||
"""Generate a plausible IP from a /8 prefix."""
|
||||
return f"{ip8}.{random.randint(1, 254)}.{random.randint(1, 254)}.{random.randint(1, 254)}"
|
||||
|
||||
|
||||
def make_server_ip(cnt: str) -> str:
|
||||
"""Generate a server IP that looks like a CDN/cloud provider."""
|
||||
prefixes = {
|
||||
"US": "203.0.113", # example/documentation range
|
||||
"SE": "198.51.100",
|
||||
"DE": "203.0.114",
|
||||
"RU": "198.52.100",
|
||||
"GB": "203.0.115",
|
||||
}
|
||||
p = prefixes.get(cnt, "203.0.113")
|
||||
return f"{p}.{random.randint(1, 254)}"
|
||||
|
||||
|
||||
def main():
|
||||
print(f"[1/3] 读取 WNL 数据集: {WNL_CSV}")
|
||||
|
||||
# Read the origin file and parse all rows
|
||||
rows_data = [] # list of (sni, service_label, pkt1_bytes)
|
||||
with open(WNL_CSV, encoding="utf-8") as f:
|
||||
# CSV with tab delimiter
|
||||
reader = csv.reader(f, delimiter="\t")
|
||||
headers = next(reader)
|
||||
print(f" Headers: {headers}")
|
||||
|
||||
for row in reader:
|
||||
if not row or len(row) < 4:
|
||||
continue
|
||||
sni = row[0].strip()
|
||||
label = row[1].strip()
|
||||
# PKT_1_payload is comma-separated decimal ints
|
||||
pkt1_str = row[2].strip()
|
||||
if not pkt1_str:
|
||||
continue
|
||||
try:
|
||||
pkt1_bytes = bytes(int(b.strip()) for b in pkt1_str.split(",") if b.strip())
|
||||
except ValueError:
|
||||
continue
|
||||
rows_data.append((sni, label, pkt1_bytes))
|
||||
|
||||
print(f" 总共 {len(rows_data)} 行")
|
||||
|
||||
# Count labels
|
||||
label_counts = {}
|
||||
for _, lbl, _ in rows_data:
|
||||
label_counts[lbl] = label_counts.get(lbl, 0) + 1
|
||||
print(f" Service classes ({len(label_counts)}):")
|
||||
for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]):
|
||||
print(f" {lbl}: {cnt}")
|
||||
|
||||
print(f"\n[2/3] 解析 TLS ClientHello 并生成 44 列格式")
|
||||
|
||||
# Generate output
|
||||
out_rows = []
|
||||
parse_errors = 0
|
||||
|
||||
for idx, (sni, label, pkt1_bytes) in enumerate(rows_data):
|
||||
tls_info = parse_tls_clienthello(pkt1_bytes)
|
||||
if not tls_info["record_version"]:
|
||||
parse_errors += 1
|
||||
|
||||
prof = SERVICE_PROFILES.get(label, SERVICE_PROFILES["Web"])
|
||||
ip8 = prof["ip8"]
|
||||
cnt = prof["cnt"]
|
||||
dur_range = prof["dur_range"]
|
||||
isp = prof["isp"]
|
||||
org = prof["org"]
|
||||
city = prof["city"]
|
||||
|
||||
# Synthetic IPs
|
||||
src_ip = make_ip(ip8, idx) # client IP (:ips)
|
||||
dst_ip = make_server_ip(cnt) # server IP (:ipd)
|
||||
|
||||
# TLS version: prefer CH version, fallback to record version
|
||||
tls_ver = tls_info["ch_version"] or tls_info["record_version"] or "03 03"
|
||||
|
||||
# Cipher suite
|
||||
cipher_suites = tls_info["cipher_suites"]
|
||||
cipher_suite_hex = tls_info["cipher_suite_hex"]
|
||||
cipher_suite_name = ""
|
||||
if cipher_suites:
|
||||
cs_val = cipher_suites[0]
|
||||
cipher_suite_name = CIPHER_HEX_MAP.get(cs_val, f"UNKNOWN_0x{cs_val:04X}")
|
||||
else:
|
||||
# Default for TLS 1.3
|
||||
cipher_suite_hex = "13 01"
|
||||
cipher_suite_name = "TLS_AES_128_GCM_SHA256"
|
||||
|
||||
# Named curve
|
||||
supported_groups = tls_info["supported_groups"]
|
||||
curves_hex = tls_info["curves_hex"]
|
||||
curve_name = ""
|
||||
if supported_groups:
|
||||
cg_val = supported_groups[0]
|
||||
curve_name = CURVE_HEX_MAP.get(cg_val, f"unknown_curve_0x{cg_val:04X}")
|
||||
else:
|
||||
curves_hex = "00 1d" # default x25519
|
||||
curve_name = "x25519"
|
||||
|
||||
# SNI: use parsed or the raw SNI column
|
||||
sni_val = tls_info.get("sni") or sni
|
||||
|
||||
# Duration based on service class
|
||||
duration = round(random.uniform(*dur_range), 2)
|
||||
|
||||
# Random values for synthetic columns
|
||||
rand_bytes = random.randint(1000, 200000)
|
||||
rand_packets = random.randint(5, 100)
|
||||
ses_val = round(random.uniform(100, 1000), 4)
|
||||
tmo_val = round(random.uniform(100, 500), 4)
|
||||
ksz_val = random.choice([128, 256, 384, 512])
|
||||
cnrs_val = random.choice(["True", ""])
|
||||
isrs_val = random.choice(["True", ""])
|
||||
|
||||
# Timestamp: base it on row index spread over a month
|
||||
base_dt = datetime(2024, 6, 1, 0, 0, 0)
|
||||
dt = base_dt + timedelta(seconds=random.randint(0, 30 * 24 * 3600))
|
||||
ts_str = dt.strftime("%Y-%m-%d %H:%M:%S")
|
||||
unix_ts = int(dt.timestamp())
|
||||
|
||||
# Random hex random bytes
|
||||
rnd_hex = " ".join(f"{random.randint(0, 255):02x}" for _ in range(32))
|
||||
|
||||
row_data = [
|
||||
src_ip, # :ips
|
||||
dst_ip, # :ipd
|
||||
str(random.randint(40000, 65000)), # :prs (source port)
|
||||
"443", # :prd (destination port, typically 443)
|
||||
"US", # scnt (source country)
|
||||
cnt, # dcnt (destination country)
|
||||
dst_ip, # server-ip
|
||||
src_ip, # client-ip
|
||||
tls_ver, # 0ver
|
||||
sni_val, # snam
|
||||
sni_val, # cnam
|
||||
str(duration), # 4dur
|
||||
str(ses_val), # 8ses
|
||||
str(tmo_val), # 2tmo
|
||||
str(ksz_val), # 4ksz
|
||||
cnrs_val, # cnrs
|
||||
isrs_val, # isrs
|
||||
str(rand_bytes), # 8ack
|
||||
str(rand_packets), # 8ppk
|
||||
ts_str, # 8dbd
|
||||
"6", # 1ipp (TCP)
|
||||
"packet_capture", # 4dbn
|
||||
"wan-link", # tabl
|
||||
"wnl_tls", # name
|
||||
"gw-01", # source-node
|
||||
cipher_suite_name, # cipher-suite
|
||||
curve_name, # ecdhe-named-curve
|
||||
cipher_suite_hex, # 0cph
|
||||
curves_hex, # 0crv
|
||||
rnd_hex, # 0rnd
|
||||
str(unix_ts), # 0rnt
|
||||
str(idx + 1), # row
|
||||
dt.strftime("%H:%M:%S"), # time
|
||||
ts_str, # timestamp
|
||||
"", # :ips.latd (leave blank)
|
||||
"", # :ips.lond (leave blank)
|
||||
"", # :ipd.latd (leave blank)
|
||||
"", # :ipd.lond (leave blank)
|
||||
"", # :ips.ispn (leave blank)
|
||||
"", # :ipd.ispn (leave blank)
|
||||
"", # :ips.orgn (leave blank)
|
||||
"", # :ipd.orgn (leave blank)
|
||||
"", # :ips.city (leave blank)
|
||||
"", # :ipd.city (leave blank)
|
||||
]
|
||||
|
||||
# Append label as the LAST column
|
||||
row_data.append(label)
|
||||
|
||||
out_rows.append(row_data)
|
||||
|
||||
print(f" TLS 解析失败: {parse_errors}/{len(rows_data)}")
|
||||
print(f" 成功生成: {len(out_rows)} 行")
|
||||
|
||||
# Write output CSV
|
||||
print(f"\n[3/3] 写入输出: {OUT_CSV}")
|
||||
header = USER_HEADER + ["label"]
|
||||
with open(OUT_CSV, "w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(header)
|
||||
writer.writerows(out_rows)
|
||||
|
||||
print(f" 完成! 输出 {len(out_rows)} 行 x {len(header)} 列")
|
||||
print(f" Columns: {header}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,4 @@
|
||||
@echo off
|
||||
cd /d "%~dp0"
|
||||
runtime\python\python.exe manage.py shell
|
||||
pause
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
@@ -0,0 +1,10 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">
|
||||
<defs>
|
||||
<linearGradient id="g" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0%" stop-color="#4361ee"/>
|
||||
<stop offset="100%" stop-color="#7b2ff7"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
<circle cx="32" cy="32" r="28" fill="url(#g)"/>
|
||||
<text x="32" y="42" text-anchor="middle" font-size="32" font-family="sans-serif" fill="#fff">璇</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 417 B |
Vendored
+6
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,62 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}Cluster #{{ cluster.cluster_label }} - Run #{{ run.id }}{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<a href="{% url 'analysis:cluster_overview' run.id %}" style="color:#4361ee;text-decoration:none;font-size:0.9rem;">← Back to overview</a>
|
||||
<h2 style="margin-top:0.5rem;">Cluster #{{ cluster.cluster_label }}</h2>
|
||||
<p>{{ cluster.size }} entities ({{ cluster.proportion|floatformat:2 }}% of total)</p>
|
||||
<p>Silhouette Score: {{ cluster.silhouette_score|floatformat:4|default:"-" }}</p>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">
|
||||
<div class="card">
|
||||
<h2>Feature Summary (top 50)</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Feature</th>
|
||||
<th>Mean</th>
|
||||
<th>Std</th>
|
||||
<th>Median</th>
|
||||
<th>Dist. Score</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for f in features %}
|
||||
<tr>
|
||||
<td><code>{{ f.feature_name }}</code></td>
|
||||
<td>{{ f.mean|floatformat:3|default:"-" }}</td>
|
||||
<td>{{ f.std|floatformat:3|default:"-" }}</td>
|
||||
<td>{{ f.median|floatformat:3|default:"-" }}</td>
|
||||
<td>{{ f.distinguishing_score|floatformat:3|default:"-" }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="5">No features</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Entities (top 50)</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Entity</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for e in entities %}
|
||||
<tr>
|
||||
<td><code>{{ e.entity_value }}</code></td>
|
||||
<td><a href="{% url 'analysis:entity_profile' e.id %}" class="btn btn-primary">Profile</a></td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="2">No entities</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,286 @@
|
||||
{% extends 'base.html' %}
|
||||
{% load static %}
|
||||
{% block title %}Cluster Overview - Run #{{ run.id }}{% endblock %}
|
||||
{% block content %}
|
||||
<style>
|
||||
.feature-chart { width: 100%; display: block; }
|
||||
#featureChartsContainer canvas { max-height: 400px; }
|
||||
</style>
|
||||
<div class="card">
|
||||
<h2>Cluster Overview — Run #{{ run.id }}</h2>
|
||||
<p>Entity column: <code>{{ run.entity_column }}</code> | {{ run.entity_count }} entities, {{ run.cluster_count }} clusters</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>PCA-2D Embedding</h2>
|
||||
<canvas id="scatterChart" class="scatter-canvas"></canvas>
|
||||
</div>
|
||||
|
||||
{% if geo_count > 0 %}
|
||||
<div class="card">
|
||||
<h2>地理位置分布 <span class="badge badge-info">{{ geo_count }} entities</span></h2>
|
||||
{% if geo_skipped > 0 %}
|
||||
<p style="color:#888;font-size:0.85rem;">{{ geo_skipped }} entities excluded (missing lat/lon)</p>
|
||||
{% endif %}
|
||||
<canvas id="geoChart" class="scatter-canvas"></canvas>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<div class="card" id="clusterSizesCard">
|
||||
<h2>Cluster Size Distribution</h2>
|
||||
<canvas id="clusterSizesChart" class="scatter-canvas"></canvas>
|
||||
</div>
|
||||
|
||||
<div class="card" id="silhouetteCard">
|
||||
<h2>Silhouette Score Comparison</h2>
|
||||
<canvas id="silhouetteChart" class="scatter-canvas"></canvas>
|
||||
</div>
|
||||
|
||||
<div class="card" id="topFeaturesCard">
|
||||
<h2>Top Distinguishing Features per Cluster</h2>
|
||||
<div id="featureChartsContainer"></div>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">
|
||||
{% for c in clusters %}
|
||||
<div class="card">
|
||||
<h3>Cluster #{{ c.cluster_label }} <span class="badge badge-info">{{ c.size }} entities</span></h3>
|
||||
<p>Proportion: {{ c.proportion|floatformat:3 }}</p>
|
||||
<p>Silhouette: {{ c.silhouette_score|floatformat:4|default:"-" }}</p>
|
||||
<h4 style="margin-top:0.75rem;font-size:0.9rem;">Top Distinguishing Features</h4>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Feature</th><th>Score</th><th>Mean</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for f in c.features.all|slice:":5" %}
|
||||
<tr>
|
||||
<td>{{ f.feature_name }}</td>
|
||||
<td>{{ f.distinguishing_score|floatformat:3|default:"-" }}</td>
|
||||
<td>{{ f.mean|floatformat:3|default:"-" }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="3">No features</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<a href="{% url 'analysis:cluster_detail' run.id c.cluster_label %}" class="btn btn-primary" style="margin-top:0.75rem;">Detail</a>
|
||||
</div>
|
||||
{% empty %}
|
||||
<div class="card">
|
||||
<div class="empty-state"><p>No clusters found.</p></div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
|
||||
{% if noise %}
|
||||
<div class="card">
|
||||
<h3>Noise (Cluster -1)</h3>
|
||||
<p>{{ noise.size }} entities classified as noise</p>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<script>
|
||||
// Pure Canvas scatter plot — no Chart.js needed (offline compatible)
|
||||
const scatterData = {{ scatter_data_json|safe }};
|
||||
const colors = ['#4361ee','#f72585','#7209b7','#4cc9f0','#f8961e','#43aa8b','#f94144','#577590','#e9c46a','#9b5de5'];
|
||||
|
||||
function drawScatter(canvasId, data, xLabel, yLabel) {
|
||||
const canvas = document.getElementById(canvasId);
|
||||
if (!canvas || data.length === 0) return;
|
||||
const ctx = canvas.getContext('2d');
|
||||
const W = canvas.parentElement.clientWidth || 600;
|
||||
const H = 400;
|
||||
canvas.width = W; canvas.height = H;
|
||||
const pad = { top: 30, bottom: 50, left: 60, right: 30 };
|
||||
const plotW = W - pad.left - pad.right;
|
||||
const plotH = H - pad.top - pad.bottom;
|
||||
|
||||
// Compute bounds
|
||||
const xs = data.map(d => d.x); const ys = data.map(d => d.y);
|
||||
const xMin = Math.min(...xs), xMax = Math.max(...xs);
|
||||
const yMin = Math.min(...ys), yMax = Math.max(...ys);
|
||||
const xRange = xMax - xMin || 1; const yRange = yMax - yMin || 1;
|
||||
|
||||
const toX = v => pad.left + (v - xMin) / xRange * plotW;
|
||||
const toY = v => pad.top + plotH - (v - yMin) / yRange * plotH;
|
||||
|
||||
// Clear
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
|
||||
// Grid lines
|
||||
ctx.strokeStyle = '#eee'; ctx.lineWidth = 1;
|
||||
for (let i = 0; i <= 5; i++) {
|
||||
const y = pad.top + i * plotH / 5;
|
||||
ctx.beginPath(); ctx.moveTo(pad.left, y); ctx.lineTo(W - pad.right, y); ctx.stroke();
|
||||
}
|
||||
|
||||
// Axes
|
||||
ctx.strokeStyle = '#333'; ctx.lineWidth = 2;
|
||||
ctx.beginPath(); ctx.moveTo(pad.left, pad.top); ctx.lineTo(pad.left, H - pad.bottom); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(pad.left, H - pad.bottom); ctx.lineTo(W - pad.right, H - pad.bottom); ctx.stroke();
|
||||
|
||||
// Labels
|
||||
ctx.fillStyle = '#666'; ctx.font = '13px sans-serif'; ctx.textAlign = 'center';
|
||||
ctx.fillText(xLabel, W / 2, H - 5);
|
||||
ctx.save(); ctx.translate(15, H / 2); ctx.rotate(-Math.PI / 2); ctx.fillText(yLabel, 0, 0); ctx.restore();
|
||||
|
||||
// Points
|
||||
data.forEach(d => {
|
||||
const cx = toX(d.x), cy = toY(d.y);
|
||||
const color = d.label === -1 ? 'rgba(150,150,150,0.5)' : colors[Math.abs(d.label) % colors.length];
|
||||
const r = d.label === -1 ? 2 : 4;
|
||||
ctx.beginPath(); ctx.arc(cx, cy, r, 0, Math.PI * 2);
|
||||
ctx.fillStyle = color; ctx.fill();
|
||||
});
|
||||
|
||||
// Tooltip on hover
|
||||
canvas.onmousemove = function(e) {
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
const mx = e.clientX - rect.left, my = e.clientY - rect.top;
|
||||
const hit = data.find(d => Math.abs(toX(d.x) - mx) < 6 && Math.abs(toY(d.y) - my) < 6);
|
||||
if (hit) {
|
||||
canvas.title = `${hit.entity} (${hit.label === -1 ? 'Noise' : 'Cluster ' + hit.label})`;
|
||||
}
|
||||
};
|
||||
|
||||
// Legend
|
||||
const uniqueLabels = [...new Set(data.map(d => d.label))];
|
||||
let lx = W - pad.right - 120, ly = pad.top + 5;
|
||||
ctx.font = '11px sans-serif'; ctx.textAlign = 'left';
|
||||
uniqueLabels.forEach(label => {
|
||||
const color = label === -1 ? 'rgba(150,150,150,0.8)' : colors[Math.abs(label) % colors.length];
|
||||
ctx.fillStyle = color; ctx.fillRect(lx, ly, 10, 10);
|
||||
ctx.fillStyle = '#333'; ctx.fillText(label === -1 ? 'Noise' : `Cluster ${label}`, lx + 15, ly + 10);
|
||||
ly += 18;
|
||||
});
|
||||
}
|
||||
|
||||
drawScatter('scatterChart', scatterData, 'PCA-1', 'PCA-2');
|
||||
|
||||
{% if geo_count > 0 %}
|
||||
const geoData = {{ geo_data_json|safe }};
|
||||
drawScatter('geoChart', geoData, 'Longitude', 'Latitude');
|
||||
{% endif %}
|
||||
|
||||
// ── Cluster size horizontal bar chart ──
|
||||
const clusterSizes = {{ cluster_sizes_json|safe }};
|
||||
const topFeaturesAll = {{ top_features_json|safe }};
|
||||
const chartColors = ['#4361ee','#f72585','#7209b7','#4cc9f0','#f8961e','#43aa8b','#f94144','#577590','#e9c46a','#9b5de5'];
|
||||
|
||||
if (clusterSizes.length > 0) {
|
||||
// Cluster size bar chart
|
||||
(function() {
|
||||
const canvas = document.getElementById('clusterSizesChart');
|
||||
if (!canvas) return;
|
||||
const ctx = canvas.getContext('2d');
|
||||
const W = canvas.parentElement.clientWidth || 600;
|
||||
const H = Math.max(200, clusterSizes.length * 40 + 60);
|
||||
canvas.width = W; canvas.height = H;
|
||||
const pad = { top: 20, bottom: 30, left: 80, right: 30 };
|
||||
const plotW = W - pad.left - pad.right;
|
||||
const plotH = H - pad.top - pad.bottom;
|
||||
const maxSize = Math.max(...clusterSizes.map(d => d.size));
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
clusterSizes.forEach((d, i) => {
|
||||
const barH = Math.min(30, plotH / clusterSizes.length - 4);
|
||||
const y = pad.top + i * (barH + 4);
|
||||
const bw = (d.size / maxSize) * plotW;
|
||||
ctx.fillStyle = chartColors[i % chartColors.length];
|
||||
ctx.fillRect(pad.left, y, bw, barH);
|
||||
ctx.fillStyle = '#333';
|
||||
ctx.font = '12px sans-serif';
|
||||
ctx.textAlign = 'right';
|
||||
ctx.fillText('#' + d.label, pad.left - 5, y + barH / 2 + 4);
|
||||
ctx.textAlign = 'left';
|
||||
ctx.fillStyle = '#666';
|
||||
ctx.fillText(d.size + ' (' + (d.size / clusterSizes.reduce((a,b) => a+b.size, 0) * 100).toFixed(1) + '%)', pad.left + bw + 5, y + barH / 2 + 4);
|
||||
});
|
||||
})();
|
||||
|
||||
// Silhouette score bar chart
|
||||
(function() {
|
||||
const canvas = document.getElementById('silhouetteChart');
|
||||
if (!canvas) return;
|
||||
const silData = clusterSizes.filter(d => d.silhouette !== null && d.silhouette !== undefined);
|
||||
if (silData.length === 0) {
|
||||
canvas.parentElement.style.display = 'none';
|
||||
return;
|
||||
}
|
||||
const ctx = canvas.getContext('2d');
|
||||
const W = canvas.parentElement.clientWidth || 600;
|
||||
const H = Math.max(200, silData.length * 40 + 60);
|
||||
canvas.width = W; canvas.height = H;
|
||||
const pad = { top: 20, bottom: 30, left: 80, right: 40 };
|
||||
const plotW = W - pad.left - pad.right;
|
||||
const plotH = H - pad.top - pad.bottom;
|
||||
const maxScore = Math.max(1, ...silData.map(d => Math.abs(d.silhouette)));
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
silData.forEach((d, i) => {
|
||||
const barH = Math.min(30, plotH / silData.length - 4);
|
||||
const y = pad.top + i * (barH + 4);
|
||||
const bw = (d.silhouette / maxScore) * plotW * 0.5;
|
||||
const x = pad.left + plotW / 2;
|
||||
ctx.fillStyle = d.silhouette >= 0 ? '#43aa8b' : '#f94144';
|
||||
ctx.fillRect(x, y, d.silhouette >= 0 ? bw : -bw, barH);
|
||||
ctx.fillStyle = '#333';
|
||||
ctx.font = '12px sans-serif';
|
||||
ctx.textAlign = 'right';
|
||||
ctx.fillText('#' + d.label, pad.left - 5, y + barH / 2 + 4);
|
||||
ctx.textAlign = 'left';
|
||||
ctx.fillStyle = '#666';
|
||||
ctx.fillText(d.silhouette.toFixed(4), d.silhouette >= 0 ? x + bw + 5 : x - bw - 40, y + barH / 2 + 4);
|
||||
});
|
||||
// Zero line
|
||||
ctx.strokeStyle = '#999'; ctx.lineWidth = 1;
|
||||
ctx.beginPath(); ctx.moveTo(pad.left + plotW / 2, pad.top); ctx.lineTo(pad.left + plotW / 2, H - pad.bottom); ctx.stroke();
|
||||
ctx.fillStyle = '#999'; ctx.font = '11px sans-serif'; ctx.textAlign = 'center';
|
||||
ctx.fillText('0', pad.left + plotW / 2, H - 5);
|
||||
ctx.fillText('+1', W - pad.right, H - 5);
|
||||
ctx.fillText('-1', pad.left, H - 5);
|
||||
})();
|
||||
|
||||
// Top features per cluster horizontal bar charts
|
||||
(function() {
|
||||
const container = document.getElementById('featureChartsContainer');
|
||||
if (!container) return;
|
||||
clusterSizes.forEach(d => {
|
||||
const label = d.label;
|
||||
const features = topFeaturesAll[String(label)] || [];
|
||||
if (features.length === 0) return;
|
||||
const card = document.createElement('div');
|
||||
card.style.marginBottom = '1.5rem';
|
||||
card.innerHTML = '<h4 style="margin-bottom:0.5rem;">Cluster #' + label + ' - Top Features</h4><canvas class="feature-chart" data-label="' + label + '" style="width:100%;height:' + Math.max(120, features.length * 28 + 30) + 'px;"></canvas>';
|
||||
container.appendChild(card);
|
||||
const canvas = card.querySelector('canvas');
|
||||
if (!canvas) return;
|
||||
const ctx = canvas.getContext('2d');
|
||||
const W2 = canvas.parentElement.clientWidth || 600;
|
||||
const H2 = Math.max(120, features.length * 28 + 30);
|
||||
canvas.width = W2; canvas.height = H2;
|
||||
const pad2 = { top: 10, bottom: 20, left: 100, right: 60 };
|
||||
const plotW2 = W2 - pad2.left - pad2.right;
|
||||
const plotH2 = H2 - pad2.top - pad2.bottom;
|
||||
const maxScore = Math.max(0.01, ...features.map(f => Math.abs(f.score || 0)));
|
||||
ctx.clearRect(0, 0, W2, H2);
|
||||
features.forEach((f, i) => {
|
||||
const barH = Math.min(22, plotH2 / features.length - 3);
|
||||
const y = pad2.top + i * (barH + 3);
|
||||
const score = f.score || 0;
|
||||
const bw = (score / maxScore) * plotW2;
|
||||
ctx.fillStyle = chartColors[label % chartColors.length];
|
||||
ctx.fillRect(pad2.left, y, bw, barH);
|
||||
ctx.fillStyle = '#333';
|
||||
ctx.font = '11px sans-serif';
|
||||
ctx.textAlign = 'right';
|
||||
const name = f.feature_name.length > 18 ? f.feature_name.slice(0, 16) + '..' : f.feature_name;
|
||||
ctx.fillText(name, pad2.left - 5, y + barH / 2 + 4);
|
||||
ctx.textAlign = 'left';
|
||||
ctx.fillStyle = '#666';
|
||||
ctx.fillText(score.toFixed(3), pad2.left + bw + 5, y + barH / 2 + 4);
|
||||
});
|
||||
});
|
||||
})();
|
||||
}
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,54 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}首页 - 天璇{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>TLS Flow Analysis Dashboard</h2>
|
||||
<p style="color:#666;margin-bottom:1rem;">Upload CSV flow data, detect entities, cluster behavior profiles, and visualize results.</p>
|
||||
</div>
|
||||
|
||||
{% if runs %}
|
||||
<div class="card">
|
||||
<h2>Recent Analysis Runs</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>Started</th>
|
||||
<th>Status</th>
|
||||
<th>Entity Column</th>
|
||||
<th>Entities</th>
|
||||
<th>Clusters</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td>#{{ run.id }}</td>
|
||||
<td>{{ run.created_at|date:"Y-m-d H:i" }}</td>
|
||||
<td>
|
||||
{% if run.status == 'completed' %}
|
||||
<span class="badge badge-success">Completed</span>
|
||||
{% elif run.status == 'failed' %}
|
||||
<span class="badge badge-danger">Failed</span>
|
||||
{% else %}
|
||||
<span class="badge badge-info">{{ run.get_status_display }}</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td>{{ run.entity_column|default:"-" }}</td>
|
||||
<td>{{ run.entity_count|default:"-" }}</td>
|
||||
<td>{{ run.cluster_count|default:"-" }}</td>
|
||||
<td><a href="{% url 'analysis:run_detail' run.id %}" class="btn btn-primary">View</a></td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="card empty-state">
|
||||
<div style="font-size:3rem;">📊</div>
|
||||
<p>No analysis runs yet. Use MCP tools to load data and start analysis.</p>
|
||||
<p style="font-size:0.9rem;color:#bbb;margin-top:0.5rem;">Call <code>load_data()</code> via MCP to begin.</p>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,74 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}Entity: {{ entity.entity_value }} - TLS Analyzer{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<a href="javascript:history.back()" style="color:#4361ee;text-decoration:none;font-size:0.9rem;">← Back</a>
|
||||
<h2 style="margin-top:0.5rem;">Entity: <code>{{ entity.entity_value }}</code></h2>
|
||||
<p>Run #{{ run.id }} | Cluster: <span class="badge badge-info">#{{ entity.cluster_label|default:"unassigned" }}</span></p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Aggregated Features</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Feature</th>
|
||||
<th>Value</th>
|
||||
<th>Cluster Mean</th>
|
||||
<th>Deviation (Z-score)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for col, dev in deviations.items %}
|
||||
<tr>
|
||||
<td><code>{{ col }}</code></td>
|
||||
<td>{{ dev.value|floatformat:3 }}</td>
|
||||
<td>{{ dev.cluster_mean|floatformat:3 }}</td>
|
||||
<td style="color:{% if dev.zscore > 2 %}#f72585{% elif dev.zscore < -2 %}#4361ee{% else %}#666{% endif %};">
|
||||
{{ dev.zscore|floatformat:2 }}
|
||||
</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="4">No feature data available.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{% if entity.embedding_x is not None and entity.embedding_y is not None %}
|
||||
<div class="card">
|
||||
<h2>PCA Position</h2>
|
||||
<p>X: {{ entity.embedding_x|floatformat:3 }}, Y: {{ entity.embedding_y|floatformat:3 }}</p>
|
||||
<canvas id="entityScatter" class="scatter-canvas" height="300"></canvas>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const canvas = document.getElementById('entityScatter');
|
||||
if (canvas) {
|
||||
const ctx = canvas.getContext('2d');
|
||||
const W = canvas.parentElement.clientWidth || 400, H = 300;
|
||||
canvas.width = W; canvas.height = H;
|
||||
const pad = { top: 30, bottom: 40, left: 50, right: 20 };
|
||||
const x = {{ entity.embedding_x }}, y = {{ entity.embedding_y }};
|
||||
const margin = 0.2; // 20% padding around the point
|
||||
const xRange = Math.abs(x) * margin * 2 || 1;
|
||||
const yRange = Math.abs(y) * margin * 2 || 1;
|
||||
const xMin = x - xRange, xMax = x + xRange;
|
||||
const yMin = y - yRange, yMax = y + yRange;
|
||||
|
||||
const toX = v => pad.left + (v - xMin) / (xMax - xMin) * (W - pad.left - pad.right);
|
||||
const toY = v => pad.top + H - pad.bottom - (v - yMin) / (yMax - yMin) * (H - pad.top - pad.bottom);
|
||||
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
ctx.strokeStyle = '#ddd'; ctx.lineWidth = 1;
|
||||
ctx.beginPath(); ctx.moveTo(pad.left, toY(0)); ctx.lineTo(W - pad.right, toY(0)); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(toX(0), pad.top); ctx.lineTo(toX(0), H - pad.bottom); ctx.stroke();
|
||||
|
||||
ctx.fillStyle = '#f72585';
|
||||
ctx.beginPath(); ctx.arc(toX(x), toY(y), 8, 0, Math.PI * 2); ctx.fill();
|
||||
ctx.fillStyle = '#fff'; ctx.font = 'bold 10px sans-serif'; ctx.textAlign = 'center';
|
||||
ctx.fillText('●', toX(x), toY(y) + 3);
|
||||
}
|
||||
</script>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,56 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}Run #{{ run.id }} - TLS Analyzer{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>Run #{{ run.id }} — {{ run.created_at|date:"Y-m-d H:i" }}</h2>
|
||||
<p>Status: <span class="badge {% if run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-info{% endif %}">{{ run.get_status_display }}</span></p>
|
||||
<p>CSV: <code>{{ run.csv_glob }}</code></p>
|
||||
<p>Entity Column: {{ run.entity_column|default:"(not detected)" }}</p>
|
||||
</div>
|
||||
|
||||
<div class="grid-3">
|
||||
<div class="card" style="text-align:center;">
|
||||
<div class="stat">{{ run.total_flows|default:"0" }}</div>
|
||||
<div class="stat-label">Total Flows</div>
|
||||
</div>
|
||||
<div class="card" style="text-align:center;">
|
||||
<div class="stat">{{ run.entity_count|default:"0" }}</div>
|
||||
<div class="stat-label">Entities</div>
|
||||
</div>
|
||||
<div class="card" style="text-align:center;">
|
||||
<div class="stat">{{ run.cluster_count|default:"0" }}</div>
|
||||
<div class="stat-label">Clusters</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Clusters</h2>
|
||||
<a href="{% url 'analysis:cluster_overview' run.id %}" class="btn btn-primary" style="margin-bottom:1rem;">View Cluster Overview</a>
|
||||
{% if clusters %}
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Cluster</th>
|
||||
<th>Size</th>
|
||||
<th>Proportion</th>
|
||||
<th>Silhouette</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for c in clusters %}
|
||||
<tr>
|
||||
<td><span class="badge badge-info">#{{ c.cluster_label }}</span></td>
|
||||
<td>{{ c.size }}</td>
|
||||
<td>{{ c.proportion|floatformat:2 }}</td>
|
||||
<td>{{ c.silhouette_score|floatformat:4|default:"-" }}</td>
|
||||
<td><a href="{% url 'analysis:cluster_detail' run.id c.cluster_label %}" class="btn btn-primary">Detail</a></td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<div class="empty-state"><p>No clusters yet.</p></div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,42 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}Analysis Runs - TLS Analyzer{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>All Analysis Runs</h2>
|
||||
{% if runs %}
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>Started</th>
|
||||
<th>Status</th>
|
||||
<th>CSV Glob</th>
|
||||
<th>Entities</th>
|
||||
<th>Clusters</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td>#{{ run.id }}</td>
|
||||
<td>{{ run.created_at|date:"Y-m-d H:i" }}</td>
|
||||
<td><span class="badge {% if run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-info{% endif %}">{{ run.get_status_display }}</span></td>
|
||||
<td style="max-width:300px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
|
||||
<td>{{ run.entity_count|default:"-" }}</td>
|
||||
<td>{{ run.cluster_count|default:"-" }}</td>
|
||||
<td><a href="{% url 'analysis:run_detail' run.id %}" class="btn btn-primary">Detail</a></td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="pagination">
|
||||
{% if page > 1 %}<a href="?page={{ page|add:-1 }}">← Prev</a>{% endif %}
|
||||
<span class="active">{{ page }}</span>
|
||||
{% if page < total_pages %}<a href="?page={{ page|add:1 }}">Next →</a>{% endif %}
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="empty-state"><p>No runs yet.</p></div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,55 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-hans">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{% block title %}天璇{% endblock %}</title>
|
||||
<link rel="icon" type="image/svg+xml" href="/static/tianxuan/favicon.svg">
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f7fa; color: #1a1a2e; }
|
||||
nav { background: #1a1a2e; padding: 1rem 2rem; display: flex; align-items: center; gap: 2rem; }
|
||||
nav a { color: #e0e0e0; text-decoration: none; font-size: 0.95rem; }
|
||||
nav a:hover { color: #fff; }
|
||||
nav .brand { color: #fff; font-size: 1.2rem; font-weight: 700; }
|
||||
.container { max-width: 1200px; margin: 0 auto; padding: 2rem; }
|
||||
.card { background: #fff; border-radius: 8px; padding: 1.5rem; margin-bottom: 1.5rem; box-shadow: 0 1px 3px rgba(0,0,0,0.1); }
|
||||
.card h2 { margin-bottom: 1rem; color: #1a1a2e; font-size: 1.1rem; }
|
||||
table { width: 100%; border-collapse: collapse; }
|
||||
th, td { text-align: left; padding: 0.75rem 0.5rem; border-bottom: 1px solid #eee; font-size: 0.9rem; }
|
||||
th { font-weight: 600; color: #666; }
|
||||
.badge { display: inline-block; padding: 0.2rem 0.6rem; border-radius: 12px; font-size: 0.8rem; font-weight: 600; }
|
||||
.badge-success { background: #d4edda; color: #155724; }
|
||||
.badge-warning { background: #fff3cd; color: #856404; }
|
||||
.badge-info { background: #d1ecf1; color: #0c5460; }
|
||||
.badge-danger { background: #f8d7da; color: #721c24; }
|
||||
.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 1.5rem; }
|
||||
.grid-3 { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 1.5rem; }
|
||||
.stat { font-size: 2rem; font-weight: 700; color: #1a1a2e; }
|
||||
.stat-label { font-size: 0.85rem; color: #666; margin-top: 0.25rem; }
|
||||
.btn { display: inline-block; padding: 0.5rem 1rem; border-radius: 6px; text-decoration: none; font-size: 0.9rem; cursor: pointer; border: none; }
|
||||
.btn-primary { background: #4361ee; color: #fff; }
|
||||
.btn-primary:hover { background: #3a56d4; }
|
||||
.scatter-canvas { max-height: 500px; }
|
||||
.pagination { display: flex; justify-content: center; gap: 0.5rem; margin-top: 1.5rem; }
|
||||
.pagination a, .pagination span { padding: 0.4rem 0.8rem; border-radius: 4px; text-decoration: none; color: #4361ee; background: #fff; border: 1px solid #ddd; }
|
||||
.pagination .active { background: #4361ee; color: #fff; border-color: #4361ee; }
|
||||
.empty-state { text-align: center; padding: 3rem; color: #999; }
|
||||
.empty-state p { font-size: 1rem; margin-top: 0.5rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<nav>
|
||||
<span class="brand">天璇</span>
|
||||
<a href="{% url 'analysis:dashboard' %}">首页</a>
|
||||
<a href="{% url 'analysis:upload' %}">上传数据</a>
|
||||
<a href="{% url 'analysis:manual' %}">手动分析</a>
|
||||
<a href="{% url 'analysis:auto' %}">LLM分析</a>
|
||||
<a href="{% url 'analysis:run_list' %}">运行记录</a>
|
||||
<a href="{% url 'analysis:globe' %}">地球</a>
|
||||
</nav>
|
||||
<div class="container">
|
||||
{% block content %}{% endblock %}
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,108 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}LLM 自动分析 - 天璇{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>LLM 自动分析</h2>
|
||||
<p style="color:#666;">LLM 将自动编排工具链完成：加载 → 检测实体 → 聚合 → 聚类 → 特征提取</p>
|
||||
{% if not llm_configured %}
|
||||
<div style="background:#fff3cd;padding:1rem;border-radius:6px;margin:1rem 0;">
|
||||
⚠️ LLM 尚未配置。请先在 <a href="/config/">配置页面</a> 设置 base_url 和 API Key。
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>选择数据集</h3>
|
||||
{% if runs %}
|
||||
<table>
|
||||
<thead><tr><th></th><th>ID</th><th>文件</th><th>实体列</th><th>状态</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td><input type="radio" name="auto_run_id" value="{{ run.id }}"></td>
|
||||
<td>#{{ run.id }}</td>
|
||||
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
|
||||
<td>{{ run.entity_column|default:"-" }}</td>
|
||||
<td><span class="badge badge-success">{{ run.status }}</span></td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="5" style="text-align:center;">暂无数据。先去 <a href="/upload/">上传 CSV</a>。</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<button class="btn btn-primary" style="margin-top:1rem;" onclick="startAuto()" {% if not llm_configured %}disabled{% endif %}>开始自动分析</button>
|
||||
{% endif %}
|
||||
<div id="autoProgress" style="display:none;margin-top:1rem;">
|
||||
<p>LLM 编排中... <span id="autoStatus"></span></p>
|
||||
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
|
||||
<div id="autoBar" style="background:#4361ee;height:100%;width:0%;"></div>
|
||||
</div>
|
||||
<button id="cancelBtn" class="btn" style="background:#f72585;color:white;display:none;" onclick="cancelAnalysis()">取消分析</button>
|
||||
<div id="liveLog" style="background:#1a1a2e;color:#e0e0e0;padding:1rem;border-radius:6px;margin-top:1rem;max-height:300px;overflow:auto;font-size:0.8rem;display:none;"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
let logPoll = null;
|
||||
let statusPoll = null;
|
||||
|
||||
async function startAuto() {
|
||||
const selected = document.querySelector('input[name="auto_run_id"]:checked');
|
||||
if (!selected) { alert('请先选择一个数据集'); return; }
|
||||
document.getElementById('autoProgress').style.display = 'block';
|
||||
document.getElementById('cancelBtn').style.display = 'inline-block';
|
||||
document.getElementById('liveLog').style.display = 'block';
|
||||
document.getElementById('liveLog').innerHTML = '';
|
||||
const resp = await fetch('/analyze/llm/', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', 'X-CSRFToken': '{{ csrf_token }}' },
|
||||
body: JSON.stringify({ run_id: parseInt(selected.value), entity_column: '' }),
|
||||
});
|
||||
const data = await resp.json();
|
||||
const runId = selected.value;
|
||||
|
||||
statusPoll = setInterval(async () => {
|
||||
const r = await fetch(`/runs/${runId}/status/`);
|
||||
const s = await r.json();
|
||||
const pct = s.progress_pct || 0;
|
||||
const msg = s.progress_msg || s.status;
|
||||
document.getElementById('autoStatus').textContent = `${msg} (${pct}%)`;
|
||||
document.getElementById('autoBar').style.width = pct + '%';
|
||||
if (s.run_log) {
|
||||
document.getElementById('liveLog').innerHTML = s.run_log.split('\n').map(l => l.trim()).filter(l => l).join('\n') + '\n';
|
||||
document.getElementById('liveLog').scrollTop = document.getElementById('liveLog').scrollHeight;
|
||||
}
|
||||
if (s.status === 'completed') { clearInterval(statusPoll); clearInterval(logPoll); document.getElementById('autoStatus').textContent = '分析完成！'; }
|
||||
if (s.status === 'failed') { clearInterval(statusPoll); clearInterval(logPoll); document.getElementById('autoStatus').textContent = '失败: ' + (s.error_message || ''); }
|
||||
}, 1000);
|
||||
|
||||
// Poll live logs every 3 seconds
|
||||
let lastPos = 0;
|
||||
logPoll = setInterval(async () => {
|
||||
try {
|
||||
const r = await fetch(`/logs/?pos=${lastPos}`);
|
||||
const txt = await r.text();
|
||||
const logDiv = document.getElementById('liveLog');
|
||||
// Check if response is a number (position marker) followed by new content
|
||||
const parts = txt.split('\n', 2);
|
||||
if (parts.length >= 2) {
|
||||
const newPos = parseInt(parts[0], 10);
|
||||
if (!isNaN(newPos)) { lastPos = newPos; }
|
||||
if (parts[1]) {
|
||||
logDiv.innerHTML += parts[1] + '\n';
|
||||
logDiv.scrollTop = logDiv.scrollHeight;
|
||||
}
|
||||
}
|
||||
} catch (e) { /* ignore fetch errors during polling */ }
|
||||
}, 3000);
|
||||
}
|
||||
|
||||
function cancelAnalysis() {
|
||||
if (statusPoll) { clearInterval(statusPoll); statusPoll = null; }
|
||||
if (logPoll) { clearInterval(logPoll); logPoll = null; }
|
||||
document.getElementById('autoStatus').textContent = '已取消';
|
||||
document.getElementById('cancelBtn').style.display = 'none';
|
||||
document.getElementById('liveLog').innerHTML += '\n--- 分析已取消 ---\n';
|
||||
}
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,174 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}配置 - TLS Analyzer{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>系统配置</h2>
|
||||
<p style="color:#666;margin-bottom:1rem;">修改配置后点击「保存配置」持久化到 <code>config/config.yaml</code>。</p>
|
||||
</div>
|
||||
|
||||
<form id="config-form" method="post">
|
||||
{% csrf_token %}
|
||||
|
||||
{# ── Server ── #}
|
||||
<div class="card">
|
||||
<h2>服务器</h2>
|
||||
<div class="grid-3">
|
||||
<div class="form-group">
|
||||
<label for="server_host">Host</label>
|
||||
<input type="text" id="server_host" name="server_host" value="{{ config.server.host }}" class="form-input">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="server_port">Port</label>
|
||||
<input type="number" id="server_port" name="server_port" value="{{ config.server.port }}" class="form-input">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="checkbox-label">
|
||||
<input type="checkbox" name="server_debug" value="true" {% if config.server.debug %}checked{% endif %}>
|
||||
Debug 模式
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Data ── #}
|
||||
<div class="card">
|
||||
<h2>数据</h2>
|
||||
<div class="grid-2">
|
||||
<div class="form-group">
|
||||
<label class="checkbox-label">
|
||||
<input type="checkbox" name="data_schema_strict" value="true" {% if config.data.schema_strict %}checked{% endif %}>
|
||||
严格 Schema 校验
|
||||
</label>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label class="checkbox-label">
|
||||
<input type="checkbox" name="data_recursive" value="true" {% if config.data.recursive %}checked{% endif %}>
|
||||
递归扫描
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Clustering ── #}
|
||||
<div class="card">
|
||||
<h2>聚类</h2>
|
||||
<div class="grid-2">
|
||||
<div class="form-group">
|
||||
<label for="clustering_algorithm">算法</label>
|
||||
<select id="clustering_algorithm" name="clustering_algorithm" class="form-input">
|
||||
<option value="hdbscan" {% if config.clustering.algorithm == 'hdbscan' %}selected{% endif %}>HDBSCAN</option>
|
||||
<option value="kmeans" {% if config.clustering.algorithm == 'kmeans' %}selected{% endif %}>K-Means</option>
|
||||
<option value="dbscan" {% if config.clustering.algorithm == 'dbscan' %}selected{% endif %}>DBSCAN</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="clustering_min_cluster_size">最小簇大小</label>
|
||||
<input type="number" id="clustering_min_cluster_size" name="clustering_min_cluster_size"
|
||||
value="{{ config.clustering.min_cluster_size }}" min="2" class="form-input">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── LLM ── #}
|
||||
<div class="card">
|
||||
<h2>LLM 连接</h2>
|
||||
<div class="form-group">
|
||||
<label class="checkbox-label">
|
||||
<input type="checkbox" name="llm_enabled" value="true" {% if config.llm.enabled %}checked{% endif %}>
|
||||
启用 LLM
|
||||
</label>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="llm_base_url">Base URL</label>
|
||||
<input type="text" id="llm_base_url" name="llm_base_url" value="{{ config.llm.base_url }}"
|
||||
placeholder="https://api.openai.com/v1" class="form-input">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="llm_api_key">API Key</label>
|
||||
<input type="password" id="llm_api_key" name="llm_api_key" value="{{ config.llm.api_key }}"
|
||||
placeholder="sk-..." class="form-input">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="llm_model">Model</label>
|
||||
<input type="text" id="llm_model" name="llm_model" value="{{ config.llm.model }}"
|
||||
placeholder="gpt-4" class="form-input">
|
||||
</div>
|
||||
|
||||
{# LLM connectivity test button + result area #}
|
||||
<div style="margin-top:1rem; display:flex; align-items:center; gap:1rem; flex-wrap:wrap;">
|
||||
<button type="button" id="btn-llm-test" class="btn btn-primary" onclick="testLLM()">测试连通性</button>
|
||||
<span id="llm-test-result" style="font-size:0.9rem;"></span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Save button ── #}
|
||||
<div class="card" style="text-align:right;">
|
||||
<button type="submit" class="btn btn-primary" style="padding:0.6rem 2rem;">保存配置</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
{% if saved %}
|
||||
<script>
|
||||
// Flash-style feedback: briefly highlight the message
|
||||
const el = document.createElement('div');
|
||||
el.style.cssText = 'position:fixed;top:1rem;right:1rem;background:#155724;color:#fff;padding:0.8rem 1.5rem;border-radius:6px;font-size:0.9rem;z-index:999;box-shadow:0 2px 8px rgba(0,0,0,0.2);';
|
||||
el.textContent = '✅ 配置已保存';
|
||||
document.body.appendChild(el);
|
||||
setTimeout(() => el.remove(), 3000);
|
||||
</script>
|
||||
{% endif %}
|
||||
|
||||
<script>
|
||||
async function testLLM() {
|
||||
const btn = document.getElementById('btn-llm-test');
|
||||
const result = document.getElementById('llm-test-result');
|
||||
btn.disabled = true;
|
||||
btn.textContent = '测试中...';
|
||||
result.textContent = '';
|
||||
result.style.color = '';
|
||||
|
||||
const payload = {
|
||||
base_url: document.getElementById('llm_base_url').value || '',
|
||||
api_key: document.getElementById('llm_api_key').value || '',
|
||||
model: document.getElementById('llm_model').value || '',
|
||||
};
|
||||
|
||||
try {
|
||||
const resp = await fetch('/api/llm/test/', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', 'X-CSRFToken': getCSRF() },
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
const data = await resp.json();
|
||||
if (data.success) {
|
||||
result.style.color = '#155724';
|
||||
result.textContent = `✅ ${data.message} (${data.latency_ms}ms)`;
|
||||
} else {
|
||||
result.style.color = '#721c24';
|
||||
result.textContent = `❌ ${data.message}`;
|
||||
}
|
||||
} catch (err) {
|
||||
result.style.color = '#721c24';
|
||||
result.textContent = `❌ 请求失败: ${err.message}`;
|
||||
} finally {
|
||||
btn.disabled = false;
|
||||
btn.textContent = '测试连通性';
|
||||
}
|
||||
}
|
||||
|
||||
function getCSRF() {
|
||||
return document.querySelector('[name=csrfmiddlewaretoken]')?.value || '';
|
||||
}
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.form-group { margin-bottom: 1rem; }
|
||||
.form-group label { display: block; font-size: 0.85rem; color: #555; margin-bottom: 0.3rem; font-weight: 600; }
|
||||
.form-input { width: 100%; padding: 0.5rem 0.75rem; border: 1px solid #ddd; border-radius: 6px; font-size: 0.9rem; }
|
||||
.form-input:focus { outline: none; border-color: #4361ee; box-shadow: 0 0 0 2px rgba(67,97,238,0.15); }
|
||||
.checkbox-label { display: flex; align-items: center; gap: 0.5rem; font-size: 0.9rem; color: #333; cursor: pointer; }
|
||||
.checkbox-label input[type="checkbox"] { width: 1rem; height: 1rem; cursor: pointer; }
|
||||
.grid-2 .form-group, .grid-3 .form-group { min-width: 0; }
|
||||
.card h2 { font-size: 1rem; border-bottom: 1px solid #eee; padding-bottom: 0.5rem; margin-bottom: 1rem; }
|
||||
</style>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,235 @@
|
||||
{% extends 'base.html' %}
|
||||
{% load static %}
|
||||
{% block title %}TLS 流量地球 - 天璇{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>TLS 流量地球可视化</h2>
|
||||
<p style="color:#666;">三维地球上的流量弧线，显示源和目标之间的 TLS 连接</p>
|
||||
<p style="font-size:0.8rem;color:#999;">
|
||||
<span id="arcCount">0</span> 条流量 |
|
||||
颜色: <span style="color:#4361ee;">TLSv1.3</span>
|
||||
<span style="color:#f72585;">TLSv1.2</span>
|
||||
<span style="color:#7209b7;">TLSv1.1/1.0</span>
|
||||
<span style="color:#4cc9f0;">其他</span>
|
||||
<span style="color:#888;margin-left:0.5rem;">| 支持 hex 格式 (03 03 → TLSv1.2)</span>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{% if all_runs %}
|
||||
<div class="card" style="margin-bottom:1rem;">
|
||||
<h3>选择数据运行</h3>
|
||||
<form method="get" action="{% url 'analysis:globe' %}" id="runSelectForm">
|
||||
<div style="display:flex;flex-wrap:wrap;gap:0.5rem;align-items:center;">
|
||||
{% for r in all_runs %}
|
||||
<label style="font-size:0.85rem;cursor:pointer;">
|
||||
<input type="checkbox" name="runs" value="{{ r.id }}"
|
||||
{% if r.id in selected_ids %}checked{% endif %}>
|
||||
Run #{{ r.id }}
|
||||
</label>
|
||||
{% endfor %}
|
||||
<button type="submit" class="btn btn-primary" style="margin-left:auto;">更新地球</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<div class="card" style="margin-bottom:1rem;padding:0.8rem;">
|
||||
<div style="display:flex;gap:1.5rem;align-items:center;flex-wrap:wrap;">
|
||||
<label style="font-size:0.85rem;cursor:pointer;">
|
||||
<input type="checkbox" id="toggleGrid" checked onchange="gridGroup.visible = this.checked;">
|
||||
经纬网格
|
||||
</label>
|
||||
<label style="font-size:0.85rem;cursor:pointer;">
|
||||
<input type="checkbox" id="toggleBorders" checked onchange="borderGroup.visible = this.checked;">
|
||||
国境线
|
||||
</label>
|
||||
<span style="font-size:0.85rem;color:#666;">鼠标拖拽旋转 · 滚轮缩放</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card" style="padding:0;overflow:hidden;background:radial-gradient(ellipse at center, #0a0a2e 0%, #000 100%);">
|
||||
<div id="globeContainer" style="width:100%;height:70vh;min-height:400px;cursor:grab;"></div>
|
||||
</div>
|
||||
|
||||
<script src="{% static 'tianxuan/three.min.js' %}"></script>
|
||||
<script src="{% static 'tianxuan/world_borders.js' %}"></script>
|
||||
<script>
|
||||
var flowData = {{ geo_flows|safe }};
|
||||
var container = document.getElementById('globeContainer');
|
||||
var H = container.clientHeight || window.innerHeight * 0.8;
|
||||
var W = container.clientWidth;
|
||||
|
||||
var scene = new THREE.Scene();
|
||||
var camera = new THREE.PerspectiveCamera(45, W/H, 0.1, 1000);
|
||||
camera.position.set(0, 0.5, 12);
|
||||
|
||||
var renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
|
||||
renderer.setSize(W, H);
|
||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
||||
renderer.setClearColor(0x000000, 0);
|
||||
container.appendChild(renderer.domElement);
|
||||
|
||||
var texLoader = new THREE.TextureLoader();
|
||||
var earthMap = texLoader.load("{% static 'tianxuan/earth_atmos_2048.jpg' %}");
|
||||
|
||||
var earthGeo = new THREE.SphereGeometry(5, 96, 96);
|
||||
var earthMat = new THREE.MeshPhongMaterial({ map: earthMap, specular: new THREE.Color(0x333333), shininess: 5 });
|
||||
var earth = new THREE.Mesh(earthGeo, earthMat);
|
||||
scene.add(earth);
|
||||
|
||||
var glowGeo = new THREE.SphereGeometry(5.08, 64, 64);
|
||||
var glowMat = new THREE.MeshPhongMaterial({ color: 0x4361ee, transparent: true, opacity: 0.12, side: THREE.BackSide });
|
||||
scene.add(new THREE.Mesh(glowGeo, glowMat));
|
||||
|
||||
var gridMat = new THREE.LineBasicMaterial({ color: 0xffffff, transparent: true, opacity: 0.12 });
|
||||
var gridGroup = new THREE.Group();
|
||||
for (var lat = -80; lat <= 80; lat += 10) {
|
||||
var phi = (90 - lat) * Math.PI / 180; var pts = [];
|
||||
for (var i = 0; i <= 64; i++) {
|
||||
var theta = i / 64 * Math.PI * 2;
|
||||
pts.push(new THREE.Vector3(5.02 * Math.sin(phi) * Math.cos(theta), 5.02 * Math.cos(phi), 5.02 * Math.sin(phi) * Math.sin(theta)));
|
||||
}
|
||||
gridGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), gridMat));
|
||||
}
|
||||
for (var lon = 0; lon < 360; lon += 10) {
|
||||
var pts = [];
|
||||
for (var i = 0; i <= 32; i++) {
|
||||
var phi = i / 32 * Math.PI; var theta = lon * Math.PI / 180;
|
||||
pts.push(new THREE.Vector3(5.02 * Math.sin(phi) * Math.cos(theta), 5.02 * Math.cos(phi), 5.02 * Math.sin(phi) * Math.sin(theta)));
|
||||
}
|
||||
gridGroup.add(new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), gridMat));
|
||||
}
|
||||
scene.add(gridGroup);
|
||||
|
||||
// ── World borders (loaded from static file) ──
|
||||
var borderGroup = new THREE.Group();
|
||||
var borderMat = new THREE.LineBasicMaterial({ color: 0x88dd88, transparent: true, opacity: 0.4, depthTest: true, depthWrite: false });
|
||||
var borderData = []; // { line, center }
|
||||
function renderBorders(borders) {
|
||||
if (!borders) return;
|
||||
for (var c = 0; c < borders.length; c++) {
|
||||
var pts = [];
|
||||
for (var p = 0; p < borders[c].length; p++) {
|
||||
var lat = borders[c][p][0], lon = borders[c][p][1];
|
||||
var phi = (90 - lat) * Math.PI / 180;
|
||||
var theta = (lon + 180) * Math.PI / 180;
|
||||
pts.push(new THREE.Vector3(5.01 * Math.sin(phi) * (-Math.cos(theta)), 5.01 * Math.cos(phi), 5.01 * Math.sin(phi) * Math.sin(theta)));
|
||||
}
|
||||
if (pts.length > 2) {
|
||||
var line = new THREE.Line(new THREE.BufferGeometry().setFromPoints(pts), borderMat);
|
||||
borderGroup.add(line);
|
||||
// Compute center of polygon (average of all points)
|
||||
var cx = 0, cy = 0, cz = 0;
|
||||
for (var p = 0; p < pts.length; p++) { cx += pts[p].x; cy += pts[p].y; cz += pts[p].z; }
|
||||
borderData.push({ line: line, center: new THREE.Vector3(cx / pts.length, cy / pts.length, cz / pts.length) });
|
||||
}
|
||||
}
|
||||
}
|
||||
if (typeof worldBorders !== 'undefined') renderBorders(worldBorders);
|
||||
scene.add(borderGroup);
|
||||
|
||||
var starsGeo = new THREE.BufferGeometry();
|
||||
var starPos = new Float32Array(5000 * 3);
|
||||
for (var i = 0; i < 5000 * 3; i++) starPos[i] = (Math.random() - 0.5) * 400;
|
||||
starsGeo.setAttribute('position', new THREE.BufferAttribute(starPos, 3));
|
||||
scene.add(new THREE.Points(starsGeo, new THREE.PointsMaterial({ color: 0xffffff, size: 0.2, transparent: true, opacity: 0.7 })));
|
||||
|
||||
scene.add(new THREE.AmbientLight(0x222244));
|
||||
var dl = new THREE.DirectionalLight(0xffffff, 1.2); dl.position.set(10, 15, 10); scene.add(dl);
|
||||
scene.add(new THREE.DirectionalLight(0x8888ff, 0.3));
|
||||
|
||||
function ll2pos(lat, lon, r) {
|
||||
var phi = (90 - lat) * Math.PI / 180;
|
||||
var theta = (lon + 180) * Math.PI / 180;
|
||||
r = r || 5.5; // well above earth surface (5.0) for depth testing
|
||||
return new THREE.Vector3(-r * Math.sin(phi) * Math.cos(theta), r * Math.cos(phi), r * Math.sin(phi) * Math.sin(theta));
|
||||
}
|
||||
|
||||
// ── Pulse arcs: faint tube + moving trail spheres ──
|
||||
// Uses depthTest:true — Three.js naturally hides back-face arcs behind earth
|
||||
var arcGroup = new THREE.Group();
|
||||
var arcColors = { 'TLSv1.3': 0x4361ee, 'TLSv1.2': 0xf72585, 'TLSv1.1': 0x7209b7, 'TLSv1.0': 0x7209b7 };
|
||||
var TRAIL = 16;
|
||||
var SPACING = 0.018;
|
||||
var arcData = [];
|
||||
|
||||
flowData.slice(0, 120).forEach(function(f) {
|
||||
var src = ll2pos(f.slat, f.slon, 5.03);
|
||||
var dst = ll2pos(f.dlat, f.dlon, 5.03);
|
||||
if (src.distanceTo(dst) < 1.0) return;
|
||||
var mid = new THREE.Vector3().addVectors(src, dst).multiplyScalar(0.5);
|
||||
var dist = src.distanceTo(dst);
|
||||
mid.normalize().multiplyScalar(6.5 + dist * 0.2);
|
||||
var curve = new THREE.QuadraticBezierCurve3(src, mid, dst);
|
||||
var c = arcColors[f.tls] || 0x4cc9f0;
|
||||
var col = new THREE.Color(c);
|
||||
|
||||
var tube = new THREE.Mesh(
|
||||
new THREE.TubeGeometry(curve, 24, 0.03, 6, false),
|
||||
new THREE.MeshBasicMaterial({ color: c, transparent: true, opacity: 0.15, depthTest: true, depthWrite: false })
|
||||
);
|
||||
arcGroup.add(tube);
|
||||
|
||||
var spheres = [];
|
||||
for (var i = 0; i < TRAIL; i++) {
|
||||
spheres.push(new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.04, 6, 6),
|
||||
new THREE.MeshBasicMaterial({ color: col, transparent: true, opacity: 1 - i / TRAIL, depthTest: true, depthWrite: false })
|
||||
));
|
||||
}
|
||||
for (var i = 0; i < spheres.length; i++) arcGroup.add(spheres[i]);
|
||||
arcData.push({ curve: curve, spheres: spheres, tube: tube, offset: Math.random() * 1000 });
|
||||
});
|
||||
|
||||
scene.add(arcGroup);
|
||||
document.getElementById('arcCount').textContent = arcData.length;
|
||||
|
||||
|
||||
|
||||
function updatePulses(time) {
|
||||
var speed = 0.6;
|
||||
for (var a = 0; a < arcData.length; a++) {
|
||||
var d = arcData[a];
|
||||
var phase = ((time * speed + d.offset / 1000) % 1 + 1) % 1;
|
||||
for (var s = 0; s < d.spheres.length; s++) {
|
||||
var t = phase - s * SPACING; if (t < 0) t += 1;
|
||||
var pt = d.curve.getPoint(t);
|
||||
d.spheres[s].position.copy(pt);
|
||||
d.spheres[s].material.opacity = 1 - s / TRAIL;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var inertiaY = 0.0005, inertiaX = 0;
|
||||
function animate() {
|
||||
requestAnimationFrame(animate);
|
||||
inertiaY *= 0.97; inertiaX *= 0.97;
|
||||
if (Math.abs(inertiaY) < 0.0005) inertiaY = Math.sign(inertiaY || 1) * 0.0005;
|
||||
if (Math.abs(inertiaX) < 0.0003) inertiaX = 0;
|
||||
earth.rotation.y += inertiaY; arcGroup.rotation.y += inertiaY; gridGroup.rotation.y += inertiaY; borderGroup.rotation.y += inertiaY;
|
||||
earth.rotation.x += inertiaX; arcGroup.rotation.x += inertiaX; gridGroup.rotation.x += inertiaX; borderGroup.rotation.x += inertiaX;
|
||||
updatePulses(Date.now() / 1000);
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
animate();
|
||||
|
||||
window.addEventListener('resize', function() {
|
||||
H = container.clientHeight || window.innerHeight * 0.8;
|
||||
var w = container.clientWidth; renderer.setSize(w, H); camera.aspect = w / H; camera.updateProjectionMatrix();
|
||||
});
|
||||
|
||||
var dragging = false, px = 0, py = 0;
|
||||
renderer.domElement.addEventListener('mousedown', function(e) {
|
||||
if (e.button === 1) { e.preventDefault(); camera.position.set(0, 0.5, 12); return; }
|
||||
dragging = true; px = e.clientX; py = e.clientY; inertiaY = 0; inertiaX = 0;
|
||||
});
|
||||
renderer.domElement.addEventListener('wheel', function(e) { e.preventDefault(); camera.position.z += e.deltaY * 0.01; if (camera.position.z < 6) camera.position.z = 6; if (camera.position.z > 30) camera.position.z = 30; }, { passive: false });
|
||||
window.addEventListener('mousemove', function(e) {
|
||||
if (!dragging) return;
|
||||
var dx = e.clientX - px, dy = e.clientY - py;
|
||||
inertiaY = dx * 0.008; inertiaX = dy * 0.005;
|
||||
px = e.clientX; py = e.clientY;
|
||||
});
|
||||
window.addEventListener('mouseup', function() { dragging = false; });
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,5 @@
|
||||
{% extends 'base.html' %}{% block content %}
|
||||
<div class="card"><h2>系统日志 (最近200行)</h2>
|
||||
<pre style="background:#1a1a2e;color:#e0e0e0;padding:1rem;overflow:auto;max-height:80vh;font-size:0.8rem;">{% for line in lines %}{{ line }}{% endfor %}</pre>
|
||||
<meta http-equiv="refresh" content="10">
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,145 @@
|
||||
{% extends 'base.html' %}
|
||||
{% load query_helpers %}
|
||||
{% block title %}手动分析 - 天璇{% endblock %}
|
||||
{% block content %}
|
||||
<div class="card">
|
||||
<h2>手动分析 - 第 {{ step }} 步 / 共 3 步</h2>
|
||||
</div>
|
||||
|
||||
{% if step == 1 %}
|
||||
<div class="card">
|
||||
<h3>Step 1: 选择数据集</h3>
|
||||
<p>选择已上传且预处理完成的数据集：</p>
|
||||
{% if runs %}
|
||||
<form method="get" action="/analyze/manual/" style="margin-top:1rem;">
|
||||
<input type="hidden" name="step" value="2">
|
||||
<table>
|
||||
<thead><tr><th></th><th>ID</th><th>文件</th><th>行数</th><th>实体列</th><th>状态</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td><input type="radio" name="run_id" value="{{ run.id }}" required></td>
|
||||
<td>#{{ run.id }}</td>
|
||||
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
|
||||
<td>{{ run.total_flows|default:"-" }}</td>
|
||||
<td>{{ run.entity_column|default:"-" }}</td>
|
||||
<td><span class="badge badge-success">{{ run.status }}</span></td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr><td colspan="6" style="text-align:center;">暂无已预处理的数据集。先去 <a href="/upload/">上传页面</a> 上传 CSV。</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
<button type="submit" class="btn btn-primary" style="margin-top:1rem;">下一步 →</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if step == 2 %}
|
||||
<div class="card">
|
||||
<h3>Step 2: 配置聚类参数</h3>
|
||||
<form method="get" action="/analyze/manual/">
|
||||
<input type="hidden" name="step" value="3">
|
||||
<input type="hidden" name="run_id" value="{{ request.GET.run_id }}">
|
||||
<div style="margin-bottom:1rem;">
|
||||
<label>实体列（entity_columns）：</label>
|
||||
<select name="entity_columns" multiple style="width:100%;min-height:100px;">
|
||||
{% if entity_candidates %}
|
||||
{% for c in entity_candidates %}
|
||||
<option value="{{ c.column }}" {% if c.column in selected_entity_cols %}selected{% endif %}>{{ c.column }} (score: {{ c.score }})</option>
|
||||
{% endfor %}
|
||||
{% elif schema_keys %}
|
||||
{% for col in schema_keys %}
|
||||
<option value="{{ col }}" {% if col in selected_entity_cols %}selected{% endif %}>{{ col }}</option>
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
<option value="">暂无可选列</option>
|
||||
{% endif %}
|
||||
</select>
|
||||
<small style="color:#666;">按住 Ctrl 多选，留空则使用自动检测结果</small>
|
||||
</div>
|
||||
<div style="margin-bottom:1rem;">
|
||||
<label>特征列（feature_columns）：</label>
|
||||
<select name="feature_columns" multiple style="width:100%;min-height:100px;">
|
||||
{% if feature_options %}
|
||||
{% for opt in feature_options %}
|
||||
<option value="{{ opt.name }}">{{ opt.name }} ({{ opt.dtype }})</option>
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
<option value="">暂无可选数值列</option>
|
||||
{% endif %}
|
||||
</select>
|
||||
<small style="color:#666;">按住 Ctrl 多选，留空则自动选择前10个数值列。仅显示数值类型列。</small>
|
||||
</div>
|
||||
<div style="margin-bottom:1rem;">
|
||||
<label>聚类算法：</label>
|
||||
<select name="algorithm">
|
||||
<option value="hdbscan" {% if default_algo == 'hdbscan' %}selected{% endif %}>HDBSCAN（自动确定簇数）</option>
|
||||
<option value="kmeans" {% if default_algo == 'kmeans' %}selected{% endif %}>KMeans（需预设簇数）</option>
|
||||
</select>
|
||||
</div>
|
||||
<div style="margin-bottom:1rem;">
|
||||
<label>最小簇大小（min_cluster_size）：</label>
|
||||
<input type="number" name="min_cluster_size" value="{{ default_min_size }}" min="2" max="100">
|
||||
</div>
|
||||
<a href="/analyze/manual/?step=1" class="btn" style="background:#ddd;margin-right:0.5rem;">← 上一步</a>
|
||||
<button type="submit" class="btn btn-primary">下一步 →</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if step == 3 %}
|
||||
<div class="card">
|
||||
<h3>Step 3: 确认并运行</h3>
|
||||
<p>请确认以下配置，确认无误后点击"开始分析"：</p>
|
||||
<table>
|
||||
<tr><td>数据集 ID</td><td>#{{ request.GET.run_id }}</td></tr>
|
||||
<tr><td>实体列</td><td>{{ request.GET|getlist:"entity_columns"|default:"自动检测" }}</td></tr>
|
||||
<tr><td>特征列</td><td>{{ request.GET.feature_columns|default:"自动选择" }}</td></tr>
|
||||
<tr><td>聚类算法</td><td>{{ request.GET.algorithm }}</td></tr>
|
||||
<tr><td>最小簇大小</td><td>{{ request.GET.min_cluster_size }}</td></tr>
|
||||
</table>
|
||||
<div style="margin-top:1rem;">
|
||||
<a href="/analyze/manual/?step=2&run_id={{ request.GET.run_id }}" class="btn" style="background:#ddd;margin-right:0.5rem;">← 上一步</a>
|
||||
<button class="btn btn-primary" onclick="runAnalysis()">开始分析</button>
|
||||
</div>
|
||||
<div id="runProgress" style="display:none;margin-top:1rem;">
|
||||
<p>分析进行中... <span id="runStatus"></span></p>
|
||||
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
|
||||
<div id="runBar" style="background:#4361ee;height:100%;width:0%;"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
async function runAnalysis() {
|
||||
document.getElementById('runProgress').style.display = 'block';
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
const entityCols = params.getAll('entity_columns');
|
||||
const featureCols = params.getAll('feature_columns');
|
||||
const resp = await fetch('/analyze/run/', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json',
|
||||
'X-CSRFToken': '{{ csrf_token }}' },
|
||||
body: JSON.stringify({
|
||||
run_id: parseInt('{{ request.GET.run_id }}'),
|
||||
entity_column: '',
|
||||
entity_columns: entityCols.length ? entityCols : undefined,
|
||||
feature_columns: featureCols.length ? featureCols : undefined,
|
||||
algorithm: '{{ request.GET.algorithm }}',
|
||||
min_cluster_size: parseInt('{{ request.GET.min_cluster_size }}'),
|
||||
}),
|
||||
});
|
||||
const data = await resp.json();
|
||||
const runId = '{{ request.GET.run_id }}';
|
||||
const poll = setInterval(async () => {
|
||||
const r = await fetch(`/runs/${runId}/status/`);
|
||||
const s = await r.json();
|
||||
document.getElementById('runStatus').textContent = s.status;
|
||||
if (s.status === 'completed') { clearInterval(poll); window.location.href = `/runs/${runId}/`; }
|
||||
if (s.status === 'failed') { clearInterval(poll); document.getElementById('runStatus').textContent = '失败: ' + (s.error_message || ''); }
|
||||
}, 1000);
|
||||
}
|
||||
</script>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,125 @@
|
||||
{% extends 'base.html' %}
|
||||
{% block title %}上传数据 - 天璇{% endblock %}
|
||||
{% block content %}
|
||||
<style>
|
||||
#uploadCard { position: sticky; top: 1rem; z-index: 10; }
|
||||
</style>
|
||||
<div class="card" id="uploadCard">
|
||||
<h2>上传 CSV 文件</h2>
|
||||
<p style="color:#666;margin-bottom:1rem;">支持 .csv 文件，多文件批量上传，自动解压 .zip</p>
|
||||
<form id="uploadForm" enctype="multipart/form-data">
|
||||
{% csrf_token %}
|
||||
<div style="border:2px dashed #ccc;border-radius:8px;padding:3rem;text-align:center;cursor:pointer;background:#fafafa;" id="dropZone">
|
||||
<div style="font-size:2rem;margin-bottom:0.5rem;">📂</div>
|
||||
<p>拖拽 CSV 文件到此处，或点击选择</p>
|
||||
<input type="file" name="files" multiple accept=".csv,.zip" style="display:none;" id="fileInput">
|
||||
</div>
|
||||
<div id="submitBar" style="margin-top:1rem;display:none;">
|
||||
<button type="submit" class="btn btn-primary" id="submitBtn">开始上传</button>
|
||||
</div>
|
||||
<div id="fileList" style="margin-top:0.5rem;"></div>
|
||||
</form>
|
||||
<div id="progressArea" style="display:none;margin-top:1rem;">
|
||||
<p>上传中... <span id="progressText"></span></p>
|
||||
<div style="background:#eee;height:8px;border-radius:4px;overflow:hidden;">
|
||||
<div id="progressBar" style="background:#4361ee;height:100%;width:0%;"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card" style="margin-top:1.5rem;">
|
||||
<h3>已上传的数据集</h3>
|
||||
{% if runs %}
|
||||
<table>
|
||||
<thead><tr><th>ID</th><th>文件</th><th>行数</th><th>状态</th><th>操作</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr>
|
||||
<td>#{{ run.id }}</td>
|
||||
<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;">{{ run.csv_glob }}</td>
|
||||
<td>{{ run.total_flows|default:"-" }}</td>
|
||||
<td><span class="badge {% if run.status == 'ready' or run.status == 'completed' %}badge-success{% elif run.status == 'failed' %}badge-danger{% else %}badge-warning{% endif %}">{{ run.status }}</span></td>
|
||||
<td>
|
||||
{% if run.status == 'loading' or run.status == 'profiling' or run.status == 'aggregating' or run.status == 'clustering' or run.status == 'extracting' %}
|
||||
<button class="btn btn-danger" style="background:#dc3545;color:#fff;padding:0.25rem 0.5rem;font-size:0.8rem;" onclick="deleteRun({{ run.id }}, true)">删除</button>
|
||||
{% else %}
|
||||
<button class="btn btn-danger" style="background:#dc3545;color:#fff;padding:0.25rem 0.5rem;font-size:0.8rem;" onclick="deleteRun({{ run.id }})">删除</button>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<p style="color:#999;">暂无已上传的数据集</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const dropZone = document.getElementById('dropZone');
|
||||
const fileInput = document.getElementById('fileInput');
|
||||
const fileList = document.getElementById('fileList');
|
||||
const submitBar = document.getElementById('submitBar');
|
||||
const submitBtn = document.getElementById('submitBtn');
|
||||
let selectedFiles = [];
|
||||
|
||||
dropZone.addEventListener('click', () => fileInput.click());
|
||||
dropZone.addEventListener('dragover', e => { e.preventDefault(); dropZone.style.borderColor = '#4361ee'; });
|
||||
dropZone.addEventListener('dragleave', () => dropZone.style.borderColor = '#ccc');
|
||||
dropZone.addEventListener('drop', e => { e.preventDefault(); handleFiles(e.dataTransfer.files); });
|
||||
fileInput.addEventListener('change', () => handleFiles(fileInput.files));
|
||||
|
||||
function handleFiles(files) {
|
||||
selectedFiles = [...files];
|
||||
fileList.innerHTML = selectedFiles.map((f, i) =>
|
||||
`<div style="padding:0.5rem;background:#f5f7fa;margin:0.25rem 0;border-radius:4px;">${i+1}. ${f.name} (${(f.size/1024).toFixed(0)} KB)</div>`
|
||||
).join('');
|
||||
submitBar.style.display = 'block';
|
||||
}
|
||||
|
||||
document.getElementById('uploadForm').addEventListener('submit', async (e) => {
|
||||
e.preventDefault();
|
||||
const form = new FormData();
|
||||
selectedFiles.forEach(f => form.append('files', f));
|
||||
document.getElementById('progressArea').style.display = 'block';
|
||||
submitBtn.disabled = true;
|
||||
try {
|
||||
const resp = await fetch('/upload/csv/', { method: 'POST', body: form,
|
||||
headers: { 'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]').value } });
|
||||
const data = await resp.json();
|
||||
document.getElementById('progressText').textContent = '上传完成，后台处理中...';
|
||||
// Poll for completion
|
||||
const runId = data.run_id;
|
||||
const poll = setInterval(async () => {
|
||||
const r = await fetch(`/runs/${runId}/status/`);
|
||||
const s = await r.json();
|
||||
const pct = s.progress_pct || 0;
|
||||
const msg = s.progress_msg || s.status;
|
||||
document.getElementById('progressBar').style.width = pct + '%';
|
||||
document.getElementById('progressText').textContent = `${msg} (${pct}%)`;
|
||||
if (s.status === 'ready' || s.status === 'completed') { clearInterval(poll); window.location.href = `/runs/${runId}/`; }
|
||||
if (s.status === 'failed') { clearInterval(poll); document.getElementById('progressText').textContent = '处理失败: ' + (s.error_message || '未知错误'); }
|
||||
}, 1000);
|
||||
} catch (err) { document.getElementById('progressText').textContent = '上传失败: ' + err.message; submitBtn.disabled = false; }
|
||||
});
|
||||
|
||||
async function deleteRun(runId, processing) {
|
||||
const msg = processing ? '后台处理中，删除将中断分析，确认删除？' : '确定要删除此数据集吗？此操作不可撤销。';
|
||||
if (!confirm(msg)) return;
|
||||
try {
|
||||
const resp = await fetch(`/runs/${runId}/delete/`, {
|
||||
method: 'POST',
|
||||
headers: { 'X-CSRFToken': document.querySelector('[name=csrfmiddlewaretoken]').value },
|
||||
});
|
||||
const data = await resp.json();
|
||||
if (data.deleted) {
|
||||
location.reload();
|
||||
} else {
|
||||
alert('删除失败');
|
||||
}
|
||||
} catch (err) {
|
||||
alert('删除失败: ' + err.message);
|
||||
}
|
||||
}
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,367 @@
|
||||
"""Tests for analysis.data_loader — BOM detection, schema validation, load/config."""
|
||||
|
||||
import csv
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import polars as pl
|
||||
|
||||
from analysis.data_loader import (
|
||||
_file_count,
|
||||
_merge_schema_entries,
|
||||
_resolve_encoding,
|
||||
detect_bom,
|
||||
load_config,
|
||||
load_csv_directory,
|
||||
validate_schemas,
|
||||
)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# BOM detection
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestDetectBOM:
|
||||
"""detect_bom() should return the correct encoding string for each BOM."""
|
||||
|
||||
def test_utf8_bom(self):
|
||||
"""UTF-8 BOM (EF BB BF) → 'utf-8-sig'."""
|
||||
path = _write_bytes(b'\xef\xbb\xbfcol1,col2\n1,2\n')
|
||||
try:
|
||||
assert detect_bom(path) == 'utf-8-sig'
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_utf16le_bom(self):
|
||||
"""UTF-16 LE BOM (FF FE) → 'utf-16le'."""
|
||||
path = _write_bytes(b'\xff\xfec\x00o\x00l\x001\x00,\x00c\x00o\x00l\x002\x00\n\x001\x00,\x002\x00\n')
|
||||
try:
|
||||
assert detect_bom(path) == 'utf-16le'
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_utf16be_bom(self):
|
||||
"""UTF-16 BE BOM (FE FF) → 'utf-16be'."""
|
||||
path = _write_bytes(b'\xfe\xff\x00c\x00o\x00l\x001\x00,\x00c\x00o\x00l\x002\x00\n\x001\x00,\x002\x00\n')
|
||||
try:
|
||||
assert detect_bom(path) == 'utf-16be'
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_no_bom(self):
|
||||
"""Plain ASCII/UTF-8 without BOM → 'utf-8'."""
|
||||
path = _write_bytes(b'col1,col2\n1,2\n')
|
||||
try:
|
||||
assert detect_bom(path) == 'utf-8'
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_empty_file(self):
|
||||
"""Empty file → 'utf-8'."""
|
||||
path = _write_bytes(b'')
|
||||
try:
|
||||
assert detect_bom(path) == 'utf-8'
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Schema validation
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestValidateSchemas:
|
||||
"""validate_schemas() must raise ValueError when column sets differ."""
|
||||
|
||||
def test_matching_schemas(self):
|
||||
"""Identical schemas → no exception."""
|
||||
schemas = [
|
||||
{':ips': 'Utf8', '8ack': 'Int64'},
|
||||
{':ips': 'Utf8', '8ack': 'Int64'},
|
||||
]
|
||||
validate_schemas(schemas) # should not raise
|
||||
|
||||
def test_empty_list(self):
|
||||
"""Empty list → no exception."""
|
||||
validate_schemas([])
|
||||
|
||||
def test_schema_mismatch_missing_column(self):
|
||||
"""One file missing a column → ValueError."""
|
||||
schemas = [
|
||||
{':ips': 'Utf8', '8ack': 'Int64'},
|
||||
{':ips': 'Utf8'}, # missing '8ack'
|
||||
]
|
||||
with pytest.raises(ValueError, match=r'missing columns'):
|
||||
validate_schemas(schemas)
|
||||
|
||||
def test_schema_mismatch_extra_column(self):
|
||||
"""One file has an extra column → ValueError."""
|
||||
schemas = [
|
||||
{':ips': 'Utf8'},
|
||||
{':ips': 'Utf8', 'extra_col': 'Int64'},
|
||||
]
|
||||
with pytest.raises(ValueError, match=r'extra columns'):
|
||||
validate_schemas(schemas)
|
||||
|
||||
def test_schema_mismatch_both_sides(self):
|
||||
"""Both missing and extra columns → ValueError with both messages."""
|
||||
schemas = [
|
||||
{'a': 'Int64', 'b': 'Utf8'},
|
||||
{'a': 'Int64', 'c': 'Float64'},
|
||||
]
|
||||
with pytest.raises(ValueError):
|
||||
validate_schemas(schemas)
|
||||
|
||||
|
||||
class TestMergeSchemaEntries:
|
||||
"""_merge_schema_entries() should merge without raising."""
|
||||
|
||||
def test_simple_merge(self):
|
||||
merged, conflicts = _merge_schema_entries([
|
||||
{'a': 'Int64', 'b': 'Utf8'},
|
||||
{'a': 'Int64', 'c': 'Float64'},
|
||||
])
|
||||
assert set(merged.keys()) == {'a', 'b', 'c'}
|
||||
assert merged['a'] == 'Int64'
|
||||
|
||||
def test_type_conflict(self):
|
||||
"""Different dtypes for same column → last wins, conflict recorded."""
|
||||
merged, conflicts = _merge_schema_entries([
|
||||
{'val': 'Int64'},
|
||||
{'val': 'Float64'},
|
||||
])
|
||||
assert merged['val'] == 'Float64'
|
||||
assert 'val' in conflicts
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# CSV loading
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestLoadCSVDirectory:
|
||||
"""End-to-end load_csv_directory() tests with temporary CSV files."""
|
||||
|
||||
def test_basic_load(self):
|
||||
"""Single CSV → LazyFrame + correct schema + row/file counts."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
path = Path(tmpdir) / 'test.csv'
|
||||
_write_csv(path, [
|
||||
[':ips', ':ipd', '8ack'],
|
||||
['10.0.0.1', '10.0.0.2', '100'],
|
||||
['10.0.0.1', '10.0.0.3', '200'],
|
||||
['10.0.0.2', '10.0.0.4', '150'],
|
||||
])
|
||||
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
|
||||
str(path), encoding='utf-8'
|
||||
)
|
||||
|
||||
assert isinstance(lf, pl.LazyFrame)
|
||||
assert ':ips' in schema
|
||||
assert ':ipd' in schema
|
||||
assert '8ack' in schema
|
||||
assert file_count == 1
|
||||
assert row_count > 0
|
||||
assert memory_mb >= 0
|
||||
|
||||
# Verify data round-trip
|
||||
df = lf.collect()
|
||||
assert len(df) == 3
|
||||
assert set(df[':ips'].to_list()) == {'10.0.0.1', '10.0.0.2'}
|
||||
|
||||
def test_multiple_files(self):
|
||||
"""Multiple CSVs with same schema → merged LazyFrame."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
p1 = Path(tmpdir) / 'part1.csv'
|
||||
p2 = Path(tmpdir) / 'part2.csv'
|
||||
_write_csv(p1, [['x'], ['1'], ['2']])
|
||||
_write_csv(p2, [['x'], ['3'], ['4']])
|
||||
|
||||
glob_pattern = str(Path(tmpdir) / 'part*.csv')
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
|
||||
glob_pattern, encoding='utf-8'
|
||||
)
|
||||
|
||||
assert file_count == 2
|
||||
df = lf.collect()
|
||||
assert len(df) == 4
|
||||
assert sorted(df['x'].to_list()) == [1, 2, 3, 4]
|
||||
|
||||
def test_no_files(self):
|
||||
"""No matching files → FileNotFoundError."""
|
||||
with pytest.raises(FileNotFoundError, match=r'No files match'):
|
||||
load_csv_directory('__nonexistent_glarg_*.csv')
|
||||
|
||||
def test_schema_mismatch_across_files_strict(self):
|
||||
"""Files with different column sets + schema_strict=True → ValueError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
_write_csv(Path(tmpdir) / 'a.csv', [['x', 'y'], ['1', '2']])
|
||||
_write_csv(Path(tmpdir) / 'b.csv', [['x'], ['3']])
|
||||
|
||||
with pytest.raises(ValueError, match=r'Schema mismatch'):
|
||||
load_csv_directory(str(Path(tmpdir) / '*.csv'), schema_strict=True)
|
||||
|
||||
def test_schema_mismatch_across_files_lenient(self):
|
||||
"""Files with different column sets + schema_strict=False → merge with nulls."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
_write_csv(Path(tmpdir) / 'a.csv', [['x', 'y'], ['1', '2']])
|
||||
_write_csv(Path(tmpdir) / 'b.csv', [['x'], ['3']])
|
||||
|
||||
lf, schema, *_ = load_csv_directory(str(Path(tmpdir) / '*.csv'))
|
||||
df = lf.collect()
|
||||
assert len(df) == 2
|
||||
assert 'x' in df.columns
|
||||
assert 'y' in df.columns
|
||||
# Row from b.csv should have null for 'y'
|
||||
assert df['y'].is_null().sum() == 1
|
||||
|
||||
def test_schema_override(self):
|
||||
"""schema_override forces a specific column dtype."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
path = Path(tmpdir) / 'test.csv'
|
||||
_write_csv(path, [['val'], ['123'], ['456']])
|
||||
|
||||
lf, schema, *_ = load_csv_directory(
|
||||
str(path), encoding='utf-8',
|
||||
schema_override={'val': 'Float64'},
|
||||
)
|
||||
|
||||
df = lf.collect()
|
||||
assert df['val'].dtype == pl.Float64
|
||||
|
||||
def test_custom_delimiter(self):
|
||||
"""Tab-delimited file should parse correctly."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
path = Path(tmpdir) / 'test.tsv'
|
||||
with open(path, 'w', newline='') as f:
|
||||
f.write('a\tb\n1\t2\n3\t4\n')
|
||||
|
||||
lf, *_ = load_csv_directory(str(path), delimiter='\t')
|
||||
df = lf.collect()
|
||||
assert len(df) == 2
|
||||
assert df.columns == ['a', 'b']
|
||||
|
||||
def test_glob_with_subdirectory(self):
|
||||
"""Recursive glob should find files in subdirectories."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
sub = Path(tmpdir) / 'sub'
|
||||
sub.mkdir()
|
||||
_write_csv(sub / 'nested.csv', [['x'], ['42']])
|
||||
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
|
||||
str(Path(tmpdir) / '**' / '*.csv')
|
||||
)
|
||||
assert file_count == 1
|
||||
assert row_count > 0
|
||||
|
||||
def test_latlon_with_plus_and_blank(self):
|
||||
"""CSV with lat/lon columns containing '+', '' loads without error
|
||||
and the column is Float64 after cleaning."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
path = Path(tmpdir) / 'latlon_test.csv'
|
||||
with open(path, 'w', newline='') as f:
|
||||
f.write('src_ip,latitude,longitude\n')
|
||||
f.write('10.0.0.1,+,120.0\n')
|
||||
f.write('10.0.0.2,35.5,\n')
|
||||
f.write('10.0.0.3,-,\n')
|
||||
f.write('10.0.0.4,40.0,121.5\n')
|
||||
|
||||
lf, schema, *_ = load_csv_directory(str(path))
|
||||
df = lf.collect()
|
||||
|
||||
# lat/lon columns should be Float64 in schema
|
||||
assert 'latitude' in schema
|
||||
assert 'longitude' in schema
|
||||
assert 'latitude' in df.columns
|
||||
assert 'longitude' in df.columns
|
||||
|
||||
# Check dtypes are Float64
|
||||
assert df['latitude'].dtype == pl.Float64
|
||||
assert df['longitude'].dtype == pl.Float64
|
||||
|
||||
# Standalone '+' and blank should become null
|
||||
assert df['latitude'].is_null().sum() == 2 # rows 0,2
|
||||
assert df['longitude'].is_null().sum() == 2 # rows 1,2
|
||||
|
||||
# Valid values survive
|
||||
assert df['latitude'].drop_nulls().to_list() == [35.5, 40.0]
|
||||
assert df['longitude'].drop_nulls().to_list() == [120.0, 121.5]
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Config loading
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestLoadConfig:
|
||||
"""load_config() YAML loading and error handling."""
|
||||
|
||||
def test_load_valid_yaml(self):
|
||||
path = _write_yaml('key: value\nnested:\n inner: 42\n')
|
||||
try:
|
||||
config = load_config(path)
|
||||
assert config['key'] == 'value'
|
||||
assert config['nested']['inner'] == 42
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_load_empty_yaml(self):
|
||||
path = _write_yaml('')
|
||||
try:
|
||||
config = load_config(path)
|
||||
assert config == {}
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_config_path_none(self):
|
||||
assert load_config(None) == {}
|
||||
|
||||
def test_config_not_found(self):
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_config('/__nonexistent__/config.yaml')
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Utility functions
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestResolveEncoding:
|
||||
def test_bom_overrides_default(self):
|
||||
assert _resolve_encoding('utf-8', 'utf-16le') == 'utf-16le'
|
||||
|
||||
def test_no_bom_uses_default(self):
|
||||
# Polars >= 1.0 uses 'utf8' (not 'utf-8')
|
||||
assert _resolve_encoding('utf-8', 'utf-8') == 'utf8'
|
||||
assert _resolve_encoding('latin-1', 'utf-8') == 'latin-1'
|
||||
|
||||
|
||||
class TestFileCount:
|
||||
def test_count(self):
|
||||
assert _file_count(['a', 'b', 'c']) == 3
|
||||
assert _file_count([]) == 0
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Helpers
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
def _write_csv(path: Path, rows: list[list[str]]) -> None:
|
||||
"""Write rows as a CSV file."""
|
||||
with open(path, 'w', newline='') as f:
|
||||
writer = csv.writer(f)
|
||||
for row in rows:
|
||||
writer.writerow(row)
|
||||
|
||||
|
||||
def _write_bytes(data: bytes) -> str:
|
||||
"""Write raw bytes to a temp file; return the path string."""
|
||||
with tempfile.NamedTemporaryFile(suffix='.csv', delete=False) as f:
|
||||
f.write(data)
|
||||
return f.name
|
||||
|
||||
|
||||
def _write_yaml(content: str) -> str:
|
||||
"""Write YAML string to a temp file; return the path string."""
|
||||
with tempfile.NamedTemporaryFile(suffix='.yaml', mode='w', delete=False) as f:
|
||||
f.write(content)
|
||||
return f.name
|
||||
@@ -0,0 +1,365 @@
|
||||
"""End-to-end integration test for the full TLS analysis pipeline.
|
||||
|
||||
Pipeline
|
||||
1. Create synthetic TLS flow CSV with known entity clusters.
|
||||
2. Load via ``load_csv_directory``.
|
||||
3. Store in ``SessionStore``.
|
||||
4. Auto-detect entity column (``detect_entity_column``).
|
||||
5. Aggregate flows → entity profiles (``aggregate_by_entity``).
|
||||
6. Run HDBSCAN clustering on aggregated numeric features.
|
||||
7. Extract per-cluster distinguishing features.
|
||||
|
||||
All data is synthetic, deterministic (seed=42), and self-contained —
|
||||
no external files or network access required.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import polars as pl
|
||||
import pytest
|
||||
|
||||
from analysis.data_loader import load_csv_directory
|
||||
from analysis.entity_detector import detect_entity_column
|
||||
from analysis.entity_aggregator import aggregate_by_entity
|
||||
from analysis.session_store import SessionStore
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Fixtures
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_store():
|
||||
"""Reset the SessionStore singleton before and after each test."""
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
yield
|
||||
store.drop_all()
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Synthetic data generator
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
def create_synthetic_flows(seed: int = 42) -> pl.DataFrame:
|
||||
"""Generate deterministic TLS flow data with three known entity groups.
|
||||
|
||||
Uses the user's exact column naming convention only.
|
||||
|
||||
Returns
|
||||
-------
|
||||
pl.DataFrame
|
||||
~1000 rows with columns.
|
||||
Entities (``:ips``) are partitioned into three behavioural
|
||||
groups that HDBSCAN should recover after aggregation.
|
||||
"""
|
||||
rng = np.random.default_rng(seed)
|
||||
|
||||
# ── Entity profiles ──────────────────────────────────────────────
|
||||
# Group A: 10 high-volume entities (50 flows each)
|
||||
group_a = [
|
||||
{
|
||||
':ips': f'10.0.0.{i+1}',
|
||||
'n_flows': 50,
|
||||
'8ack_mu': 50_000, '8ack_sigma': 5_000,
|
||||
'4dur_mu': 30.0, '4dur_sigma': 5.0,
|
||||
'dst_range': (100, 150),
|
||||
'0ver': 'TLSv1.3',
|
||||
'cipher_suite': 'TLS_AES_256_GCM_SHA384',
|
||||
}
|
||||
for i in range(10)
|
||||
]
|
||||
|
||||
# Group B: 15 medium-volume entities (20 flows each)
|
||||
group_b = [
|
||||
{
|
||||
':ips': f'10.0.0.{i+11}',
|
||||
'n_flows': 20,
|
||||
'8ack_mu': 5_000, '8ack_sigma': 1_000,
|
||||
'4dur_mu': 15.0, '4dur_sigma': 3.0,
|
||||
'dst_range': (150, 200),
|
||||
'0ver': 'TLSv1.2',
|
||||
'cipher_suite': 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
|
||||
}
|
||||
for i in range(15)
|
||||
]
|
||||
|
||||
# Group C: 25 low-volume entities (6 flows each)
|
||||
group_c = [
|
||||
{
|
||||
':ips': f'10.0.0.{i+26}',
|
||||
'n_flows': 6,
|
||||
'8ack_mu': 500, '8ack_sigma': 100,
|
||||
'4dur_mu': 5.0, '4dur_sigma': 1.0,
|
||||
'dst_range': (200, 250),
|
||||
'0ver': 'TLSv1.2',
|
||||
'cipher_suite': 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256',
|
||||
}
|
||||
for i in range(25)
|
||||
]
|
||||
|
||||
entities = group_a + group_b + group_c
|
||||
# Total flows: 10×50 + 15×20 + 25×6 = 500 + 300 + 150 = 950
|
||||
|
||||
# ── Generate flow rows ────────────────────────────────────────────
|
||||
rows: list[dict] = []
|
||||
dst_ports = [80, 443, 8080, 8443]
|
||||
protos = ['TCP', 'UDP']
|
||||
proto_weights = [0.9, 0.1]
|
||||
|
||||
for ent in entities:
|
||||
lo, hi = ent['dst_range']
|
||||
for _ in range(ent['n_flows']):
|
||||
dst_idx = int(rng.integers(lo, hi))
|
||||
row = {
|
||||
':ips': ent[':ips'],
|
||||
':ipd': f"10.0.{dst_idx >> 8}.{dst_idx & 0xff}",
|
||||
':prd': int(rng.choice(dst_ports)),
|
||||
'8ack': int(max(0, rng.normal(ent['8ack_mu'],
|
||||
ent['8ack_sigma']))),
|
||||
'4dur': round(max(0.1, abs(rng.normal(ent['4dur_mu'],
|
||||
ent['4dur_sigma']))), 2),
|
||||
'1ipp': str(rng.choice(protos, p=proto_weights)),
|
||||
'0ver': ent['0ver'],
|
||||
'cipher_suite': ent['cipher_suite'],
|
||||
'snam': f"host{rng.integers(1, 200)}.example.com",
|
||||
'timestamp': (
|
||||
f"2024-06-15 "
|
||||
f"{rng.integers(0, 23):02d}:"
|
||||
f"{rng.integers(0, 59):02d}:"
|
||||
f"{rng.integers(0, 59):02d}"
|
||||
),
|
||||
}
|
||||
rows.append(row)
|
||||
|
||||
return pl.DataFrame(rows)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Pipeline tests
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestFullPipeline:
|
||||
"""End-to-end: synthetic CSV → load → entity detect → aggregate → cluster."""
|
||||
|
||||
def _write_csv(self, df: pl.DataFrame, tmpdir: str) -> str:
|
||||
"""Write a Polars DataFrame to a temp CSV; return the file path."""
|
||||
path = Path(tmpdir) / 'synthetic_flows.csv'
|
||||
df.write_csv(path)
|
||||
return str(path)
|
||||
|
||||
def test_full_pipeline(self):
|
||||
"""Smoke-test the full pipeline — every step produces valid output."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# ── Step 1: Create synthetic data ─────────────────────────────
|
||||
df_flows = create_synthetic_flows(seed=42)
|
||||
assert len(df_flows) > 0, 'Synthetic data should not be empty'
|
||||
|
||||
csv_path = self._write_csv(df_flows, tmpdir)
|
||||
|
||||
# ── Step 2: Load data ─────────────────────────────────────────
|
||||
lf, schema, row_count, file_count, memory_mb = load_csv_directory(
|
||||
csv_path, encoding='utf-8',
|
||||
)
|
||||
assert isinstance(lf, pl.LazyFrame), 'Expected LazyFrame'
|
||||
assert file_count == 1
|
||||
assert row_count > 0
|
||||
assert memory_mb >= 0
|
||||
assert ':ips' in schema
|
||||
|
||||
# ── Step 3: Store in session ──────────────────────────────────
|
||||
store = SessionStore()
|
||||
store.store_dataset(
|
||||
'e2e_ds', lf, schema=schema,
|
||||
metadata={'row_count': row_count, 'csv_glob': csv_path},
|
||||
)
|
||||
|
||||
# ── Step 4: Entity detection ──────────────────────────────────
|
||||
det_result = detect_entity_column('e2e_ds')
|
||||
assert 'error' not in det_result, f'detect_entity_column failed: {det_result.get("error")}'
|
||||
# :ips is the catch-all entity column (strings with good unique ratio)
|
||||
assert det_result['recommended'] in (':ips', ':ipd', 'snam'), (
|
||||
f'Expected :ips, :ipd or snam, got {det_result["recommended"]}'
|
||||
)
|
||||
assert len(det_result['candidates']) > 0
|
||||
|
||||
# Verify recommended_multi exists
|
||||
assert 'recommended_multi' in det_result, 'recommended_multi field missing'
|
||||
assert len(det_result['recommended_multi']) > 0, 'recommended_multi is empty'
|
||||
|
||||
# ── Step 5: Entity aggregation ────────────────────────────────
|
||||
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
|
||||
df_agg = agg_lf.collect()
|
||||
|
||||
# Should be exactly 50 entities (10 + 15 + 25)
|
||||
entity_count = len(df_agg)
|
||||
assert entity_count == 50, f'Expected 50 entities, got {entity_count}'
|
||||
|
||||
# Verify per-entity flow counts (sort by flow_count desc so groups
|
||||
# are contiguous: A=50, B=20, C=6)
|
||||
df_sorted = df_agg.sort('flow_count', descending=True)
|
||||
# Group A: 10 entities × 50 flows
|
||||
for i in range(10):
|
||||
assert df_sorted['flow_count'][i] == 50, (
|
||||
f'Row {i}: expected 50, got {df_sorted["flow_count"][i]}'
|
||||
)
|
||||
# Group B: 15 entities × 20 flows
|
||||
for i in range(10, 25):
|
||||
assert df_sorted['flow_count'][i] == 20, (
|
||||
f'Row {i}: expected 20, got {df_sorted["flow_count"][i]}'
|
||||
)
|
||||
# Group C: 25 entities × 6 flows
|
||||
for i in range(25, 50):
|
||||
assert df_sorted['flow_count'][i] == 6, (
|
||||
f'Row {i}: expected 6, got {df_sorted["flow_count"][i]}'
|
||||
)
|
||||
|
||||
# Core feature columns are present
|
||||
for col in ('flow_count', 'total_bytes_sent',
|
||||
'avg_duration', 'unique_dst_ips', 'unique_dst_ports',
|
||||
'unique_tls_versions', 'unique_protocols',
|
||||
'first_seen', 'last_seen'):
|
||||
assert col in feature_cols, f'Missing feature column: {col}'
|
||||
|
||||
# ── Step 6: Clustering ────────────────────────────────────────
|
||||
numeric_cols = [
|
||||
'flow_count', 'total_bytes_sent',
|
||||
'avg_duration', 'unique_dst_ips', 'unique_dst_ports',
|
||||
]
|
||||
available_cols = [c for c in numeric_cols if c in df_agg.columns]
|
||||
|
||||
data = df_agg.select(available_cols).to_numpy()
|
||||
assert data.shape[0] == 50, 'All 50 entities should have features'
|
||||
|
||||
# Standard-scale + HDBSCAN
|
||||
from sklearn.preprocessing import StandardScaler
|
||||
from sklearn.cluster import HDBSCAN
|
||||
|
||||
data_scaled = StandardScaler().fit_transform(data)
|
||||
clusterer = HDBSCAN(min_cluster_size=5, metric='euclidean')
|
||||
labels = clusterer.fit_predict(data_scaled)
|
||||
|
||||
n_clusters = len(set(labels) - {-1})
|
||||
assert n_clusters > 0, (
|
||||
f'HDBSCAN should find ≥1 cluster, found {n_clusters} '
|
||||
f'(labels={set(labels)})'
|
||||
)
|
||||
|
||||
noise_count = int((labels == -1).sum())
|
||||
# At most 40% noise allowed (we have well-separated groups)
|
||||
noise_ratio = noise_count / len(labels)
|
||||
assert noise_ratio < 0.40, (
|
||||
f'Noise ratio too high: {noise_ratio:.2%} '
|
||||
f'({noise_count}/{len(labels)})'
|
||||
)
|
||||
|
||||
# ── Step 7: Feature extraction ────────────────────────────────
|
||||
unique_labels = sorted(set(l for l in labels if l >= 0))
|
||||
assert len(unique_labels) > 0, 'Need at least one valid cluster'
|
||||
|
||||
# Compute per-cluster distinguishing features using z-score
|
||||
features = []
|
||||
for label in unique_labels:
|
||||
mask = labels == label
|
||||
cluster_data = data[mask]
|
||||
for col_idx, col_name in enumerate(available_cols):
|
||||
global_mean = float(data[:, col_idx].mean())
|
||||
global_std = float(data[:, col_idx].std()) or 1.0
|
||||
cluster_mean = float(cluster_data[:, col_idx].mean())
|
||||
z = (cluster_mean - global_mean) / global_std
|
||||
features.append({
|
||||
'cluster': int(label),
|
||||
'feature': col_name,
|
||||
'z_score': round(z, 4),
|
||||
})
|
||||
|
||||
assert len(features) > 0, 'Expected at least one extracted feature'
|
||||
# Each cluster should have features for all numeric columns
|
||||
assert len(features) >= len(unique_labels) * len(available_cols)
|
||||
|
||||
# ── Step 8: Subnet aggregation function ────────────────────────
|
||||
from analysis.entity_aggregator import ip_to_subnet
|
||||
assert callable(ip_to_subnet)
|
||||
assert ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'
|
||||
|
||||
# Note: SessionStore is cleaned up by the ``clean_store`` fixture
|
||||
|
||||
def test_synthetic_data_shape(self):
|
||||
"""Verify the synthetic data generator produces expected shapes."""
|
||||
df = create_synthetic_flows(seed=42)
|
||||
assert len(df.columns) == 10, f'Expected 10 columns, got {len(df.columns)}'
|
||||
# Total: 10×50 + 15×20 + 25×6 = 950
|
||||
assert len(df) == 950, f'Expected 950 rows, got {len(df)}'
|
||||
|
||||
# Verify unique entity count
|
||||
n_entities = df[':ips'].n_unique()
|
||||
assert n_entities == 50, f'Expected 50 unique :ips, got {n_entities}'
|
||||
|
||||
# Group A should have TLSv1.3
|
||||
group_a_ips = [f'10.0.0.{i+1}' for i in range(10)]
|
||||
group_a_tls = df.filter(
|
||||
pl.col(':ips').is_in(group_a_ips)
|
||||
)['0ver'].unique().to_list()
|
||||
assert group_a_tls == ['TLSv1.3']
|
||||
|
||||
# Group C should have CHACHA20 cipher
|
||||
group_c_ips = [f'10.0.0.{i+26}' for i in range(25)]
|
||||
group_c_cipher = df.filter(
|
||||
pl.col(':ips').is_in(group_c_ips)
|
||||
)['cipher_suite'].unique().to_list()
|
||||
assert any('CHACHA20' in c for c in group_c_cipher)
|
||||
|
||||
|
||||
class TestPipelineEdgeCases:
|
||||
"""Edge cases for the analysis pipeline."""
|
||||
|
||||
def test_empty_dataset_rejected(self):
|
||||
"""Empty dataset stored → detect_entity_column returns error."""
|
||||
store = SessionStore()
|
||||
empty = pl.DataFrame({':ips': pl.Series([], dtype=pl.Utf8)})
|
||||
schema = {}
|
||||
store.store_dataset(
|
||||
'empty_ds', empty.lazy(), schema=schema,
|
||||
metadata={'row_count': 0},
|
||||
)
|
||||
result = detect_entity_column('empty_ds')
|
||||
# May fail or return empty candidates; either is acceptable
|
||||
# but it should not crash with an unhandled exception.
|
||||
assert 'error' in result or result['recommended'] == ''
|
||||
|
||||
def test_single_entity_clustering(self):
|
||||
"""Single entity → aggregation works but clustering may not split."""
|
||||
data = pl.DataFrame({
|
||||
':ips': ['10.0.0.1'] * 20,
|
||||
':ipd': ['10.0.0.2'] * 20,
|
||||
'8ack': [100] * 20,
|
||||
'4dur': [1.0] * 20,
|
||||
'1ipp': ['TCP'] * 20,
|
||||
'0ver': ['TLSv1.2'] * 20,
|
||||
'cipher_suite': ['AES'] * 20,
|
||||
'snam': ['host1.example.com'] * 20,
|
||||
'timestamp': ['2024-01-01 00:00:00'] * 20,
|
||||
':prd': [443] * 20,
|
||||
})
|
||||
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
|
||||
lf = data.lazy()
|
||||
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
|
||||
df_agg = agg_lf.collect()
|
||||
|
||||
assert len(df_agg) == 1
|
||||
assert df_agg['flow_count'][0] == 20
|
||||
|
||||
def test_aggregate_preserves_schema_types(self):
|
||||
"""Aggregated columns should have expected Polars dtypes."""
|
||||
data = pl.DataFrame({
|
||||
':ips': ['10.0.0.1'] * 10,
|
||||
'8ack': [100] * 10,
|
||||
'4dur': [1.5] * 10,
|
||||
})
|
||||
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
|
||||
lf = data.lazy()
|
||||
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
|
||||
df = agg_lf.collect()
|
||||
|
||||
assert df['flow_count'].dtype == pl.UInt32 or df['flow_count'].dtype == pl.Int64
|
||||
@@ -0,0 +1,322 @@
|
||||
"""Tests for entity detection (entity_detector) and aggregation (entity_aggregator).
|
||||
|
||||
Every test isolates the SessionStore singleton via the ``clean_store`` fixture.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import polars as pl
|
||||
|
||||
from analysis.session_store import SessionStore
|
||||
from analysis.entity_detector import _score_name, detect_entity_column
|
||||
from analysis.entity_aggregator import (
|
||||
_find_column,
|
||||
aggregate_by_entity,
|
||||
build_aggregation_params,
|
||||
ip_to_subnet,
|
||||
)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# Fixtures
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_store():
|
||||
"""Reset the SessionStore singleton before and after each test."""
|
||||
store = SessionStore()
|
||||
store.drop_all()
|
||||
yield
|
||||
store.drop_all()
|
||||
|
||||
|
||||
def _store_dataset(
|
||||
df: pl.DataFrame,
|
||||
dataset_id: str = 'test_ds',
|
||||
) -> str:
|
||||
"""Helper: store a DataFrame as a dataset in the session store.
|
||||
|
||||
Returns the dataset_id.
|
||||
"""
|
||||
store = SessionStore()
|
||||
schema = {name: str(dtype) for name, dtype in zip(df.columns, df.dtypes)}
|
||||
store.store_dataset(
|
||||
dataset_id=dataset_id,
|
||||
lazyframe=df.lazy(),
|
||||
schema=schema,
|
||||
metadata={'row_count': len(df)},
|
||||
)
|
||||
return dataset_id
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# _score_name (unit-level)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestScoreName:
|
||||
"""_score_name() scores column names against entity keywords."""
|
||||
|
||||
def test_exact_match(self):
|
||||
"""Exact keyword match → 3.0."""
|
||||
score, kw = _score_name(':ips')
|
||||
assert score == 3.0
|
||||
assert kw == ':ips'
|
||||
|
||||
def test_suffix_match(self):
|
||||
"""Suffix match via _ separator → 3.0."""
|
||||
score, kw = _score_name('my_snam')
|
||||
assert score == 3.0
|
||||
assert kw == 'snam'
|
||||
|
||||
def test_prefix_match(self):
|
||||
"""Prefix match via _ separator → 3.0."""
|
||||
score, kw = _score_name('snam_value')
|
||||
assert score == 3.0
|
||||
assert kw == 'snam'
|
||||
|
||||
def test_substring_not_matched(self):
|
||||
"""Substring-only match now returns 0.0 (exact matching only)."""
|
||||
score, kw = _score_name('myipaddr')
|
||||
assert score == 0.0
|
||||
assert kw == ''
|
||||
|
||||
def test_no_match(self):
|
||||
score, kw = _score_name('random_column_xyz')
|
||||
assert score == 0.0
|
||||
assert kw == ''
|
||||
|
||||
def test_hyphen_normalised(self):
|
||||
"""Hyphen normalised to underscore for matching."""
|
||||
score, kw = _score_name('cipher-suite')
|
||||
assert score == 3.0
|
||||
assert kw == 'cipher_suite'
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# detect_entity_column (integration-level)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestDetectEntityColumn:
|
||||
"""detect_entity_column() should find entity columns from stored data."""
|
||||
|
||||
def test_detect_ips(self):
|
||||
"""Dataset with ``:ips`` → recommended is ``:ips``."""
|
||||
data = pl.DataFrame({
|
||||
':ips': [f'10.0.0.{i}' for i in range(1, 51)] * 10,
|
||||
':ipd': [f'10.0.1.{i}' for i in range(1, 501)],
|
||||
'8ack': [100] * 500,
|
||||
})
|
||||
ds_id = _store_dataset(data)
|
||||
result = detect_entity_column(ds_id)
|
||||
|
||||
assert result['recommended'] == ':ips'
|
||||
assert len(result['candidates']) > 0
|
||||
assert result['candidates'][0]['column'] == ':ips'
|
||||
assert result['total_rows'] == 500
|
||||
assert result['scored_columns'] >= 1
|
||||
|
||||
def test_detect_snam(self):
|
||||
"""Datasets with ``snam`` but no IP column → snam is recommended."""
|
||||
data = pl.DataFrame({
|
||||
'snam': [f'host{i}.example.com' for i in range(100)],
|
||||
'4dur': [float(i) for i in range(100)],
|
||||
})
|
||||
ds_id = _store_dataset(data)
|
||||
result = detect_entity_column(ds_id)
|
||||
|
||||
assert result['recommended'] == 'snam'
|
||||
assert result['candidates'][0]['matched_keyword'] == 'snam'
|
||||
|
||||
def test_no_entity_column(self):
|
||||
"""No entity keywords matched → candidates still found but matched_keyword empty."""
|
||||
data = pl.DataFrame({
|
||||
'col_a': [str(i) for i in range(100)],
|
||||
'col_b': [str(i * 2) for i in range(100)],
|
||||
'value_f64': [float(i) for i in range(100)],
|
||||
})
|
||||
ds_id = _store_dataset(data)
|
||||
result = detect_entity_column(ds_id)
|
||||
|
||||
# String columns are candidates (detector scores on uniqueness, not just keywords)
|
||||
assert len(result['candidates']) >= 2
|
||||
assert all(c['matched_keyword'] == '' for c in result['candidates'])
|
||||
# recommended will be the highest-scoring string column
|
||||
assert result['recommended'] in ('col_a', 'col_b')
|
||||
|
||||
def test_all_numeric_no_strings(self):
|
||||
"""No string/categorical columns → no candidates."""
|
||||
data = pl.DataFrame({
|
||||
'value': list(range(100)),
|
||||
'flag': [0, 1] * 50,
|
||||
})
|
||||
ds_id = _store_dataset(data)
|
||||
result = detect_entity_column(ds_id)
|
||||
|
||||
assert result['recommended'] == ''
|
||||
assert len(result['candidates']) == 0
|
||||
|
||||
def test_dataset_not_found(self):
|
||||
"""Unknown dataset_id → error response."""
|
||||
result = detect_entity_column('nonexistent_ds')
|
||||
assert 'error' in result
|
||||
assert result['recommended'] == ''
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# _find_column (helper for aggregation)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestFindColumn:
|
||||
"""_find_column() matches keywords against schema column names."""
|
||||
|
||||
ACK_KEYWORDS = ('8ack',)
|
||||
|
||||
def test_exact_match(self):
|
||||
schema = {'8ack': 'Int64', 'other': 'Utf8'}
|
||||
col = _find_column(schema, self.ACK_KEYWORDS)
|
||||
assert col == '8ack'
|
||||
|
||||
def test_no_match(self):
|
||||
schema = {'something_else': 'Int64'}
|
||||
col = _find_column(schema, self.ACK_KEYWORDS)
|
||||
assert col is None
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# aggregate_by_entity
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestAggregateByEntity:
|
||||
"""aggregate_by_entity() groups flows and computes per-entity features."""
|
||||
|
||||
def test_two_entities(self):
|
||||
"""50 flows across 2 entities → 2 aggregated rows with correct counts."""
|
||||
data = pl.DataFrame({
|
||||
':ips': ['10.0.0.1'] * 20 + ['10.0.0.2'] * 30,
|
||||
':ipd': ['10.0.1.1'] * 10 + ['10.0.1.2'] * 10
|
||||
+ ['10.0.1.3'] * 30,
|
||||
':prd': [80] * 20 + [443] * 30,
|
||||
'8ack': [100] * 20 + [200] * 30,
|
||||
'4dur': [1.0] * 20 + [2.0] * 30,
|
||||
'0ver': ['TLSv1.2'] * 20 + ['TLSv1.3'] * 30,
|
||||
'cipher_suite': ['AES'] * 20 + ['CHACHA20'] * 30,
|
||||
'1ipp': ['TCP'] * 50,
|
||||
'timestamp': ['2024-01-01 00:00:00'] * 50,
|
||||
})
|
||||
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
|
||||
lf = data.lazy()
|
||||
|
||||
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
|
||||
df = agg_lf.collect()
|
||||
|
||||
# Two entities
|
||||
assert len(df) == 2
|
||||
|
||||
# Flow counts
|
||||
df_sorted = df.sort(':ips')
|
||||
assert df_sorted['flow_count'][0] == 20 # 10.0.0.1
|
||||
assert df_sorted['flow_count'][1] == 30 # 10.0.0.2
|
||||
|
||||
# Bytes — each column is now perfectly aligned with entity groups
|
||||
assert df_sorted['total_bytes_sent'][0] == 20 * 100
|
||||
assert df_sorted['total_bytes_sent'][1] == 30 * 200
|
||||
|
||||
# Core feature columns present
|
||||
assert 'flow_count' in feature_cols
|
||||
assert 'total_bytes_sent' in feature_cols
|
||||
assert 'avg_duration' in feature_cols
|
||||
assert 'unique_dst_ips' in feature_cols
|
||||
assert 'unique_dst_ports' in feature_cols
|
||||
assert 'unique_protocols' in feature_cols
|
||||
assert 'first_seen' in feature_cols
|
||||
assert 'last_seen' in feature_cols
|
||||
|
||||
def test_single_entity(self):
|
||||
"""Single entity → single aggregated row."""
|
||||
data = pl.DataFrame({
|
||||
':ips': ['10.0.0.1'] * 5,
|
||||
'8ack': [100] * 5,
|
||||
':ipd': ['10.0.0.2'] * 5,
|
||||
})
|
||||
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
|
||||
lf = data.lazy()
|
||||
agg_lf, _ = aggregate_by_entity(lf, ':ips', schema)
|
||||
df = agg_lf.collect()
|
||||
assert len(df) == 1
|
||||
assert df['flow_count'][0] == 5
|
||||
|
||||
def test_nonexistent_entity_column(self):
|
||||
"""Unknown entity column → ValueError."""
|
||||
data = pl.DataFrame({'a': [1, 2, 3]})
|
||||
schema = {'a': 'Int64'}
|
||||
lf = data.lazy()
|
||||
with pytest.raises(ValueError, match=r"Entity column\(s\) '\[\'xyz\'\]'"):
|
||||
aggregate_by_entity(lf, 'xyz', schema)
|
||||
|
||||
def test_schema_with_missing_columns(self):
|
||||
"""Aggregation still works when some optional columns are missing."""
|
||||
data = pl.DataFrame({
|
||||
':ips': ['10.0.0.1'] * 3 + ['10.0.0.2'] * 3,
|
||||
':ipd': ['10.0.0.3'] * 6,
|
||||
# No 8ack, 4dur, etc.
|
||||
})
|
||||
schema = {name: str(dtype) for name, dtype in zip(data.columns, data.dtypes)}
|
||||
lf = data.lazy()
|
||||
agg_lf, feature_cols = aggregate_by_entity(lf, ':ips', schema)
|
||||
|
||||
df = agg_lf.collect()
|
||||
assert len(df) == 2
|
||||
assert df['flow_count'][0] == 3
|
||||
# Missing features should have None values
|
||||
assert df['total_bytes_sent'][0] is None
|
||||
assert 'flow_count' in feature_cols
|
||||
assert 'unique_dst_ips' in feature_cols
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# build_aggregation_params
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestBuildAggregationParams:
|
||||
"""build_aggregation_params() builds correct expressions per schema."""
|
||||
|
||||
def test_flow_count_always_present(self):
|
||||
params = build_aggregation_params({'a': 'Int64'})
|
||||
names = [p[0] for p in params]
|
||||
assert 'flow_count' in names
|
||||
|
||||
def test_ack_detected(self):
|
||||
params = build_aggregation_params({'8ack': 'Int64'})
|
||||
names = [p[0] for p in params]
|
||||
assert 'total_bytes_sent' in names
|
||||
|
||||
def test_ack_not_detected(self):
|
||||
params = build_aggregation_params({'x': 'Int64'})
|
||||
param_map = dict(params)
|
||||
assert param_map['total_bytes_sent'] is not None # still has a fallback expr
|
||||
# The value should be a Polars literal (None)
|
||||
names = [p[0] for p in params]
|
||||
assert 'total_bytes_sent' in names
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# ip_to_subnet (IP subnet mask conversion)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestIpToSubnet:
|
||||
"""ip_to_subnet() converts IP strings to subnet prefixes."""
|
||||
|
||||
def test_ip_to_subnet_24(self):
|
||||
"""ip_to_subnet with /24 mask."""
|
||||
assert ip_to_subnet('10.0.1.15', 24) == '10.0.1.0/24'
|
||||
assert ip_to_subnet('192.168.1.255', 24) == '192.168.1.0/24'
|
||||
|
||||
def test_ip_to_subnet_28(self):
|
||||
"""ip_to_subnet with /28 mask."""
|
||||
assert ip_to_subnet('10.0.1.15', 28) == '10.0.1.0/28'
|
||||
assert ip_to_subnet('10.0.1.250', 28) == '10.0.1.240/28'
|
||||
|
||||
def test_ip_to_subnet_invalid(self):
|
||||
"""ip_to_subnet with invalid inputs."""
|
||||
assert ip_to_subnet(None, 24) is None
|
||||
assert ip_to_subnet('', 24) == ''
|
||||
assert ip_to_subnet('not_an_ip', 24) == 'not_an_ip'
|
||||
@@ -0,0 +1,421 @@
|
||||
"""Tests for the data type classifier engine (type_classifier) and IP
|
||||
clustering utilities (ip_clustering).
|
||||
|
||||
Covers: INT, FLOAT, ENUM, HEX, URL, IPv4, BOOL_ENUM detection via both
|
||||
name-based and value-based heuristics, config override, schema-level
|
||||
classification, and ``/24`` subnet grouping.
|
||||
"""
|
||||
import pytest
|
||||
import polars as pl
|
||||
|
||||
from analysis.type_classifier import (
|
||||
DataType,
|
||||
classify_column,
|
||||
classify_schema,
|
||||
HEX_PATTERN,
|
||||
URL_PATTERN,
|
||||
IPv4_PATTERN,
|
||||
BOOL_ENUM_VALUES,
|
||||
LAT_LON_NAMES,
|
||||
)
|
||||
from analysis.ip_clustering import ip_to_int, get_subnet, cluster_ips
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — INT
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyInt:
|
||||
def test_int_series(self):
|
||||
s = pl.Series('bytes_sent', [100, 200, 300])
|
||||
assert classify_column('bytes_sent', s) == DataType.INT
|
||||
|
||||
def test_unsigned_int(self):
|
||||
s = pl.Series('packets', [0, 1, 255, 65535])
|
||||
assert classify_column('packets', s) == DataType.INT
|
||||
|
||||
def test_small_int64(self):
|
||||
s = pl.Series('count', [1, 2, 3], dtype=pl.Int64)
|
||||
assert classify_column('count', s) == DataType.INT
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — FLOAT
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyFloat:
|
||||
def test_float_series(self):
|
||||
s = pl.Series('duration', [1.5, 2.7, 0.3])
|
||||
assert classify_column('duration', s) == DataType.FLOAT
|
||||
|
||||
def test_float_dtype(self):
|
||||
s = pl.Series('score', [0.1, 0.2], dtype=pl.Float32)
|
||||
assert classify_column('score', s) == DataType.FLOAT
|
||||
|
||||
def test_float_with_ints(self):
|
||||
"""Values look like ints but dtype is float → still FLOAT."""
|
||||
s = pl.Series('ratio', [1.0, 2.0, 3.0])
|
||||
assert classify_column('ratio', s) == DataType.FLOAT
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — ENUM (< 20 unique values)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyEnum:
|
||||
def test_few_unique_strings(self):
|
||||
s = pl.Series('tls_version', ['TLSv1.2', 'TLSv1.3', 'TLSv1.2'])
|
||||
assert classify_column('tls_version', s) == DataType.ENUM
|
||||
|
||||
def test_exactly_19_unique(self):
|
||||
vals = [str(i) for i in range(19)]
|
||||
s = pl.Series('version_code', vals * 2)
|
||||
assert classify_column('version_code', s) == DataType.ENUM
|
||||
|
||||
def test_many_unique_strings(self):
|
||||
vals = [f'str_{i}' for i in range(50)]
|
||||
s = pl.Series('desc', vals)
|
||||
assert classify_column('desc', s) == DataType.STRING
|
||||
|
||||
def test_name_version_hint(self):
|
||||
"""Column name contains 'version' → name heuristic returns ENUM."""
|
||||
s = pl.Series('tls_ver', ['1.2', '1.3', '1.2'])
|
||||
assert classify_column('tls_ver', s) == DataType.ENUM
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — HEX ("0e a6 3f" format)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyHex:
|
||||
def test_single_hex_pair(self):
|
||||
s = pl.Series('cipher', ['0e', 'a6', '3f'])
|
||||
assert classify_column('cipher', s) == DataType.HEX
|
||||
|
||||
def test_multiple_hex_pairs(self):
|
||||
s = pl.Series('cipher_suite', ['0e a6 3f', 'ab cd ef 01'])
|
||||
assert classify_column('cipher_suite', s) == DataType.HEX
|
||||
|
||||
def test_name_cipher_hint(self):
|
||||
"""Column name containing 'cipher' triggers name heuristic."""
|
||||
s = pl.Series('tls_cipher', ['ab cd', 'ef 01'])
|
||||
assert classify_column('tls_cipher', s) == DataType.HEX
|
||||
|
||||
def test_name_suite_hint(self):
|
||||
s = pl.Series('suite_id', ['00 01', '00 02'])
|
||||
assert classify_column('suite_id', s) == DataType.HEX
|
||||
|
||||
def test_invalid_hex_not_matched(self):
|
||||
"""Not all values are valid hex pairs → should not be HEX."""
|
||||
s = pl.Series('x', ['0e a6', 'zz yy'])
|
||||
result = classify_column('x', s)
|
||||
assert result != DataType.HEX
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — URL
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyUrl:
|
||||
def test_https(self):
|
||||
s = pl.Series('dst_url', ['https://example.com', 'https://test.org'])
|
||||
assert classify_column('dst_url', s) == DataType.URL
|
||||
|
||||
def test_http(self):
|
||||
s = pl.Series('url', ['http://example.com/path', 'http://test.org'])
|
||||
assert classify_column('url', s) == DataType.URL
|
||||
|
||||
def test_name_url_hint(self):
|
||||
s = pl.Series('request_uri', ['https://api.example.com/v1'])
|
||||
assert classify_column('request_uri', s) == DataType.URL
|
||||
|
||||
def test_not_url(self):
|
||||
"""Plain domain names without http:// should not be URL."""
|
||||
s = pl.Series('domain', ['example.com', 'test.org'])
|
||||
assert classify_column('domain', s) != DataType.URL
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — IPv4
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyIPv4:
|
||||
def test_ip_addresses(self):
|
||||
s = pl.Series('ip', ['192.168.1.1', '10.0.0.1', '172.16.0.1'])
|
||||
assert classify_column('ip', s) == DataType.IPv4
|
||||
|
||||
def test_ip_in_name(self):
|
||||
s = pl.Series('src_ip', ['10.0.0.1', '10.0.0.2'])
|
||||
# Name-based heuristic triggers first
|
||||
assert classify_column('src_ip', s) == DataType.IPv4
|
||||
|
||||
def test_addr_suffix(self):
|
||||
s = pl.Series('destination_addr', ['1.2.3.4', '5.6.7.8'])
|
||||
assert classify_column('destination_addr', s) == DataType.IPv4
|
||||
|
||||
def test_invalid_octet_range(self):
|
||||
"""Octet > 255 → should not match IPv4 via values."""
|
||||
s = pl.Series('x', ['192.168.1.256', '10.0.0.1'])
|
||||
result = classify_column('x', s)
|
||||
assert result != DataType.IPv4 # 256 invalid, may fall through
|
||||
|
||||
def test_ip_plus_normal_strings(self):
|
||||
"""Mix of IPs and non-IPs → not IPv4."""
|
||||
s = pl.Series('x', ['192.168.1.1', 'hello'])
|
||||
result = classify_column('x', s)
|
||||
assert result != DataType.IPv4
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — BOOL_ENUM (+, '', true/false, etc.)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyBoolEnum:
|
||||
def test_plus_and_blank(self):
|
||||
s = pl.Series('flag', ['+', '', '+', ' '])
|
||||
assert classify_column('flag', s) == DataType.BOOL_ENUM
|
||||
|
||||
def test_true_false(self):
|
||||
s = pl.Series('active', ['true', 'false', 'true'])
|
||||
assert classify_column('active', s) == DataType.BOOL_ENUM
|
||||
|
||||
def test_yes_no(self):
|
||||
s = pl.Series('enabled', ['yes', 'no', 'yes'])
|
||||
assert classify_column('enabled', s) == DataType.BOOL_ENUM
|
||||
|
||||
def test_01_strings(self):
|
||||
s = pl.Series('ok', ['0', '1', '0'])
|
||||
assert classify_column('ok', s) == DataType.BOOL_ENUM
|
||||
|
||||
def test_boolean_dtype(self):
|
||||
"""Native pl.Boolean dtype → BOOL_ENUM."""
|
||||
s = pl.Series('flag', [True, False, True])
|
||||
assert classify_column('flag', s) == DataType.BOOL_ENUM
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — LAT_LON (name-based)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyLatLon:
|
||||
def test_lat_column(self):
|
||||
s = pl.Series('lat', [35.0, 36.5])
|
||||
assert classify_column('lat', s) == DataType.LAT_LON
|
||||
|
||||
def test_latitude_column(self):
|
||||
s = pl.Series('latitude', [40.0, 41.0])
|
||||
assert classify_column('latitude', s) == DataType.LAT_LON
|
||||
|
||||
def test_lon_column(self):
|
||||
s = pl.Series('lon', [120.0, 121.5])
|
||||
assert classify_column('lon', s) == DataType.LAT_LON
|
||||
|
||||
def test_longitude_column(self):
|
||||
s = pl.Series('longitude', [122.0, 123.0])
|
||||
assert classify_column('longitude', s) == DataType.LAT_LON
|
||||
|
||||
def test_src_latitude_column(self):
|
||||
"""Column name src_latitude → LAT_LON via prefix heuristic."""
|
||||
s = pl.Series('src_latitude', [35.0, 36.0, 37.0])
|
||||
assert classify_column('src_latitude', s) == DataType.LAT_LON
|
||||
|
||||
def test_dst_longitude_column(self):
|
||||
"""Column name dst_longitude → LAT_LON via prefix heuristic."""
|
||||
s = pl.Series('dst_longitude', [120.0, 121.0, 122.0])
|
||||
assert classify_column('dst_longitude', s) == DataType.LAT_LON
|
||||
|
||||
def test_value_based_latlon_strings(self):
|
||||
"""String column with mix of '+', '', and valid floats → LAT_LON."""
|
||||
s = pl.Series('coord', ['+', '', '12.345', '-45.678', '90.0'])
|
||||
assert classify_column('coord', s) == DataType.LAT_LON
|
||||
|
||||
def test_value_based_latlon_rejects_out_of_range(self):
|
||||
"""String column with values > 180 → not LAT_LON."""
|
||||
s = pl.Series('coord', ['200', '300', '400'])
|
||||
result = classify_column('coord', s)
|
||||
assert result != DataType.LAT_LON
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — Config override (highest priority)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestConfigOverride:
|
||||
def test_config_overrides_value(self):
|
||||
"""config_type='int' forces INT even if values are float."""
|
||||
s = pl.Series('anything', [1.5, 2.5, 3.5])
|
||||
assert classify_column('anything', s, config_type='int') == DataType.INT
|
||||
|
||||
def test_config_overrides_name(self):
|
||||
"""config_type='string' forces STRING even if name matches IP."""
|
||||
s = pl.Series('src_ip', ['192.168.1.1', '10.0.0.1'])
|
||||
assert classify_column('src_ip', s, config_type='string') == DataType.STRING
|
||||
|
||||
def test_config_type_case_insensitive(self):
|
||||
s = pl.Series('x', [1, 2])
|
||||
assert classify_column('x', s, config_type='FLOAT') == DataType.FLOAT
|
||||
|
||||
def test_config_type_trimmed(self):
|
||||
s = pl.Series('x', ['a', 'b'])
|
||||
assert classify_column('x', s, config_type=' enum ') == DataType.ENUM
|
||||
|
||||
def test_unknown_config_type_falls_through(self):
|
||||
"""Unknown config_type string → ignored, falls to name/value."""
|
||||
s = pl.Series('bytes_sent', [1, 2, 3])
|
||||
result = classify_column('bytes_sent', s, config_type='unknown_type')
|
||||
assert result == DataType.INT # value-based
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_schema (full schema classification)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifySchema:
|
||||
def test_classify_all_columns(self):
|
||||
data = {
|
||||
'src_ip': ['10.0.0.1', '10.0.0.2', '10.0.0.3'],
|
||||
'dst_ip': ['10.0.1.1', '10.0.1.2', '10.0.1.3'],
|
||||
'bytes_sent': [100, 200, 300],
|
||||
'duration': [1.5, 2.5, 0.5],
|
||||
'tls_version': ['TLSv1.2', 'TLSv1.3', 'TLSv1.2'],
|
||||
'cipher': ['0e a6 3f', 'ab cd ef', '01 02'],
|
||||
'url': ['https://a.com', 'https://b.com', 'https://c.com'],
|
||||
'lat': [35.0, 36.0, 37.0],
|
||||
'lon': [120.0, 121.0, 122.0],
|
||||
}
|
||||
lf = pl.DataFrame(data).lazy()
|
||||
type_map = classify_schema(lf)
|
||||
|
||||
assert type_map['src_ip'] == DataType.IPv4
|
||||
assert type_map['dst_ip'] == DataType.IPv4
|
||||
assert type_map['bytes_sent'] == DataType.INT
|
||||
assert type_map['duration'] == DataType.FLOAT
|
||||
assert type_map['tls_version'] == DataType.ENUM
|
||||
assert type_map['cipher'] == DataType.HEX
|
||||
assert type_map['url'] == DataType.URL
|
||||
assert type_map['lat'] == DataType.LAT_LON
|
||||
assert type_map['lon'] == DataType.LAT_LON
|
||||
|
||||
def test_classify_with_config_overrides(self):
|
||||
data = {'src_ip': ['10.0.0.1', '10.0.0.2']}
|
||||
lf = pl.DataFrame(data).lazy()
|
||||
overrides = {'src_ip': 'string'}
|
||||
type_map = classify_schema(lf, config_overrides=overrides)
|
||||
assert type_map['src_ip'] == DataType.STRING
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# IP clustering utilities
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestIpToInt:
|
||||
def test_simple(self):
|
||||
assert ip_to_int('192.168.1.1') == 3232235777
|
||||
|
||||
def test_localhost(self):
|
||||
assert ip_to_int('127.0.0.1') == 2130706433
|
||||
|
||||
def test_invalid(self):
|
||||
assert ip_to_int('not_an_ip') is None
|
||||
|
||||
def test_empty_string(self):
|
||||
assert ip_to_int('') is None
|
||||
|
||||
def test_out_of_range(self):
|
||||
assert ip_to_int('999.999.999.999') is None
|
||||
|
||||
|
||||
class TestGetSubnet:
|
||||
def test_24_mask(self):
|
||||
assert get_subnet('192.168.1.5', 24) == '192.168.1.0/24'
|
||||
|
||||
def test_16_mask(self):
|
||||
assert get_subnet('10.1.2.3', 16) == '10.1.0.0/16'
|
||||
|
||||
def test_8_mask(self):
|
||||
assert get_subnet('100.200.1.1', 8) == '100.0.0.0/8'
|
||||
|
||||
def test_invalid_ip(self):
|
||||
assert get_subnet('not_an_ip', 24) is None
|
||||
|
||||
def test_default_mask(self):
|
||||
"""Default mask is /24."""
|
||||
assert get_subnet('10.0.0.15') == '10.0.0.0/24'
|
||||
|
||||
|
||||
class TestClusterIps:
|
||||
def test_single_subnet(self):
|
||||
ips = ['10.0.0.1', '10.0.0.2', '10.0.0.3']
|
||||
groups = cluster_ips(ips, mask=24)
|
||||
assert len(groups) == 1
|
||||
assert '10.0.0.0/24' in groups
|
||||
assert len(groups['10.0.0.0/24']) == 3
|
||||
|
||||
def test_multiple_subnets(self):
|
||||
ips = ['10.0.0.1', '10.0.1.1', '192.168.1.1']
|
||||
groups = cluster_ips(ips, mask=24)
|
||||
assert len(groups) == 3
|
||||
|
||||
def test_invalid_ips_grouped(self):
|
||||
ips = ['10.0.0.1', 'bad_ip', 'also_invalid']
|
||||
groups = cluster_ips(ips, mask=24)
|
||||
assert '10.0.0.0/24' in groups
|
||||
assert '__invalid__' in groups
|
||||
assert len(groups['__invalid__']) == 2
|
||||
|
||||
def test_empty_list(self):
|
||||
groups = cluster_ips([], mask=24)
|
||||
assert groups == {}
|
||||
|
||||
def test_same_subnet_then_16(self):
|
||||
"""10.0.x.x IPs with /16 mask all go to one subnet."""
|
||||
ips = ['10.0.1.1', '10.0.2.2', '10.0.255.255']
|
||||
groups = cluster_ips(ips, mask=16)
|
||||
assert len(groups) == 1
|
||||
assert '10.0.0.0/16' in groups
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — MAC address (should NOT be HEX)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyMac:
|
||||
def test_mac_address_detection(self):
|
||||
"""MAC addresses should NOT be classified as HEX."""
|
||||
s = pl.Series('mac', ['aa:bb:cc:dd:ee:ff', '00:1a:2b:3c:4d:5e'])
|
||||
result = classify_column('mac', s)
|
||||
assert result != DataType.HEX, f'MAC should not be HEX, got {result}'
|
||||
# MAC with unique addresses → STRING (high cardinality)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — PORT values (INT range → ENUM)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyPort:
|
||||
def test_port_values_detection(self):
|
||||
"""Integer values in 1-65535 range should be ENUM."""
|
||||
s = pl.Series('port', ['80', '443', '8080', '8443'])
|
||||
assert classify_column('port', s) == DataType.ENUM
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# classify_column — Value-based IPv4 (name-agnostic)
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestClassifyValueBased:
|
||||
def test_value_based_ipv4(self):
|
||||
"""IP detection should work from values alone."""
|
||||
s = pl.Series('col', ['10.0.0.1', '192.168.1.1', '8.8.8.8'])
|
||||
assert classify_column('col', s) == DataType.IPv4
|
||||
|
||||
def test_value_based_url(self):
|
||||
"""URL detection should work from values."""
|
||||
s = pl.Series('col', ['https://example.com', 'http://test.org/path'])
|
||||
assert classify_column('col', s) == DataType.URL
|
||||
|
||||
def test_value_first_over_name(self):
|
||||
"""Value pattern should take priority over name heuristic."""
|
||||
# Column named "version" with IP-like values should be IPv4, not ENUM
|
||||
s = pl.Series('version', ['10.0.0.1', '192.168.1.1'])
|
||||
assert classify_column('version', s) == DataType.IPv4
|
||||
@@ -0,0 +1,537 @@
|
||||
"""Tests for analysis.value_normalizer — TLS value normalisation.
|
||||
|
||||
Covers: TLS version (hex & display), cipher suite (hex & display),
|
||||
named curve (hex & display), CNRS/ISRS, and the top-level
|
||||
:func:`normalize_lf` integration.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import polars as pl
|
||||
|
||||
from analysis.value_normalizer import (
|
||||
TLS_VERSION_MAP,
|
||||
CIPHER_SUITE_MAP,
|
||||
CURVE_MAP,
|
||||
normalize_lf,
|
||||
)
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# TLS Version
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormTlsVersion:
|
||||
"""_norm_tls_version — hex & display → canonical enum name."""
|
||||
|
||||
def test_hex_03_04_tls_1_3(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 04']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_3'
|
||||
|
||||
def test_hex_03_03_tls_1_2(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 03']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_2'
|
||||
|
||||
def test_hex_03_02_tls_1_1(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 02']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_1'
|
||||
|
||||
def test_hex_03_01_tls_1_0(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 01']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_0'
|
||||
|
||||
def test_hex_02_00_ssl_2_0(self):
|
||||
lf = pl.LazyFrame({'0ver': ['02 00']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_2_0'
|
||||
|
||||
def test_hex_03_00_ssl_3_0(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 00']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_3_0'
|
||||
|
||||
def test_display_tlsv1_3(self):
|
||||
lf = pl.LazyFrame({'0ver': ['TLSv1.3']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_3'
|
||||
|
||||
def test_display_tlsv1_2(self):
|
||||
lf = pl.LazyFrame({'0ver': ['tlsv1.2']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_2'
|
||||
|
||||
def test_display_tls_1_1(self):
|
||||
lf = pl.LazyFrame({'0ver': ['TLS 1.1']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_1'
|
||||
|
||||
def test_display_bare_1_3(self):
|
||||
lf = pl.LazyFrame({'0ver': ['1.3']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_3'
|
||||
|
||||
def test_display_bare_1_2(self):
|
||||
lf = pl.LazyFrame({'0ver': ['1.2']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_2'
|
||||
|
||||
def test_display_tlsv1(self):
|
||||
lf = pl.LazyFrame({'0ver': ['TLSv1']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'tls_1_0'
|
||||
|
||||
def test_display_sslv2(self):
|
||||
lf = pl.LazyFrame({'0ver': ['SSLv2']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_2_0'
|
||||
|
||||
def test_display_sslv3(self):
|
||||
lf = pl.LazyFrame({'0ver': ['sslv3']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_3_0'
|
||||
|
||||
def test_display_bare_2(self):
|
||||
lf = pl.LazyFrame({'0ver': ['2']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_2_0'
|
||||
|
||||
def test_display_bare_3(self):
|
||||
lf = pl.LazyFrame({'0ver': ['3']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'ssl_3_0'
|
||||
|
||||
def test_case_insensitive(self):
|
||||
lf = pl.LazyFrame({'0ver': ['tlsv1.3', 'TLSv1.3', 'TlsV1.3']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'].to_list() == ['tls_1_3', 'tls_1_3', 'tls_1_3']
|
||||
|
||||
def test_unknown_version(self):
|
||||
lf = pl.LazyFrame({'0ver': ['ff ff']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'unknown_version'
|
||||
|
||||
def test_unknown_display(self):
|
||||
lf = pl.LazyFrame({'0ver': ['TLSv9.9']})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] == 'unknown_version'
|
||||
|
||||
def test_null_remains_null(self):
|
||||
lf = pl.LazyFrame({'0ver': [None]})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'][0] is None
|
||||
|
||||
def test_mixed_values(self):
|
||||
lf = pl.LazyFrame({'0ver': ['03 04', 'tlsv1.2', '1.3', 'ssl 2', None]})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert result['0ver_norm'].to_list() == [
|
||||
'tls_1_3', 'tls_1_2', 'tls_1_3', 'ssl_2_0', None,
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# Cipher Suite — hex
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormCipher:
|
||||
"""_norm_cipher — hex cipher suite → canonical name."""
|
||||
|
||||
def test_hex_13_01(self):
|
||||
lf = pl.LazyFrame({'0cph': ['13 01']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_aes_128_gcm_sha256'
|
||||
|
||||
def test_hex_13_02(self):
|
||||
lf = pl.LazyFrame({'0cph': ['13 02']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_aes_256_gcm_sha384'
|
||||
|
||||
def test_hex_13_03(self):
|
||||
lf = pl.LazyFrame({'0cph': ['13 03']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_chacha20_poly1305_sha256'
|
||||
|
||||
def test_hex_c0_2b(self):
|
||||
lf = pl.LazyFrame({'0cph': ['c0 2b']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
|
||||
|
||||
def test_hex_00_35(self):
|
||||
lf = pl.LazyFrame({'0cph': ['00 35']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_rsa_aes_128_cbc_sha'
|
||||
|
||||
def test_hex_00_2f(self):
|
||||
"""00 2f also maps to tls_rsa_aes_128_cbc_sha (same canonical)."""
|
||||
lf = pl.LazyFrame({'0cph': ['00 2f']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_rsa_aes_128_cbc_sha'
|
||||
|
||||
def test_unknown_hex(self):
|
||||
lf = pl.LazyFrame({'0cph': ['ab cd']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'unknown_cipher_0xab cd'
|
||||
|
||||
def test_null_remains_null(self):
|
||||
lf = pl.LazyFrame({'0cph': [None]})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] is None
|
||||
|
||||
def test_case_insensitive_hex(self):
|
||||
lf = pl.LazyFrame({'0cph': ['C0 2B']})
|
||||
result = normalize_lf(lf, {'0cph': None}).collect()
|
||||
assert result['0cph_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# Cipher Suite — display name
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormCipherDisplay:
|
||||
"""_norm_cipher_display — IANA/display name → canonical name."""
|
||||
|
||||
def test_iana_full_name(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['TLS_AES_128_GCM_SHA256']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
|
||||
|
||||
def test_iana_full_name_lowercase(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['tls_aes_256_gcm_sha384']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_aes_256_gcm_sha384'
|
||||
|
||||
def test_display_aes128gcm(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['AES 128 GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
|
||||
|
||||
def test_display_aes256gcm(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['AES 256 GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_aes_256_gcm_sha384'
|
||||
|
||||
def test_display_chacha20(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['CHACHA20-POLY1305']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_chacha20_poly1305_sha256'
|
||||
|
||||
def test_display_ecdhe_ecdsa_aes128gcm(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-ECDSA-AES128-GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_ecdsa_aes_128_gcm_sha256'
|
||||
|
||||
def test_display_ecdhe_rsa_aes256gcm(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-RSA-AES256-GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_rsa_aes_256_gcm_sha384'
|
||||
|
||||
def test_display_rsa_aes128gcm(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['RSA AES 128 GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_rsa_aes_128_gcm_sha256'
|
||||
|
||||
def test_display_hyphen_variants(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['ECDHE-RSA-AES128-GCM']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'tls_ecdhe_rsa_aes_128_gcm_sha256'
|
||||
|
||||
def test_unknown_display(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': ['CUSTOM_CIPHER_999']})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] == 'customcipher999_unknown'
|
||||
|
||||
def test_null_remains_null(self):
|
||||
lf = pl.LazyFrame({'cipher-suite': [None]})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
assert result['cipher_suite_norm'][0] is None
|
||||
|
||||
def test_mixed_display_values(self):
|
||||
lf = pl.LazyFrame({
|
||||
'cipher-suite': [
|
||||
'TLS_AES_128_GCM_SHA256',
|
||||
'AES 256 GCM',
|
||||
'ECDHE-RSA-AES128-GCM',
|
||||
'bogus',
|
||||
None,
|
||||
],
|
||||
})
|
||||
result = normalize_lf(lf, {'cipher-suite': None}).collect()
|
||||
expected = [
|
||||
'tls_aes_128_gcm_sha256',
|
||||
'tls_aes_256_gcm_sha384',
|
||||
'tls_ecdhe_rsa_aes_128_gcm_sha256',
|
||||
'bogus_unknown',
|
||||
None,
|
||||
]
|
||||
assert result['cipher_suite_norm'].to_list() == expected
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# Named Curve — hex
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormCurve:
|
||||
"""_norm_curve — hex curve → canonical name."""
|
||||
|
||||
def test_hex_00_1d_x25519(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 1d']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'x25519'
|
||||
|
||||
def test_hex_00_17_secp256r1(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 17']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_hex_00_18_secp384r1(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 18']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'secp384r1'
|
||||
|
||||
def test_hex_00_19_secp521r1(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 19']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'secp521r1'
|
||||
|
||||
def test_hex_00_1e_x448(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 1e']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'x448'
|
||||
|
||||
def test_unknown_hex_curve(self):
|
||||
lf = pl.LazyFrame({'0crv': ['ff ff']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'unknown_curve_0xff ff'
|
||||
|
||||
def test_null_remains_null(self):
|
||||
lf = pl.LazyFrame({'0crv': [None]})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] is None
|
||||
|
||||
def test_case_insensitive_hex_curve(self):
|
||||
lf = pl.LazyFrame({'0crv': ['00 1D']})
|
||||
result = normalize_lf(lf, {'0crv': None}).collect()
|
||||
assert result['0crv_norm'][0] == 'x25519'
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# Named Curve — display name
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormCurveDisplay:
|
||||
"""_norm_curve_display — display name → canonical name."""
|
||||
|
||||
def test_x25519(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['X25519']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
|
||||
|
||||
def test_curve25519(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['Curve25519']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
|
||||
|
||||
def test_secp256r1(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['secp256r1']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_prime256v1(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['prime256v1']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_p256(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P256']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_p_256(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P-256']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_p_256_underscore(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P_256']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp256r1'
|
||||
|
||||
def test_secp384r1(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['secp384r1']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
|
||||
|
||||
def test_p384(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P384']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
|
||||
|
||||
def test_p_384(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['p-384']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp384r1'
|
||||
|
||||
def test_secp521r1(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['Secp521r1']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp521r1'
|
||||
|
||||
def test_p521(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P521']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'secp521r1'
|
||||
|
||||
def test_x448(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['x448']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'x448'
|
||||
|
||||
def test_curve448(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['Curve448']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'x448'
|
||||
|
||||
def test_unknown_curve_display(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['brainpoolP256r1']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'unknown_curve_brainpoolp256r1'
|
||||
|
||||
def test_null_remains_null(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': [None]})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'][0] is None
|
||||
|
||||
def test_case_insensitive_p256(self):
|
||||
lf = pl.LazyFrame({'ecdhe-named-curve': ['P-256', 'p_256', 'P256']})
|
||||
result = normalize_lf(lf, {'ecdhe-named-curve': None}).collect()
|
||||
assert result['ecdhe_named_curve_norm'].to_list() == [
|
||||
'secp256r1', 'secp256r1', 'secp256r1',
|
||||
]
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# CNRS / ISRS
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormCnrs:
|
||||
"""_norm_cnrs — '+' → 'yes', blank/null → 'no'."""
|
||||
|
||||
def test_plus_yes(self):
|
||||
lf = pl.LazyFrame({'cnrs': ['+']})
|
||||
result = normalize_lf(lf, {'cnrs': None}).collect()
|
||||
assert result['cnrs_norm'][0] == 'yes'
|
||||
|
||||
def test_blank_no(self):
|
||||
lf = pl.LazyFrame({'cnrs': ['']})
|
||||
result = normalize_lf(lf, {'cnrs': None}).collect()
|
||||
assert result['cnrs_norm'][0] == 'no'
|
||||
|
||||
def test_null_no(self):
|
||||
lf = pl.LazyFrame({'cnrs': [None]})
|
||||
result = normalize_lf(lf, {'cnrs': None}).collect()
|
||||
assert result['cnrs_norm'][0] == 'no'
|
||||
|
||||
def test_whitespace_plus(self):
|
||||
lf = pl.LazyFrame({'cnrs': [' + ']})
|
||||
result = normalize_lf(lf, {'cnrs': None}).collect()
|
||||
assert result['cnrs_norm'][0] == 'yes'
|
||||
|
||||
def test_anything_else_no(self):
|
||||
lf = pl.LazyFrame({'cnrs': ['-', 'true', 'maybe', '0', '1']})
|
||||
result = normalize_lf(lf, {'cnrs': None}).collect()
|
||||
assert result['cnrs_norm'].to_list() == ['no', 'no', 'no', 'no', 'no']
|
||||
|
||||
def test_isrs_plus_yes(self):
|
||||
lf = pl.LazyFrame({'isrs': ['+']})
|
||||
result = normalize_lf(lf, {'isrs': None}).collect()
|
||||
assert result['isrs_norm'][0] == 'yes'
|
||||
|
||||
def test_isrs_blank_no(self):
|
||||
lf = pl.LazyFrame({'isrs': ['']})
|
||||
result = normalize_lf(lf, {'isrs': None}).collect()
|
||||
assert result['isrs_norm'][0] == 'no'
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
# Integration — normalize_lf
|
||||
# ═════════════════════════════════════════════════════════════════════════
|
||||
|
||||
class TestNormalizeLf:
|
||||
"""Top-level normalize_lf — column routing & no-op cases."""
|
||||
|
||||
def test_all_columns_at_once(self):
|
||||
lf = pl.LazyFrame({
|
||||
'0ver': ['03 04'],
|
||||
'0cph': ['13 01'],
|
||||
'cipher-suite': ['TLS_AES_128_GCM_SHA256'],
|
||||
'0crv': ['00 1d'],
|
||||
'ecdhe-named-curve': ['X25519'],
|
||||
'cnrs': ['+'],
|
||||
'isrs': [''],
|
||||
})
|
||||
type_map = {
|
||||
'0ver': None,
|
||||
'0cph': None,
|
||||
'cipher-suite': None,
|
||||
'0crv': None,
|
||||
'ecdhe-named-curve': None,
|
||||
'cnrs': None,
|
||||
'isrs': None,
|
||||
}
|
||||
result = normalize_lf(lf, type_map).collect()
|
||||
|
||||
assert result['0ver_norm'][0] == 'tls_1_3'
|
||||
assert result['0cph_norm'][0] == 'tls_aes_128_gcm_sha256'
|
||||
assert result['cipher_suite_norm'][0] == 'tls_aes_128_gcm_sha256'
|
||||
assert result['0crv_norm'][0] == 'x25519'
|
||||
assert result['ecdhe_named_curve_norm'][0] == 'x25519'
|
||||
assert result['cnrs_norm'][0] == 'yes'
|
||||
assert result['isrs_norm'][0] == 'no'
|
||||
|
||||
def test_no_matching_columns(self):
|
||||
"""Empty type_map → no _norm columns added."""
|
||||
lf = pl.LazyFrame({'0ver': ['03 04']})
|
||||
result = normalize_lf(lf, {}).collect()
|
||||
assert '0ver_norm' not in result.columns
|
||||
assert result['0ver'][0] == '03 04'
|
||||
|
||||
def test_subset_of_columns(self):
|
||||
"""Only columns present in type_map get normalised."""
|
||||
lf = pl.LazyFrame({
|
||||
'0ver': ['03 04'],
|
||||
'0cph': ['13 01'],
|
||||
'cnrs': ['+'],
|
||||
})
|
||||
result = normalize_lf(lf, {'0ver': None, 'cnrs': None}).collect()
|
||||
assert '0ver_norm' in result.columns
|
||||
assert 'cnrs_norm' in result.columns
|
||||
assert '0cph_norm' not in result.columns
|
||||
|
||||
def test_preserves_original_columns(self):
|
||||
"""Original columns are kept unchanged."""
|
||||
lf = pl.LazyFrame({
|
||||
'0ver': ['03 04'],
|
||||
'src_ip': ['10.0.0.1'],
|
||||
})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
assert '0ver' in result.columns
|
||||
assert 'src_ip' in result.columns
|
||||
assert result['src_ip'][0] == '10.0.0.1'
|
||||
|
||||
def test_duplicate_normalized_column_exists(self):
|
||||
"""If a _norm column already exists, Polars appends suffixed."""
|
||||
lf = pl.LazyFrame({
|
||||
'0ver': ['03 04'],
|
||||
'0ver_norm': ['old'],
|
||||
})
|
||||
result = normalize_lf(lf, {'0ver': None}).collect()
|
||||
# Polars will create 0ver_norm with new value
|
||||
assert result['0ver_norm'][0] == 'tls_1_3'
|
||||
|
||||
def test_lazyness_preserved(self):
|
||||
"""normalize_lf returns a LazyFrame, not materialised."""
|
||||
lf = pl.LazyFrame({'0ver': ['03 04']})
|
||||
result = normalize_lf(lf, {'0ver': None})
|
||||
assert isinstance(result, pl.LazyFrame)
|
||||
@@ -0,0 +1,85 @@
|
||||
"""LLM orchestrator optimized for 32B model via MCP tool calling.
|
||||
Short prompts, aggressive truncation, step-by-step guidance."""
|
||||
import json, logging, time, urllib.request, urllib.error, asyncio
|
||||
from typing import Optional
|
||||
from analysis.tool_registry import handle_call
|
||||
|
||||
logger = logging.getLogger('tianxuan.llm')
|
||||
|
||||
TOOLS = [
|
||||
{"type": "function", "function": {"name": "profile_data",
|
||||
"description": "Examine dataset columns and statistics",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}}, "required": ["dataset_id"]}}},
|
||||
{"type": "function", "function": {"name": "build_entity_profiles",
|
||||
"description": "Group flows by entity for user behavior analysis",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "auto_detect": {"type": "boolean"}}, "required": ["dataset_id"]}}},
|
||||
{"type": "function", "function": {"name": "run_clustering",
|
||||
"description": "Cluster entities into behavior groups",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_columns": {"type": "array", "items": {"type": "string"}}, "algorithm": {"type": "string", "enum": ["hdbscan", "kmeans"]}}, "required": ["dataset_id", "cluster_columns"]}}},
|
||||
{"type": "function", "function": {"name": "extract_features",
|
||||
"description": "Identify distinguishing features per cluster",
|
||||
"parameters": {"type": "object", "properties": {"dataset_id": {"type": "string"}, "cluster_result_id": {"type": "string"}, "method": {"type": "string", "enum": ["zscore", "anova"]}}, "required": ["dataset_id", "cluster_result_id"]}}},
|
||||
]
|
||||
|
||||
class LLMConfig:
|
||||
def __init__(self, base_url="", api_key="", model="deepseek-v4-flash", max_tokens=2048, temperature=0.1):
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.model = model
|
||||
self.max_tokens = max_tokens
|
||||
self.temperature = temperature
|
||||
|
||||
def call_llm(messages, config):
|
||||
payload = json.dumps({"model": config.model, "messages": messages, "tools": TOOLS,
|
||||
"tool_choice": "auto", "max_tokens": config.max_tokens, "temperature": config.temperature}).encode()
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if config.api_key: headers["Authorization"] = f"Bearer {config.api_key}"
|
||||
req = urllib.request.Request(f"{config.base_url}/chat/completions", data=payload, headers=headers, method='POST')
|
||||
logger.info(f'[LLM] model={config.model}')
|
||||
start = time.monotonic()
|
||||
with urllib.request.urlopen(req, timeout=90) as resp:
|
||||
d = json.loads(resp.read().decode())
|
||||
logger.info(f'[LLM] latency_ms={int((time.monotonic()-start)*1000)}')
|
||||
return d
|
||||
|
||||
def run_llm_pipeline(dataset_id, entity_col="", config=None, callback=None):
|
||||
if config is None:
|
||||
from config import get_config; cfg = get_config()
|
||||
config = LLMConfig(cfg.llm.base_url, cfg.llm.api_key, cfg.llm.model)
|
||||
msgs = [{"role": "system", "content": "Analyze dataset step by step using the available tools."},
|
||||
{"role": "user", "content": f"Profile dataset {dataset_id}, then build entities, cluster, and extract features."}]
|
||||
guides = ["", "Good. Now build entity profiles.", "Good. Now run clustering.", "Good. Now extract features.", "Good. Summarize results."]
|
||||
gi = 0
|
||||
for step in range(15):
|
||||
if step > 0 and gi < len(guides) and guides[gi]:
|
||||
msgs.append({"role": "user", "content": guides[gi]})
|
||||
gi += 1
|
||||
if callback:
|
||||
callback(step, "thinking", None)
|
||||
resp = call_llm(msgs, config)
|
||||
if not resp:
|
||||
if callback:
|
||||
callback(step, "error", "LLM call failed")
|
||||
return {"error": "LLM call failed"}
|
||||
msg = resp["choices"][0]["message"]
|
||||
if "tool_calls" not in msg or not msg["tool_calls"]:
|
||||
if callback:
|
||||
callback(step, "complete", msg.get("content", ""))
|
||||
return {"result": msg.get("content","") or "Complete", "steps": step+1}
|
||||
msgs.append(msg)
|
||||
for tc in msg["tool_calls"]:
|
||||
f = tc.get("function",{}); name = f.get("name","")
|
||||
try: args = json.loads(f.get("arguments","{}"))
|
||||
except: args = {}
|
||||
logger.info(f'[TOOL] {name}')
|
||||
if callback:
|
||||
callback(step, name, None)
|
||||
try: result = asyncio.run(handle_call(name, args))
|
||||
except Exception as e: result = {"error": str(e)}
|
||||
if callback:
|
||||
callback(step, name, result)
|
||||
msgs.append({"role": "tool", "tool_call_id": tc.get("id",""),
|
||||
"content": json.dumps(result, default=str, ensure_ascii=False)[:1200]})
|
||||
if callback:
|
||||
callback(15, "max_steps", None)
|
||||
return {"error": "Max steps", "steps": 15}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user