PM-pinou
6cbbc24f14
Break monolithic distance.py into 8 focused sub-modules: - _geo.py: IP distance functions (subnet mask, haversine, geo) - _text.py: String distance (Levenshtein + enum) + bool distance - _numeric.py: Bytes/hex popcount + timestamp FFT distance - _normalize.py: StandardScaler normalization - _svd.py: svd_denoise (TruncatedSVD, returns noise_profile) - _umap.py: umap_reduce (3D→2D fallback) - _cluster_svd.py: cluster_svd_extract (per-cluster SVD) DELETE: run_preprocessing_pipeline (dead code, replaced by ColumnProcessor) All public APIs re-exported via __init__.py. Backward compatible.
38 lines
1.6 KiB
Python
38 lines
1.6 KiB
Python
"""Distance computation package for TianXuan analysis pipeline.
|
||
|
||
Provides distance functions for various data types (IP, string, boolean,
|
||
bytes, timestamp), plus normalisation, SVD denoising, UMAP reduction,
|
||
and per-cluster SVD feature extraction.
|
||
|
||
Public API:
|
||
ip_distance — subnet-mask + Haversine geographic distance for IP pairs
|
||
string_distance — Levenshtein / enum-like distance for string columns
|
||
bool_distance — 0/1 match-to-mode distance for boolean columns
|
||
bytes_distance — Hamming (popcount) distance for hex byte columns
|
||
timestamp_fft_distance — FFT phase distance for timestamp columns
|
||
normalize_features — StandardScaler normalisation (μ=0, σ=1)
|
||
svd_denoise — TruncatedSVD noise removal (returns noise_profile)
|
||
umap_reduce — UMAP dimensionality reduction (3D→2D fallback)
|
||
cluster_svd_extract — Per-cluster SVD distinguishing feature extraction
|
||
"""
|
||
|
||
from analysis.distance._geo import ip_distance
|
||
from analysis.distance._text import bool_distance, string_distance
|
||
from analysis.distance._numeric import bytes_distance, timestamp_fft_distance
|
||
from analysis.distance._normalize import normalize_features
|
||
from analysis.distance._svd import svd_denoise
|
||
from analysis.distance._umap import umap_reduce
|
||
from analysis.distance._cluster_svd import cluster_svd_extract
|
||
|
||
__all__ = [
|
||
"ip_distance",
|
||
"string_distance",
|
||
"bool_distance",
|
||
"bytes_distance",
|
||
"timestamp_fft_distance",
|
||
"normalize_features",
|
||
"svd_denoise",
|
||
"umap_reduce",
|
||
"cluster_svd_extract",
|
||
]
|