10c9c46a84
fix(upload): remove broken django.setup() from per-file thread
2026-07-24 23:18:16 +08:00
92e1305c07
fix(upload): per-file background thread — each CSV processed+deleted immediately, no batch accumulation
2026-07-24 22:55:09 +08:00
656f0ae2ab
fix(upload): auto-delete uploaded CSV files after SQLite ingestion to save C drive space
2026-07-24 22:36:30 +08:00
806cfc0a64
preview: add run_detail preview + cluster_detail variance text descriptions
2026-07-24 14:13:04 +08:00
e54499fa78
preview: static HTML preview files for cluster UI design validation
2026-07-24 14:07:59 +08:00
9f06e40a41
refactor: clean views/clustering.py and split data_loader.py into package
...
Task A: Replace _run_clustering_pipeline (533L->228L) — remove dead
docstring code, delegate to analysis.services.clustering.
Task B: Split analysis/data_loader.py (746L) into package:
analysis/data_loader/
__init__.py — re-exports all public symbols
_csv.py — load_csv_directory, BOM/encoding detection
_schema.py — schema merging and validation
_clean.py — numeric coercion helper
_sqlite.py — save_to_db, load_from_db, load_from_db_lazy, etc.
Keep old data_loader.py as thin re-export wrapper.
Verified: manage.py check (0 issues), run_pipeline (3 clusters, OK)
2026-07-24 13:52:07 +08:00
6cbbc24f14
refactor: split distance.py (1114 lines) into analysis/distance/ package
...
Break monolithic distance.py into 8 focused sub-modules:
- _geo.py: IP distance functions (subnet mask, haversine, geo)
- _text.py: String distance (Levenshtein + enum) + bool distance
- _numeric.py: Bytes/hex popcount + timestamp FFT distance
- _normalize.py: StandardScaler normalization
- _svd.py: svd_denoise (TruncatedSVD, returns noise_profile)
- _umap.py: umap_reduce (3D→2D fallback)
- _cluster_svd.py: cluster_svd_extract (per-cluster SVD)
DELETE: run_preprocessing_pipeline (dead code, replaced by ColumnProcessor)
All public APIs re-exported via __init__.py. Backward compatible.
2026-07-24 13:50:44 +08:00
c54917791e
cleanup: remove dead backup files, orphaned static files, fix merge conflict markers
2026-07-24 13:43:49 +08:00
3fd3896aba
merge: service layer + frontend static extraction — resolve clustering.py conflict
2026-07-24 13:30:59 +08:00
66f68a2062
refactor(services): extract clustering pipeline from views/ to analysis/services/
...
Move _run_clustering_pipeline (~295 lines) from analysis/views/clustering.py
into analysis/services/clustering.py as three clean functions:
- run_clustering_pipeline: main pipeline orchestration
- compute_umap_embedding: UMAP 3D→2D fallback computation
- save_entity_profiles: persist cluster labels + UMAP coords to ORM
Views/clustering.py now delegates via thin wrapper. Callers (auto.py,
manual.py, run_pipeline.py) import directly from services.clustering.
2026-07-24 13:27:11 +08:00
b768d02987
refactor: extract inline JS/CSS from templates into static files
...
- static/tianxuan/globe_embed.js: Three.js globe scene (initGlobe function)
- static/tianxuan/timeline.js: LLM workflow timeline (buildTimeline, renderTimeline, escapeHtml, prettyJson, toggleStep)
- static/tianxuan/markdown.js: Markdown-to-HTML converter (renderMarkdown)
- static/tianxuan/cluster.css: Shared cluster page styles (pills, cards, feats, detail panel)
Updated templates:
- auto.html: use timeline.js, remove 140 lines of inline JS
- cluster_overview.html: use cluster.css, keep layout-specific styles
- cluster_detail.html: use cluster.css, keep entity-card styles
2026-07-24 13:26:07 +08:00
908bfcaa32
refactor(types): ColumnType.distance() + ColumnProcessor dispatch, distance.py→TYPE_REGISTRY delegation
2026-07-24 13:15:34 +08:00
0ca1e87b46
wip: type system refactor — ColumnProcessor + distance dispatch
2026-07-24 13:13:51 +08:00
30e48ba282
perf: increase sampling coverage — MAX_SAMPLE_ROWS 30K→200K, LOW_MEMORY 100K→500K, UMAP_BATCH 1K→2K
2026-07-24 12:57:33 +08:00
df0f7d621f
refactor(types+svd): OOP type system + SVD denoise replaces columns instead of _svd_ append
2026-07-24 12:55:37 +08:00
92c1b0f768
feat(tools): LLM-mandated _timeout parameter on every MCP tool, enforced in dispatch layer
2026-07-24 12:45:29 +08:00
d24507e541
fix(llm): remove step limit and tool timeout, use while loop with 999 cap
2026-07-24 12:41:25 +08:00
f683a084f3
fix: nl_describe variance awareness + timestamp formatting + AGENTS.md update
2026-07-24 12:38:00 +08:00
1463e56a42
fix(auto): remove duplicate JS function definitions (126 lines dedup)
2026-07-24 12:00:21 +08:00
244967bad5
fix(cluster_detail): show raw data rows instead of entity profiles
2026-07-24 11:58:31 +08:00
d2e34ca43d
fix(cluster_detail): always persist per-row cluster labels to EntityProfile for data row display
2026-07-24 11:56:02 +08:00
b135b18779
fix(i18n): full Chinese translation of cluster_detail page
2026-07-24 11:53:43 +08:00
2c1aad6bd1
feat(run_detail): global SVD feature analysis + LLM workflow timeline
2026-07-24 11:51:23 +08:00
5280ce5819
fix(umap): LazyFrame.sample not available in Polars 1.42, use DataFrame.sample after collect
2026-07-24 11:39:00 +08:00
e2c64fd068
fix: raw row clustering — remove entity concept, use row index, all numeric features
2026-07-24 11:35:18 +08:00
fa13ebcf72
fix(auto): markdown rendering for LLM answer and thinking blocks
2026-07-24 11:29:02 +08:00
d7f9db098c
fix(llm): save intermediate thinking — DeepSeek reasoning_content fallback, force non-empty thought, increase API truncation to 50K
2026-07-24 11:26:34 +08:00
982cdcf83e
chore: sync gitignore from master (GeoIP chunks, mmdb, playwright-mcp)
2026-07-24 11:22:34 +08:00
8e320b0059
chore: add Playwright test screenshots, mass test generator, update gitignore
2026-07-23 23:39:35 +08:00
1105f12d89
merge: beta-clean refactoring — views/tools package split, code cleanup, perf, docs
2026-07-23 23:39:22 +08:00
e340b49c71
chore: gitignore test CSV directory (too large for push)
2026-07-23 23:39:20 +08:00
9508cc8d58
perf: optimize for 25M rows on 8GB RAM
...
- globe.py: add streaming=True to all lf.head().collect() calls (3 places)
- entities.py: chunked aggregation (100K batches) for datasets > 1M rows
- constants.py: reduce MAX_SAMPLE_ROWS 50000->30000 for 8GB RAM
- tools/__init__.py: add __all__ for '*' re-export (fix tool_registry import)
- clustering.py: fix misplaced logger/import traceback syntax errors
- analysis.py: fix indentation of pass statement
- anomalies.py: fix misplaced import traceback syntax error
2026-07-23 22:54:45 +08:00
88b26f16fe
docs: rewrite all markdown documentation for views/tools package refactoring
...
- Update README.md: new architecture (views/ 12 modules, tools/ 17 modules), remove simple_analysis refs, 30 MCP tools, 8GB RAM spec
- Update AGENTS.md: project structure tree, 30 tools organized by implementation module, remove entity_detector/entity_aggregator refs
- Update docs/工作流.md: full pipeline v3.0, code references to tools/ package, remove simple_analysis section (note moved to master)
- Update docs/故障诊断手册.md: fix code location references, add WebGL2 tips, remove Leaflet/simple_analysis refs
Branch: beta-clean
2026-07-23 22:51:36 +08:00
ef60e3e489
cleanup: consolidate TLS_HEX_MAP, extract constants, remove dead code, add traceback logging
...
3.1 Unified TLS_HEX_MAP: defined once in type_classifier.py, imported in globe.py and clustering.py
3.2 Created analysis/constants.py: centralized RANDOM_SEED, MAX_SAMPLE_ROWS, UMAP constants, GLOBE_MAX_ROWS, etc.
3.3 Removed 14 pointless try/except:raise dead code blocks across tools/ and views/
3.4 Added logger.error(traceback) to all bare except Exception: blocks in tools/ and views/
3.5 Moved inline import traceback/asyncio to top-level imports in features.py, clustering.py, pipeline.py
3.6 Removed auto_profile_module side-effect imports from 6 modules
2026-07-23 22:44:42 +08:00
885da89b6f
refactor: split tool_registry.py (3368 lines) into analysis/tools/ package
...
- 15 sub-modules: _helpers, _registry, _dispatch, load_data, profile,
filter, preprocess, clustering, evaluate, features, export, entities,
anomalies, diagnostics, analysis, data_mgmt, distance_matrix
- Extracted shared helpers: _resolve_dataset(), _count_rows(),
_get_numeric_columns(), NUMERIC_DTYPES, NUMERIC_TYPE_NAMES
- Extracted _cluster_core() for shared clustering pipeline
- Thin tool_registry.py wrapper preserves backward compatibility
- Old file preserved as tool_registry_backup.py
- manage.py check passes (0 issues)
2026-07-23 22:31:17 +08:00
d4c82768a8
refactor: split analysis/views.py into views/ package (12 sub-modules)
...
- helpers.py: _extract_lat, _extract_lon, plan index utilities
- dashboard.py: dashboard, run_list, run_detail, run_status_api, start_analysis
- pipeline.py: _run_pipeline_worker, _background_process
- clustering.py: cluster_overview, cluster_detail, _run_clustering_pipeline, _get_globe_flows
- entity.py: entity_profile
- upload.py: upload_page, upload_csv, upload_csv_batch, finalize_upload, delete_upload
- auto.py: auto_page, run_llm_analysis_view
- manual.py: manual_page, manual_run_analysis
- globe.py: globe_view, _extract_flows_from_df
- config.py: config_view, llm_test
- log_viewer.py: log_viewer
- tools.py: tool_lab, tool_lab_run, tool_plan, apply_filter, reload_run_data, retry_run
All function logic preserved exactly. views/__init__.py re-exports all public
functions for backward compatibility with urls.py and run_pipeline.py.
Original views.py archived as views_backup.py.
2026-07-23 22:22:28 +08:00
123db6b08f
chore: consolidate all session fixes — migrations squash, views/tools refactor, cluster UI, geoip upgrade, simple_analysis removal, non-blocking startup, chinese UI, API key removal
2026-07-23 22:10:26 +08:00
02176cf6f4
chore: remove config.yaml from tracking (contains API keys), add example config
2026-07-23 20:51:32 +08:00
8bac406116
fix: change default port from 80 to 8000 to avoid admin permission requirement on Windows
2026-07-23 13:29:33 +08:00
12cd607837
feat: v2.0.1beta - new preprocessing pipeline, 3D UMAP visualization, startup optimization, docs rewrite
2026-07-23 12:59:09 +08:00
7d6d29475d
fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True)
2026-07-22 23:39:37 +08:00
38e8985718
Merge branch 'master' of https://gitea.cattysteve.top/HJQ/tianxuan into beta
2026-07-22 23:38:45 +08:00
035d22b234
fix: make all entity_values unique by appending row index to support bulk_create with ignore_conflicts
2026-07-22 23:17:09 +08:00
0d92a5a170
fix: unique entity_value fallback to prevent bulk_create ignore_conflicts from skipping duplicates
2026-07-22 23:14:00 +08:00
c1db7ad7eb
fix: fallback entity value when column is empty to ensure all UMAP profiles are saved
2026-07-22 23:11:32 +08:00
9f6e388ece
chore: vendor Leaflet + Chart.js locally for offline/China use
...
- Download Leaflet CSS, JS, and marker images to static/vendor/
- Download Chart.js 4.4.7 to static/vendor/
- Switch template from CDN (unpkg/cdnjs) to local static assets
- Map now works fully offline without external CDN dependencies
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 23:06:35 +08:00
67013cf29b
fix: replace OSM tiles with ArcGIS for China network compatibility
...
- Switch primary tile provider from OpenStreetMap (blocked in China)
to ArcGIS World Street Map (certified working)
- Add fallback chain: ArcGIS -> ArcGIS Gray -> OSM
- Auto-detect tile failures and fall back to next provider
- Clean up init sequence for reliable map rendering
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 23:02:17 +08:00
eaf6cb4e67
Merge remote-tracking branch 'origin/master' into feature/simple-analysis
2026-07-22 22:55:20 +08:00
2c5aece65d
fix: streaming load_from_db with fetchmany(10000) instead of fetchall() (fixes OOM on 4M rows)
2026-07-22 22:39:51 +08:00
46013afa74
feat: add simple analysis module with global map visualization
...
- Add independent 'simple_analysis' Django app with full workflow:
1. Upload/merge CSV files with auto column detection
2. Filter by cnam + cnrs + isrs (optional)
3. Clustering via HDBSCAN+haversine (geo) or /24 subnet (IP)
4. Interactive Leaflet global map with cluster-colored data points
- Integrate into nav bar, settings, and URL routing without
modifying existing analysis logic
- Cluster detail panel with IPs, top dst IPs, TLS/cipher profiles
- Chart.js bar/pie charts for cluster size distribution
- E2E Playwright tests for all workflow steps
- Support multiple CSV upload with diagonal_relaxed merge
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 22:39:16 +08:00
PM-pinou
TianXuan Developer