244967bad5
fix(cluster_detail): show raw data rows instead of entity profiles
2026-07-24 11:58:31 +08:00
d2e34ca43d
fix(cluster_detail): always persist per-row cluster labels to EntityProfile for data row display
2026-07-24 11:56:02 +08:00
b135b18779
fix(i18n): full Chinese translation of cluster_detail page
2026-07-24 11:53:43 +08:00
2c1aad6bd1
feat(run_detail): global SVD feature analysis + LLM workflow timeline
2026-07-24 11:51:23 +08:00
5280ce5819
fix(umap): LazyFrame.sample not available in Polars 1.42, use DataFrame.sample after collect
2026-07-24 11:39:00 +08:00
e2c64fd068
fix: raw row clustering — remove entity concept, use row index, all numeric features
2026-07-24 11:35:18 +08:00
fa13ebcf72
fix(auto): markdown rendering for LLM answer and thinking blocks
2026-07-24 11:29:02 +08:00
d7f9db098c
fix(llm): save intermediate thinking — DeepSeek reasoning_content fallback, force non-empty thought, increase API truncation to 50K
2026-07-24 11:26:34 +08:00
982cdcf83e
chore: sync gitignore from master (GeoIP chunks, mmdb, playwright-mcp)
2026-07-24 11:22:34 +08:00
8e320b0059
chore: add Playwright test screenshots, mass test generator, update gitignore
2026-07-23 23:39:35 +08:00
1105f12d89
merge: beta-clean refactoring — views/tools package split, code cleanup, perf, docs
2026-07-23 23:39:22 +08:00
e340b49c71
chore: gitignore test CSV directory (too large for push)
2026-07-23 23:39:20 +08:00
9508cc8d58
perf: optimize for 25M rows on 8GB RAM
...
- globe.py: add streaming=True to all lf.head().collect() calls (3 places)
- entities.py: chunked aggregation (100K batches) for datasets > 1M rows
- constants.py: reduce MAX_SAMPLE_ROWS 50000->30000 for 8GB RAM
- tools/__init__.py: add __all__ for '*' re-export (fix tool_registry import)
- clustering.py: fix misplaced logger/import traceback syntax errors
- analysis.py: fix indentation of pass statement
- anomalies.py: fix misplaced import traceback syntax error
2026-07-23 22:54:45 +08:00
88b26f16fe
docs: rewrite all markdown documentation for views/tools package refactoring
...
- Update README.md: new architecture (views/ 12 modules, tools/ 17 modules), remove simple_analysis refs, 30 MCP tools, 8GB RAM spec
- Update AGENTS.md: project structure tree, 30 tools organized by implementation module, remove entity_detector/entity_aggregator refs
- Update docs/工作流.md: full pipeline v3.0, code references to tools/ package, remove simple_analysis section (note moved to master)
- Update docs/故障诊断手册.md: fix code location references, add WebGL2 tips, remove Leaflet/simple_analysis refs
Branch: beta-clean
2026-07-23 22:51:36 +08:00
ef60e3e489
cleanup: consolidate TLS_HEX_MAP, extract constants, remove dead code, add traceback logging
...
3.1 Unified TLS_HEX_MAP: defined once in type_classifier.py, imported in globe.py and clustering.py
3.2 Created analysis/constants.py: centralized RANDOM_SEED, MAX_SAMPLE_ROWS, UMAP constants, GLOBE_MAX_ROWS, etc.
3.3 Removed 14 pointless try/except:raise dead code blocks across tools/ and views/
3.4 Added logger.error(traceback) to all bare except Exception: blocks in tools/ and views/
3.5 Moved inline import traceback/asyncio to top-level imports in features.py, clustering.py, pipeline.py
3.6 Removed auto_profile_module side-effect imports from 6 modules
2026-07-23 22:44:42 +08:00
885da89b6f
refactor: split tool_registry.py (3368 lines) into analysis/tools/ package
...
- 15 sub-modules: _helpers, _registry, _dispatch, load_data, profile,
filter, preprocess, clustering, evaluate, features, export, entities,
anomalies, diagnostics, analysis, data_mgmt, distance_matrix
- Extracted shared helpers: _resolve_dataset(), _count_rows(),
_get_numeric_columns(), NUMERIC_DTYPES, NUMERIC_TYPE_NAMES
- Extracted _cluster_core() for shared clustering pipeline
- Thin tool_registry.py wrapper preserves backward compatibility
- Old file preserved as tool_registry_backup.py
- manage.py check passes (0 issues)
2026-07-23 22:31:17 +08:00
d4c82768a8
refactor: split analysis/views.py into views/ package (12 sub-modules)
...
- helpers.py: _extract_lat, _extract_lon, plan index utilities
- dashboard.py: dashboard, run_list, run_detail, run_status_api, start_analysis
- pipeline.py: _run_pipeline_worker, _background_process
- clustering.py: cluster_overview, cluster_detail, _run_clustering_pipeline, _get_globe_flows
- entity.py: entity_profile
- upload.py: upload_page, upload_csv, upload_csv_batch, finalize_upload, delete_upload
- auto.py: auto_page, run_llm_analysis_view
- manual.py: manual_page, manual_run_analysis
- globe.py: globe_view, _extract_flows_from_df
- config.py: config_view, llm_test
- log_viewer.py: log_viewer
- tools.py: tool_lab, tool_lab_run, tool_plan, apply_filter, reload_run_data, retry_run
All function logic preserved exactly. views/__init__.py re-exports all public
functions for backward compatibility with urls.py and run_pipeline.py.
Original views.py archived as views_backup.py.
2026-07-23 22:22:28 +08:00
123db6b08f
chore: consolidate all session fixes — migrations squash, views/tools refactor, cluster UI, geoip upgrade, simple_analysis removal, non-blocking startup, chinese UI, API key removal
2026-07-23 22:10:26 +08:00
02176cf6f4
chore: remove config.yaml from tracking (contains API keys), add example config
2026-07-23 20:51:32 +08:00
8bac406116
fix: change default port from 80 to 8000 to avoid admin permission requirement on Windows
2026-07-23 13:29:33 +08:00
12cd607837
feat: v2.0.1beta - new preprocessing pipeline, 3D UMAP visualization, startup optimization, docs rewrite
2026-07-23 12:59:09 +08:00
7d6d29475d
fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True)
2026-07-22 23:39:37 +08:00
38e8985718
Merge branch 'master' of https://gitea.cattysteve.top/HJQ/tianxuan into beta
2026-07-22 23:38:45 +08:00
035d22b234
fix: make all entity_values unique by appending row index to support bulk_create with ignore_conflicts
2026-07-22 23:17:09 +08:00
0d92a5a170
fix: unique entity_value fallback to prevent bulk_create ignore_conflicts from skipping duplicates
2026-07-22 23:14:00 +08:00
c1db7ad7eb
fix: fallback entity value when column is empty to ensure all UMAP profiles are saved
2026-07-22 23:11:32 +08:00
9f6e388ece
chore: vendor Leaflet + Chart.js locally for offline/China use
...
- Download Leaflet CSS, JS, and marker images to static/vendor/
- Download Chart.js 4.4.7 to static/vendor/
- Switch template from CDN (unpkg/cdnjs) to local static assets
- Map now works fully offline without external CDN dependencies
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 23:06:35 +08:00
67013cf29b
fix: replace OSM tiles with ArcGIS for China network compatibility
...
- Switch primary tile provider from OpenStreetMap (blocked in China)
to ArcGIS World Street Map (certified working)
- Add fallback chain: ArcGIS -> ArcGIS Gray -> OSM
- Auto-detect tile failures and fall back to next provider
- Clean up init sequence for reliable map rendering
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 23:02:17 +08:00
eaf6cb4e67
Merge remote-tracking branch 'origin/master' into feature/simple-analysis
2026-07-22 22:55:20 +08:00
2c5aece65d
fix: streaming load_from_db with fetchmany(10000) instead of fetchall() (fixes OOM on 4M rows)
2026-07-22 22:39:51 +08:00
46013afa74
feat: add simple analysis module with global map visualization
...
- Add independent 'simple_analysis' Django app with full workflow:
1. Upload/merge CSV files with auto column detection
2. Filter by cnam + cnrs + isrs (optional)
3. Clustering via HDBSCAN+haversine (geo) or /24 subnet (IP)
4. Interactive Leaflet global map with cluster-colored data points
- Integrate into nav bar, settings, and URL routing without
modifying existing analysis logic
- Cluster detail panel with IPs, top dst IPs, TLS/cipher profiles
- Chart.js bar/pie charts for cluster size distribution
- E2E Playwright tests for all workflow steps
- Support multiple CSV upload with diagonal_relaxed merge
Co-Authored-By: Claude <noreply@anthropic.com >
2026-07-22 22:39:16 +08:00
29484ef033
fix: defer background processing until all upload batches complete (race condition fix)
...
Add finalize_upload endpoint that starts _background_process only after
all file batches have been saved to disk. Fixes race where glob(*.csv)
captured only first batch's files.
- upload_csv: no longer starts thread, sets status='pending'
- finalize_upload: validates dir, starts processing thread
- upload.html: calls finalize after batch upload loop completes
2026-07-22 22:34:54 +08:00
2c8d1e90ae
fix: per-file CSV scan for heterogeneous schemas, add upload file count logging
2026-07-22 22:31:49 +08:00
1df4c3c2eb
chore: bump VERSION to v2.0.0beta, reimport TlsDB, remove stale test files
2026-07-22 21:53:25 +08:00
edaca3b903
merge: sync with master (keep beta requirements with umap-learn)
2026-07-22 21:42:42 +08:00
7b9084b33a
chore: clean PCA references, integrate SVD pipeline, update AGENTS.md
2026-07-22 21:10:13 +08:00
585dbb4ec4
refactor: Wave 3 - UMAP-2D replace PCA scatter, TruncatedSVD replace PCA preprocessing, update AGENTS.md
2026-07-22 21:07:03 +08:00
0264bade2e
refactor: Wave 2 - auto.html remove dataset selector + timeline merge + file_count fix
2026-07-22 20:46:10 +08:00
2a64f554c4
fix: Wave 1 - 3 MCP crashes + delete whitelist + exception propagation
2026-07-22 20:41:25 +08:00
d9b79c74b4
chore: freeze requirements from runtime Python 3.12.2 (59 packages)
2026-07-22 20:31:37 +08:00
763fdbf605
chore: freeze requirements from runtime Python 3.12.2 (59 packages)
2026-07-22 20:31:09 +08:00
0baffe4d9e
update AGENTS.md
2026-07-22 19:58:50 +08:00
7aa847ba3e
v1.1.6: cleanup — schema sort fix, batch upload, temp dir to D:, update.bat wildcard fix
2026-07-22 19:10:39 +08:00
e4689ed0a9
fix: batch upload no longer re-triggers _background_process to avoid DROP TABLE race
2026-07-22 14:01:56 +08:00
a59c153f4c
fix: reset VERSION to v1.1.6
2026-07-22 13:59:29 +08:00
db0e473119
chore: bump to v1.1.7
2026-07-22 13:58:33 +08:00
edd942695b
Revert "chore: bump to v1.1.7"
...
This reverts commit 1ab655d17940aa0905c031753b89d67fb51a61e2.
2026-07-22 13:58:29 +08:00
ef27ddde4e
chore: bump to v1.1.7
2026-07-22 13:58:22 +08:00
8c90bad9f0
v1.1.6: batch upload BATCH=5 + temp dir to D: + schema sort + update.bat fix
2026-07-22 13:58:19 +08:00
2b3f10a03c
fix: set FILE_UPLOAD_TEMP_DIR to D: drive to avoid ENOSPC on C:
2026-07-22 13:50:18 +08:00
PM-pinou
TianXuan Developer