HJQ/tianxuan
2
1
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
107 Commits 5 Branches 0 Tags
66f68a20629b005b6bad65d1cbc148ef539eca21
Commit Graph

107 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
PM-pinou 66f68a2062 refactor(services): extract clustering pipeline from views/ to analysis/services/ 
Move _run_clustering_pipeline (~295 lines) from analysis/views/clustering.py
into analysis/services/clustering.py as three clean functions:
- run_clustering_pipeline: main pipeline orchestration
- compute_umap_embedding: UMAP 3D→2D fallback computation
- save_entity_profiles: persist cluster labels + UMAP coords to ORM

Views/clustering.py now delegates via thin wrapper. Callers (auto.py,
manual.py, run_pipeline.py) import directly from services.clustering.
 2026-07-24 13:27:11 +08:00
PM-pinou b768d02987 refactor: extract inline JS/CSS from templates into static files 
- static/tianxuan/globe_embed.js: Three.js globe scene (initGlobe function)
- static/tianxuan/timeline.js: LLM workflow timeline (buildTimeline, renderTimeline, escapeHtml, prettyJson, toggleStep)
- static/tianxuan/markdown.js: Markdown-to-HTML converter (renderMarkdown)
- static/tianxuan/cluster.css: Shared cluster page styles (pills, cards, feats, detail panel)

Updated templates:
- auto.html: use timeline.js, remove 140 lines of inline JS
- cluster_overview.html: use cluster.css, keep layout-specific styles
- cluster_detail.html: use cluster.css, keep entity-card styles
 2026-07-24 13:26:07 +08:00
PM-pinou e340b49c71 chore: gitignore test CSV directory (too large for push) 2026-07-23 23:39:20 +08:00
PM-pinou 9508cc8d58 perf: optimize for 25M rows on 8GB RAM 
- globe.py: add streaming=True to all lf.head().collect() calls (3 places)
- entities.py: chunked aggregation (100K batches) for datasets > 1M rows
- constants.py: reduce MAX_SAMPLE_ROWS 50000->30000 for 8GB RAM
- tools/__init__.py: add __all__ for '*' re-export (fix tool_registry import)
- clustering.py: fix misplaced logger/import traceback syntax errors
- analysis.py: fix indentation of pass statement
- anomalies.py: fix misplaced import traceback syntax error
 2026-07-23 22:54:45 +08:00
PM-pinou 88b26f16fe docs: rewrite all markdown documentation for views/tools package refactoring 
- Update README.md: new architecture (views/ 12 modules, tools/ 17 modules), remove simple_analysis refs, 30 MCP tools, 8GB RAM spec
- Update AGENTS.md: project structure tree, 30 tools organized by implementation module, remove entity_detector/entity_aggregator refs
- Update docs/工作流.md: full pipeline v3.0, code references to tools/ package, remove simple_analysis section (note moved to master)
- Update docs/故障诊断手册.md: fix code location references, add WebGL2 tips, remove Leaflet/simple_analysis refs

Branch: beta-clean
 2026-07-23 22:51:36 +08:00
PM-pinou ef60e3e489 cleanup: consolidate TLS_HEX_MAP, extract constants, remove dead code, add traceback logging 
3.1 Unified TLS_HEX_MAP: defined once in type_classifier.py, imported in globe.py and clustering.py
3.2 Created analysis/constants.py: centralized RANDOM_SEED, MAX_SAMPLE_ROWS, UMAP constants, GLOBE_MAX_ROWS, etc.
3.3 Removed 14 pointless try/except:raise dead code blocks across tools/ and views/
3.4 Added logger.error(traceback) to all bare except Exception: blocks in tools/ and views/
3.5 Moved inline import traceback/asyncio to top-level imports in features.py, clustering.py, pipeline.py
3.6 Removed auto_profile_module side-effect imports from 6 modules
 2026-07-23 22:44:42 +08:00
PM-pinou 885da89b6f refactor: split tool_registry.py (3368 lines) into analysis/tools/ package 
- 15 sub-modules: _helpers, _registry, _dispatch, load_data, profile,
  filter, preprocess, clustering, evaluate, features, export, entities,
  anomalies, diagnostics, analysis, data_mgmt, distance_matrix
- Extracted shared helpers: _resolve_dataset(), _count_rows(),
  _get_numeric_columns(), NUMERIC_DTYPES, NUMERIC_TYPE_NAMES
- Extracted _cluster_core() for shared clustering pipeline
- Thin tool_registry.py wrapper preserves backward compatibility
- Old file preserved as tool_registry_backup.py
- manage.py check passes (0 issues)
 2026-07-23 22:31:17 +08:00
PM-pinou d4c82768a8 refactor: split analysis/views.py into views/ package (12 sub-modules) 
- helpers.py: _extract_lat, _extract_lon, plan index utilities
- dashboard.py: dashboard, run_list, run_detail, run_status_api, start_analysis
- pipeline.py: _run_pipeline_worker, _background_process
- clustering.py: cluster_overview, cluster_detail, _run_clustering_pipeline, _get_globe_flows
- entity.py: entity_profile
- upload.py: upload_page, upload_csv, upload_csv_batch, finalize_upload, delete_upload
- auto.py: auto_page, run_llm_analysis_view
- manual.py: manual_page, manual_run_analysis
- globe.py: globe_view, _extract_flows_from_df
- config.py: config_view, llm_test
- log_viewer.py: log_viewer
- tools.py: tool_lab, tool_lab_run, tool_plan, apply_filter, reload_run_data, retry_run

All function logic preserved exactly. views/__init__.py re-exports all public
functions for backward compatibility with urls.py and run_pipeline.py.
Original views.py archived as views_backup.py.
 2026-07-23 22:22:28 +08:00
PM-pinou 123db6b08f chore: consolidate all session fixes — migrations squash, views/tools refactor, cluster UI, geoip upgrade, simple_analysis removal, non-blocking startup, chinese UI, API key removal 2026-07-23 22:10:26 +08:00
PM-pinou 02176cf6f4 chore: remove config.yaml from tracking (contains API keys), add example config 2026-07-23 20:51:32 +08:00
PM-pinou 8bac406116 fix: change default port from 80 to 8000 to avoid admin permission requirement on Windows 2026-07-23 13:29:33 +08:00
PM-pinou 12cd607837 feat: v2.0.1beta - new preprocessing pipeline, 3D UMAP visualization, startup optimization, docs rewrite 2026-07-23 12:59:09 +08:00
PM-pinou 7d6d29475d fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True) 2026-07-22 23:39:37 +08:00
PM-pinou 38e8985718 Merge branch 'master' of https://gitea.cattysteve.top/HJQ/tianxuan into beta 2026-07-22 23:38:45 +08:00
PM-pinou 035d22b234 fix: make all entity_values unique by appending row index to support bulk_create with ignore_conflicts 2026-07-22 23:17:09 +08:00
PM-pinou 0d92a5a170 fix: unique entity_value fallback to prevent bulk_create ignore_conflicts from skipping duplicates 2026-07-22 23:14:00 +08:00
PM-pinou c1db7ad7eb fix: fallback entity value when column is empty to ensure all UMAP profiles are saved 2026-07-22 23:11:32 +08:00
TianXuan Developer 9f6e388ece chore: vendor Leaflet + Chart.js locally for offline/China use 
- Download Leaflet CSS, JS, and marker images to static/vendor/
- Download Chart.js 4.4.7 to static/vendor/
- Switch template from CDN (unpkg/cdnjs) to local static assets
- Map now works fully offline without external CDN dependencies

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-22 23:06:35 +08:00
TianXuan Developer 67013cf29b fix: replace OSM tiles with ArcGIS for China network compatibility 
- Switch primary tile provider from OpenStreetMap (blocked in China)
  to ArcGIS World Street Map (certified working)
- Add fallback chain: ArcGIS -> ArcGIS Gray -> OSM
- Auto-detect tile failures and fall back to next provider
- Clean up init sequence for reliable map rendering

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-22 23:02:17 +08:00
TianXuan Developer eaf6cb4e67 Merge remote-tracking branch 'origin/master' into feature/simple-analysis 2026-07-22 22:55:20 +08:00
PM-pinou 2c5aece65d fix: streaming load_from_db with fetchmany(10000) instead of fetchall() (fixes OOM on 4M rows) 2026-07-22 22:39:51 +08:00
TianXuan Developer 46013afa74 feat: add simple analysis module with global map visualization 
- Add independent 'simple_analysis' Django app with full workflow:
  1. Upload/merge CSV files with auto column detection
  2. Filter by cnam + cnrs + isrs (optional)
  3. Clustering via HDBSCAN+haversine (geo) or /24 subnet (IP)
  4. Interactive Leaflet global map with cluster-colored data points
- Integrate into nav bar, settings, and URL routing without
  modifying existing analysis logic
- Cluster detail panel with IPs, top dst IPs, TLS/cipher profiles
- Chart.js bar/pie charts for cluster size distribution
- E2E Playwright tests for all workflow steps
- Support multiple CSV upload with diagonal_relaxed merge

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-22 22:39:16 +08:00
PM-pinou 29484ef033 fix: defer background processing until all upload batches complete (race condition fix) 
Add finalize_upload endpoint that starts _background_process only after
all file batches have been saved to disk. Fixes race where glob(*.csv)
captured only first batch's files.

- upload_csv: no longer starts thread, sets status='pending'
- finalize_upload: validates dir, starts processing thread
- upload.html: calls finalize after batch upload loop completes
 2026-07-22 22:34:54 +08:00
PM-pinou 2c8d1e90ae fix: per-file CSV scan for heterogeneous schemas, add upload file count logging 2026-07-22 22:31:49 +08:00
PM-pinou 1df4c3c2eb chore: bump VERSION to v2.0.0beta, reimport TlsDB, remove stale test files 2026-07-22 21:53:25 +08:00
PM-pinou edaca3b903 merge: sync with master (keep beta requirements with umap-learn) 2026-07-22 21:42:42 +08:00
PM-pinou 7b9084b33a chore: clean PCA references, integrate SVD pipeline, update AGENTS.md 2026-07-22 21:10:13 +08:00
PM-pinou 585dbb4ec4 refactor: Wave 3 - UMAP-2D replace PCA scatter, TruncatedSVD replace PCA preprocessing, update AGENTS.md 2026-07-22 21:07:03 +08:00
PM-pinou 0264bade2e refactor: Wave 2 - auto.html remove dataset selector + timeline merge + file_count fix 2026-07-22 20:46:10 +08:00
PM-pinou 2a64f554c4 fix: Wave 1 - 3 MCP crashes + delete whitelist + exception propagation 2026-07-22 20:41:25 +08:00
PM-pinou d9b79c74b4 chore: freeze requirements from runtime Python 3.12.2 (59 packages) 2026-07-22 20:31:37 +08:00
PM-pinou 763fdbf605 chore: freeze requirements from runtime Python 3.12.2 (59 packages) 2026-07-22 20:31:09 +08:00
PM-pinou 0baffe4d9e update AGENTS.md 2026-07-22 19:58:50 +08:00
PM-pinou 7aa847ba3e v1.1.6: cleanup — schema sort fix, batch upload, temp dir to D:, update.bat wildcard fix 2026-07-22 19:10:39 +08:00
PM-pinou e4689ed0a9 fix: batch upload no longer re-triggers _background_process to avoid DROP TABLE race 2026-07-22 14:01:56 +08:00
PM-pinou a59c153f4c fix: reset VERSION to v1.1.6 2026-07-22 13:59:29 +08:00
PM-pinou db0e473119 chore: bump to v1.1.7 2026-07-22 13:58:33 +08:00
PM-pinou edd942695b Revert "chore: bump to v1.1.7" 
This reverts commit 1ab655d17940aa0905c031753b89d67fb51a61e2.
 2026-07-22 13:58:29 +08:00
PM-pinou ef27ddde4e chore: bump to v1.1.7 2026-07-22 13:58:22 +08:00
PM-pinou 8c90bad9f0 v1.1.6: batch upload BATCH=5 + temp dir to D: + schema sort + update.bat fix 2026-07-22 13:58:19 +08:00
PM-pinou 2b3f10a03c fix: set FILE_UPLOAD_TEMP_DIR to D: drive to avoid ENOSPC on C: 2026-07-22 13:50:18 +08:00
PM-pinou f18f0d84cf fix: reduce upload batch size to 5 files 2026-07-22 13:48:23 +08:00
PM-pinou 9ca5da1237 release: v1.1.6 — update.bat wildcard fix + schema column-order fix + gen_test format alignment + pipeline verify 2026-07-22 11:39:35 +08:00
PM-pinou db82f58ff2 fix: reset VERSION to v1.1.4 2026-07-21 22:33:17 +08:00
PM-pinou 4a0296761b chore: bump to v1.1.5 2026-07-21 22:33:03 +08:00
PM-pinou e44e4ead08 fix: reset VERSION to v1.1.4 (post-build) 2026-07-21 22:15:15 +08:00
PM-pinou c75a423261 chore: bump to v1.1.5 2026-07-21 22:15:02 +08:00
PM-pinou d8eadafc83 auto: update for vv1.1.4 2026-07-21 22:14:55 +08:00
PM-pinou 4629416413 fix: gitignore data CSV files and untrack test data 2026-07-21 22:09:03 +08:00
PM-pinou 61f0174781 chore: bump to v1.1.5 2026-07-21 21:48:41 +08:00