9f06e40a41
refactor: clean views/clustering.py and split data_loader.py into package
...
Task A: Replace _run_clustering_pipeline (533L->228L) — remove dead
docstring code, delegate to analysis.services.clustering.
Task B: Split analysis/data_loader.py (746L) into package:
analysis/data_loader/
__init__.py — re-exports all public symbols
_csv.py — load_csv_directory, BOM/encoding detection
_schema.py — schema merging and validation
_clean.py — numeric coercion helper
_sqlite.py — save_to_db, load_from_db, load_from_db_lazy, etc.
Keep old data_loader.py as thin re-export wrapper.
Verified: manage.py check (0 issues), run_pipeline (3 clusters, OK)
2026-07-24 13:52:07 +08:00
df0f7d621f
refactor(types+svd): OOP type system + SVD denoise replaces columns instead of _svd_ append
2026-07-24 12:55:37 +08:00
ef60e3e489
cleanup: consolidate TLS_HEX_MAP, extract constants, remove dead code, add traceback logging
...
3.1 Unified TLS_HEX_MAP: defined once in type_classifier.py, imported in globe.py and clustering.py
3.2 Created analysis/constants.py: centralized RANDOM_SEED, MAX_SAMPLE_ROWS, UMAP constants, GLOBE_MAX_ROWS, etc.
3.3 Removed 14 pointless try/except:raise dead code blocks across tools/ and views/
3.4 Added logger.error(traceback) to all bare except Exception: blocks in tools/ and views/
3.5 Moved inline import traceback/asyncio to top-level imports in features.py, clustering.py, pipeline.py
3.6 Removed auto_profile_module side-effect imports from 6 modules
2026-07-23 22:44:42 +08:00
123db6b08f
chore: consolidate all session fixes — migrations squash, views/tools refactor, cluster UI, geoip upgrade, simple_analysis removal, non-blocking startup, chinese UI, API key removal
2026-07-23 22:10:26 +08:00
12cd607837
feat: v2.0.1beta - new preprocessing pipeline, 3D UMAP visualization, startup optimization, docs rewrite
2026-07-23 12:59:09 +08:00
7d6d29475d
fix: classify cnrs/isrs as BOOL_ENUM for proper boolean cleaning (blank→False, +→True)
2026-07-22 23:39:37 +08:00
2c5aece65d
fix: streaming load_from_db with fetchmany(10000) instead of fetchall() (fixes OOM on 4M rows)
2026-07-22 22:39:51 +08:00
6a633016d2
v1.1.4: gen_test reads TlsDB coverage + schema fixes
...
- gen_test_data.py: load_coverage() from TlsDB.csv, replaced all hardcoded _blank_or rates with coverage lookups
- gen_multi.py: coverage-based per-file column dropping
- collect_schema: force Utf8 for cross-file compatibility
- _background_process: pre-scan union columns, single CREATE TABLE, no ALTER TABLE
2026-07-21 19:34:14 +08:00
3617b7aabc
release: v1.1.3 — 13 fixes across build, frontend, pipeline, and data handling
...
Build: fix incremental package (update_info.json SHA256, __delete__, test exclude, files/ prefix), fix bump_version for 3-segment, PYTHONUTF8 in update.bat
Frontend: upload→manual redirect, LLM→cluster_overview redirect, filter UI before clustering, 地球→态势 rename, fix globe drag direction+inertia
Pipeline: LLM→auto clustering+PCA, column union handling (keep extra cols, no dropping), FFT spectral analysis tool
Quality: fix 24 bare except:pass patterns, enrich LLM context with TlsDB metadata, auto-save workflows with pinning, TlsDB-based column types
2026-07-21 12:01:52 +08:00
9314ac6942
chore: bump version to v1.1.2
...
- New timeline UI: interleaved LLM reasoning + tool call display
- Fix: run_log truncation for large tool outputs
- Fix: lat/lon + values reduced to ~1% (was 99%)
- Fix: @csrf_exempt on upload/delete endpoints
- Fix: DATA_UPLOAD_MAX_NUMBER_FILES for 1998-file upload
- Fix: dtype alignment for multi-batch CSV processing
- New: tests/test_e2e_full.py with real DeepSeek LLM
- New: 1998-file test data generator
2026-07-20 23:32:32 +08:00
efc6e3ab9d
v8: fix server startup, CSV parse, upload flow, load_from_db types, integration test
2026-07-20 19:40:49 +08:00
8fa36b774e
v7: 19 issues fixed — SQLite storage, new clustering pipeline, display_id, globe rewrite, Chinese tools
...
Key changes:
- New: data_loader.py SQLite persistence with drop_sqlite_table
- New: db_utils.py retry_on_lock decorator (3 retries, exponential backoff)
- New: tool_registry.py with 27 MCP tools (filter_and_cluster, compute_scores, etc.)
- New: tls_ref.py for TLS cipher/reference data
- New: import_tlsdb.py management command
- New: scripts/start_server.py for portable runtime
- New: migrations 0003-0007 for SQLite table, display_id, llm fields
- Changed: views.py unified pipeline worker, retry_run, display_id everywhere
- Changed: models.py with display_id auto-assignment, run_type, sqlite_table, llm_thinking, tool_calls_json
- Changed: urls.py added retry_run route
- Changed: session_store.py robust JSON persistence
- Changed: AGENTS.md v7 fix summary added
- Changed: templates — globe rewrite (inertia/polar flip), auto.html (thinking/tool accordions), base.html (toast/config), all pages use display_id
- Changed: run.bat PYTHONUTF8=1
- Deleted: entity_detector.py, entity_aggregator.py (replaced by filter_and_cluster clustering pipeline)
- Test: 92/92 unit tests passing
2026-07-20 13:33:13 +08:00
b58d01c7f2
cleanup: remove v6 garbage, restore original files, unify views.py workers
...
- Delete 15 garbage scripts (wnl_to_user_csv, eval_clustering, debug_db2/3,
debug_persistence, test_orch/orch2, test_pca, test_profile, test_tool_*,
test_e2e_full, test_32b, test_chinese_path, verify_runtime)
- Delete data/wnl_converted.csv
- Delete analysis/value_normalizer.py + tests/test_value_normalizer.py
- Remove normalize_lf references from data_loader.py
- Remove _norm column lookups from entity_aggregator.py
- Fix settings.py: FILE_UPLOAD_MAX_MEMORY_SIZE 10MB -> 100MB
- Fix upload.html: add per-file X delete button + removeFile()
- Refactor views.py: unify _run_analysis_worker/_run_llm_analysis into
_run_pipeline_worker with callback pattern
- Wrap sync ORM call in tool_registry.py with sync_to_async
- Update run_pipeline.py to use _run_pipeline_worker
- Rebuild gen_test_data.py as simple version (replace 3-profile v6 version)
- Add runtime/ db.sqlite3 logs/ .omo/ to .gitignore
- Delete old .omo/plans/ plan files (tianxiu-v4/v5/v6, globe-v6)
- 120/120 tests passing
2026-07-16 22:53:07 +08:00
814f29645a
chore: initial commit of 天璇 project state before cleanup
2026-07-16 22:41:26 +08:00
PM-pinou