PM-pinou
9f06e40a41
Task A: Replace _run_clustering_pipeline (533L->228L) — remove dead
docstring code, delegate to analysis.services.clustering.
Task B: Split analysis/data_loader.py (746L) into package:
analysis/data_loader/
__init__.py — re-exports all public symbols
_csv.py — load_csv_directory, BOM/encoding detection
_schema.py — schema merging and validation
_clean.py — numeric coercion helper
_sqlite.py — save_to_db, load_from_db, load_from_db_lazy, etc.
Keep old data_loader.py as thin re-export wrapper.
Verified: manage.py check (0 issues), run_pipeline (3 clusters, OK)
31 lines
830 B
Python
31 lines
830 B
Python
"""CSV data loading, schema handling, cleaning, and SQLite persistence.
|
|
|
|
Package layout:
|
|
_csv.py — CSV scanning, BOM/encoding detection, :func:`load_csv_directory`
|
|
_schema.py — Schema merging and cross-file validation
|
|
_clean.py — Data cleaning helpers (:func:`_coerce_to_float`)
|
|
_sqlite.py — SQLite persistence (save/load/drop tables)
|
|
"""
|
|
|
|
from ._csv import (
|
|
SUPPORTED_ENCODINGS,
|
|
detect_bom,
|
|
collect_schema,
|
|
detect_schema_dtypes,
|
|
load_config,
|
|
_file_count,
|
|
_resolve_encoding,
|
|
load_csv_directory,
|
|
)
|
|
from ._schema import _merge_schema_entries, validate_schemas
|
|
from ._clean import _coerce_to_float
|
|
from ._sqlite import (
|
|
_dtype_to_sqlite,
|
|
_str_to_polars_dtype,
|
|
_rows_to_csv_df,
|
|
save_to_db,
|
|
load_from_db,
|
|
load_from_db_lazy,
|
|
drop_sqlite_table,
|
|
)
|