feat: offline GeoIP, advanced filter, performance, enriched viz #2

2026-07-23T11:40:12+08:00

zhangjianzhong commented

2026-07-23 11:40:12 +08:00

v2 improvements for simple_analysis module:

Column normalization (50+ alias mapping)
Offline GeoIP integration (reuses analysis.geoip)
Advanced AND/OR/NOT filter builder
HDBSCAN performance (float32, DBSCAN fallback)
Edge relationship computation
All-noise graceful visualization
New deps: geoip2, maxminddb

v2 improvements for simple_analysis module: - Column normalization (50+ alias mapping) - Offline GeoIP integration (reuses analysis.geoip) - Advanced AND/OR/NOT filter builder - HDBSCAN performance (float32, DBSCAN fallback) - Edge relationship computation - All-noise graceful visualization - New deps: geoip2, maxminddb

zhangjianzhong added 2 commits 2026-07-23 11:40:12 +08:00

chore: remove unused DBSCAN import, sync with beta c8de2c8fe8

feat: offline GeoIP, advanced filter, performance tuning, enriched viz 55f3b3637a

Major v2 improvements to simple analysis module:

Backend:
- Column name normalization with 50+ column alias mapping
- Offline GeoIP integration (reuses analysis.geoip module,
  no dependency on CSV lat/lon columns)
- Advanced filter builder with AND/OR/NOT conditions
- Edge relationship computation (IP pair frequency)
- HDBSCAN with GeoIP-derived coordinates + float32 memory opt
- DBSCAN fallback when HDBSCAN produces all-noise
- Jitter for duplicate GeoIP coordinates (city-level overlap)
- All-noise graceful handling (visualize all points as noise)
- Cluster detail with GeoIP city/ISP/org attributes

Deps: geoip2==5.3.0, maxminddb==3.1.1 added to requirements.txt

Co-Authored-By: Claude <noreply@anthropic.com>

zhangjianzhong added 1 commit 2026-07-23 11:45:29 +08:00

feat: rich UI for v2 — preset filter, advanced filter builder, debug panel, edge overlays e57339b67f

Frontend improvements:
- Fixed sticky action bar with step indicator and nav buttons
- Collapsible preset filter panel with localStorage memory
- Advanced filter builder with AND/OR/NOT condition rows
- Filter builder auto-populates column names from upload
- Debug info panel (collapsible) with real-time stats
- Edge relationship layer with freq slider and toggle
- Enhanced point tooltips with IP sample and coordinate info
- Anti-duplicate button debounce via setBusy() helper

Backend:
- Add ip_sample to cluster_map for tooltip display
- Increase default cluster_selection_epsilon to 0.05

Co-Authored-By: Claude <noreply@anthropic.com>

zhangjianzhong added 1 commit 2026-07-23 11:51:30 +08:00

fix: null safety for all DOM operations, preset filter optional f798c39b13

- Add safe setText() helper with null check
- Make setBusy() null-safe for all DOM lookups
- Make preset filter elements optional (try/catch)
- Add null checks for all textContent assignments
- Remove emoji from error messages to avoid encoding issues
- Clarify preset filter UI labels

Co-Authored-By: Claude <noreply@anthropic.com>

zhangjianzhong added 11 commits 2026-07-23 13:36:06 +08:00

test: Playwright E2E test for simple-analysis v2 + fix template JS syntax d81171cc1d

- Fix extra closing brace in simple_analysis.html that broke all inline JS
- Add Playwright E2E test covering upload→filter→cluster→map→edges
- Add debug support script for Playwright issue isolation
- Use small 20-row test CSV for fast test execution
- All 31 assertions pass: page load, CSV upload, advanced filter,
  HDBSCAN clustering, Leaflet map, edge relationships

Co-Authored-By: Claude <noreply@anthropic.com>

refactor: remove CNAM auto-selection from upload step c5d9889a22

- Remove CNAM frequency display, auto-selection, and related CSS from upload UI
- Remove renderCnamList, selectCnam, runFilter functions (dead code)
- Remove topCnam from SA state object
- Keep preset filter cnam field for manual pre-filtering
- Users now use Step 2 advanced filter instead
- Update Playwright test assertion

Co-Authored-By: Claude <noreply@anthropic.com>

feat: alphabetical column sort + value suggestion dropdown in filter builder 298dd42cf6

- Sort column selector alphabetically (a-z) in condition builder
- Add datalist-based value suggestions per column type
  (cnrs/isrs → +/-, proto → TCP/UDP/ICMP, tls_ver → 0303/0304, etc.)
- Remove redundant initFilterBuilder column population (now inline in addFilterCondition)

Co-Authored-By: Claude <noreply@anthropic.com>

feat: filter value suggestions from actual upload data b844c4f92f

- Add GET /simple/column-values/ API returning distinct values for any column
  (top 50 values with counts, from the uploaded DataFrame via polars)
- Update frontend onFilterColChange to fetch real data instead of hardcoded hints
- Wire datalist ID per filter row for <input list=''> binding

Co-Authored-By: Claude <noreply@anthropic.com>

feat: nested boolean group support in filter builder (parentheses logic) 8eacb38e9e

Backend:
- Add _build_tree_condition() recursive function supporting AND/OR/NOT groups
- Modify advanced_filter to accept 'root' tree format alongside legacy flat format
- Tree supports arbitrary nesting depth: {type, logic, items}

Frontend:
- Add addFilterGroup() — creates visual ( ) group with own logic selector
- Add wrapRowInGroup() — wraps existing row into a group via [()]
- Add serializeFilterTree() — recursive JSON serialization
- Update applyAdvancedFilter to POST tree format
- Update updateFilterLogicTags for nested group labeling
- Switch filter row elements from IDs to class-based selectors
- CSS for group visual nesting (dashed left border, indentation)

Now supports: cnam='google' AND (isrs='+' OR cnrs='+')

Co-Authored-By: Claude <noreply@anthropic.com>

feat: filter preset templates for common boolean patterns 14fc0c8ea1

- Add '预设组' dropdown with two predefined templates:
  1. cnrs=+ OR isrs=+  (OR group for success conditions)
  2. cnam is empty AND (cnrs=+ OR isrs=+) (AND+OR nesting)
- applyFilterPreset() builds full UI tree with conditions + groups
- Presets auto-populate column, operator, and value fields

Co-Authored-By: Claude <noreply@anthropic.com>

refactor: upload preset uses tree format, preset group as buttons ac0c7fb117

Backend:
- upload_csv supports preset_filters.root (tree format via _build_tree_condition)
- Falls back to old flat fields for backward compatibility

Frontend:
- Upload preset panel: replace cnrs/isrs/cnam/snam fields with JSON textarea
- Preset group: change <select> dropdown to visible buttons (fixes no-dropdown issue)
- applyFilterPreset now takes string arg instead of event
- restorePreset reads/writes root JSON from localStorage

Co-Authored-By: Claude <noreply@anthropic.com>

feat: value dropdown with top-50 by count + preset builder UI ab31471bd3

Backend:
- advanced_filter supports mode='preset' — overwrites entry['df'] so
  preset filtering becomes the base data for subsequent steps

Frontend - Filter rows:
- Replace <input>+<datalist> with <select class=sa-filter-val>
- onFilterColChange populates dropdown with top 50 distinct values
  sorted by count, displayed as "value (count)"
- Disables value select for is_empty/not_empty operators

Frontend - Upload preset:
- Replace JSON textarea with full condition builder (addFilterCondition/
  addFilterGroup) — same format as advanced filter in Step 2
- Builder appears after upload with columns populated
- '应用预设筛选' button POSTs to filter/advanced/ with mode='preset'
- '清除预设' resets the builder
- Remove old upload-time preset_filters collection (now post-upload)

Co-Authored-By: Claude <noreply@anthropic.com>

fix: operator change handler enables/disables value dropdown 6f6581d38d

- Add onFilterOpChange handler on <select class=sa-filter-op>
- When operator is is_empty/not_empty: disable value select, show '— 无值 —'
- When switching back to eq/neq/contains/etc: re-enable and re-fetch values
- Fixes issue where changing operator before selecting column left value
  dropdown disabled permanently
- Remove operator check from onFilterColChange (now handled separately)

Co-Authored-By: Claude <noreply@anthropic.com>

cleanup: remove preset builder, restore simple preset fields 5423a360ec

- Revert preset panel to original cnrs/isrs/cnam/snam simple fields
- Remove applyPresetFilter, addPresetGroup, addPresetCondition,
  resetPreset, populatePresetBuilder, applyFilterPreset functions
- Remove mode='preset' backend code
- Remove root-format support from upload preset (no longer needed)
- Remove preset template buttons from filter action bar
- Keep: value dropdown with top-50, filter groups, tree-format advanced filter

Co-Authored-By: Claude <noreply@anthropic.com>

docs: requirements.md, offline toast, showWarning in base df1f208cd1

- Add requirements.md documenting all changes
- Add showWarning() to base.html with .toast-warning CSS
- Add offline tile fallback message when all providers fail
- All vendor assets local, GeoIP from local file

Co-Authored-By: Claude <noreply@anthropic.com>

This pull request has changes conflicting with the target branch.

requirements.txt
simple_analysis/templates/simple_analysis/simple_analysis.html
simple_analysis/urls.py
simple_analysis/views.py

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/simple-analysis-v2:feature/simple-analysis-v2

git checkout feature/simple-analysis-v2

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: HJQ/tianxuan#2