fix(stats): preserve length & lex stats when column type widens to String

[jqnatividad]

Summary

• When a column's inferred type widens to TString mid-stream (Date→String under --infer-dates, Integer→String, Float→String, DateTime→String, etc.), TypedSum::stotlen, TypedMinMax::{str_len, strings}, and Stats::online_len previously only recorded samples that arrived after the widen — so sum_length/avg_length/min_length/max_length/lex min/max/sort_order/sortiness/stddev_length/variance_length/cv_length all reflected only the post-widen tail. The same bug also dropped pre-merge TNull samples from online_len on pure-String columns.
• Fix: track these fields unconditionally on every (non-empty) sample regardless of dispatched type. show() already gates length-stat emission on the final typ == TString, so columns that never widen are unaffected in output. Commute::merge impls for TypedSum, TypedMinMax, and OnlineStats already accumulate the affected fields correctly, so parallel-chunk merging is unchanged.
• Reproducer from issue now matches the expected output:

  $ qsv stats --infer-dates issue3855.csv | grep date2
  date2,String,true,,2026-05-13,NA,,Ascending,1,2,10,12,6,4,16,0.6667,...
  

Changes

• src/cmd/stats.rs:
  • TypedSum::add_with_parsed — track stotlen unconditionally
  • TypedMinMax::add_with_parsed — track str_len + strings (lex min/max) unconditionally on non-empty samples
  • Stats::add — lift online_len.add out of the t == TString block; use if let Some for defense-in-depth
• tests/test_stats.rs — 5 new regression tests (stats_widen_*_issue3855) covering each widening path
• tests/test_moarstats.rs, tests/test_stats.rs, resources/test/*-stats.csv — fixture updates for the equal-or-more-correct post-fix values; diffs are confined exclusively to stddev_length/variance_length/cv_length on String columns with interleaved NULLs (where the pre-fix code skipped NULLs that arrived before the column type stabilized to TString)

Fixes #3855.

Test plan

• cargo build --locked --bin qsv -F all_features
• cargo test --test tests stats_widen -F all_features — 5 new tests pass
• cargo test --test tests stats -F all_features — 749 stats tests, 0 failed
• cargo test --test tests moarstats -F all_features — 88 tests, 0 failed
• cargo test -F all_features — 2821 tests, 0 failed
• cargo +nightly fmt --check clean
• cargo clippy -F all_features clean
• Reproducer from the issue produces expected output

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

This PR fixes stats length and lexicographic string statistics when inferred column types widen to String, ensuring earlier typed samples are included in final String outputs.

Changes:

• Tracks string length totals, min/max length, lexical min/max, and online length stats across all samples before final type stabilization.
• Adds regression tests for Date/Integer/Float/DateTime-to-String widening.
• Updates expected stats fixtures for corrected String length variance/stddev/CV values.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/cmd/stats.rs	Moves String length/lex tracking outside String-only branches.
tests/test_stats.rs	Updates expected mixed-type output and adds widening regression tests.
tests/test_moarstats.rs	Updates expected fixture output for corrected String length stats.
resources/test/boston311-100-everything-date-stats-variance-stddev.csv	Updates corrected stats fixture values.
resources/test/boston311-100-antimodes-len500-stats.csv	Updates corrected stats fixture values.
resources/test/boston311-10-boolean-tf-stats.csv	Updates corrected stats fixture values.
resources/test/boston311-10-boolean-1or0-stats.csv	Updates corrected stats fixture values.