Skip to content

fix: Stable StarRocks Driver and Add test code#8619

Merged
k-anshul merged 5 commits intorilldata:mainfrom
EdwardArchive:fix_starrocks_olap_connector
Jan 19, 2026
Merged

fix: Stable StarRocks Driver and Add test code#8619
k-anshul merged 5 commits intorilldata:mainfrom
EdwardArchive:fix_starrocks_olap_connector

Conversation

@EdwardArchive
Copy link
Contributor

@EdwardArchive EdwardArchive commented Jan 12, 2026 •
edited
Loading

Related PR : #8454

Problem:

  • VARCHAR columns were returned as []byte instead of string due to MySQL
    driver behavior when scanning into any type
  • DECIMAL columns returned as float64, causing precision loss for large values

Solution:

  • Add starrocksRows wrapper with custom MapScan() using sql.NullXXX types
  • DECIMAL now returns as string to preserve full precision (same as MySQL driver)
  • Add prepareScanDest() for type-safe scan destinations

Testing:

  • Add Docker-based integration tests using StarRocks 4.0.3 all-in-one container
  • 29 test cases covering all StarRocks data types:
    • Basic types (BOOLEAN, INT, BIGINT, FLOAT, DOUBLE, VARCHAR, etc.)
    • Complex types (ARRAY, MAP, STRUCT)
    • Aggregate types (HLL, BITMAP)
    • High-precision DECIMAL (DECIMAL32, DECIMAL64, DECIMAL128)
    • Unicode strings (Korean, Chinese, Japanese, Emoji)
    • Boundary values and NULL handling
  • Refactor teststarrocks to read from init.sql file for better maintainability

Type mapping (MySQL protocol):

  • BOOLEAN → int16 (reported as TINYINT)
  • DECIMAL → string (precision preserved)
  • LARGEINT → string (128-bit exceeds int64)
  • ARRAY/MAP/STRUCT → string (JSON format)
  • JSON → string (reported as VARCHAR)

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@EdwardArchive
Add Docker-based integration tests using StarRocks 4.0.3 all-in-one c…
1ed5214 
…ontainer & fix VARCHAR binary return

Signed-off-by: Edward Kim <kjjs1996@gmail.com>
@EdwardArchive
Fix: golangci-lint errors in StarRocks driver
0ff492b 
Signed-off-by: Edward Kim <kjjs1996@gmail.com>
@EdwardArchive
Copy link
Contributor Author

EdwardArchive commented Jan 12, 2026

These test failures are due to missing environment variables (S3, GCS, Druid, MotherDuck credentials)

@EdwardArchive
fix(starrocks): align binary type mapping with MySQL driver
6401aa6 
  - Change BINARY/VARBINARY/BLOB from CODE_BYTES to CODE_STRING
    for consistency with MySQL driver behavior
  - Expand type mapping unit tests to cover all StarRocks types:
    - Boolean, Integer, Float, String, Binary, DateTime
    - Semi-structured types (JSON, ARRAY, MAP, STRUCT)
    - Aggregate-only types (HLL, BITMAP, PERCENTILE) as unsupported
  - Update integration test expectations for VARBINARY type

Signed-off-by: Edward Kim <kjjs1996@gmail.com>
@EdwardArchive EdwardArchive changed the title [Enhancement] Stable StarRocks Driver and Add test code fix : Stable StarRocks Driver and Add test code Jan 13, 2026
@EdwardArchive EdwardArchive changed the title fix : Stable StarRocks Driver and Add test code fix: Stable StarRocks Driver and Add test code Jan 13, 2026
@EdwardArchive
Copy link
Contributor Author

EdwardArchive commented Jan 13, 2026

Hi, @k-anshul
Could you take a look when you have time?

@k-anshul
Copy link
Member

k-anshul commented Jan 14, 2026 •
edited
Loading

Hey @EdwardArchive

Thank you for your contribution.
Another super important tests for stability of the driver are the tests in runtime/queries pkg especially metricsview_*_test.go. Would you like to contribute tests those tests as well ? You can do it in a follow up if preferred.

@EdwardArchive
Copy link
Contributor Author

EdwardArchive commented Jan 14, 2026

Hi @k-anshul,
Thanks for the advice — that’s an important point! I’ll add more tests for metricsview_*_test.go.

@EdwardArchive
feat(starrocks): add metricsview tests and fix driver issues
bc51e3d 
  - Add metricsview aggregation, toplist, and comparison tests for StarRocks
  - Implement Stream Load API for test data loading with FE→BE redirect handling
  - Add NULLS LAST for consistent NULL ordering in StarRocks
  - Fix TIMESTAMP handling to use sql.NullTime with parseTime=true
  - Add CONVERT_TZ for timezone support in DateTruncExpr
  - Fix schema resolution using InformationSchema lookup
  - Cast percent_of_total result to DOUBLE for consistent numeric handling
  - Update StarRocks documentation to match driver implementation

Signed-off-by: EdwardKim <kjjs1996@gmail.com>
@EdwardArchive
Copy link
Contributor Author

EdwardArchive commented Jan 15, 2026

Hey @k-anshul,
I just added tests for metricsview_*_test.go.
and some changes on backend.
Could you take a look when you have time?

@EdwardArchive
Copy link
Contributor Author

EdwardArchive commented Jan 17, 2026

These test failures are due to missing environment variables (Druid, MotherDuck) DNS and credentials

k-anshul
k-anshul reviewed Jan 19, 2026
View reviewed changes
@k-anshul k-anshul merged commit 23981e7 into rilldata:main Jan 19, 2026
23 of 27 checks passed
@k-anshul
Copy link
Member

k-anshul commented Jan 19, 2026

Thank you for your contribution.

k-anshul added a commit that referenced this pull request Feb 2, 2026
@EdwardArchive @k-anshul
fix: Stable StarRocks Driver and Add test code (#8619)
f531889 
* Add Docker-based integration tests using StarRocks 4.0.3 all-in-one container & fix VARCHAR binary return

Signed-off-by: Edward Kim <kjjs1996@gmail.com>

* Fix: golangci-lint errors in StarRocks driver

Signed-off-by: Edward Kim <kjjs1996@gmail.com>

* fix(starrocks): align binary type mapping with MySQL driver

  - Change BINARY/VARBINARY/BLOB from CODE_BYTES to CODE_STRING
    for consistency with MySQL driver behavior
  - Expand type mapping unit tests to cover all StarRocks types:
    - Boolean, Integer, Float, String, Binary, DateTime
    - Semi-structured types (JSON, ARRAY, MAP, STRUCT)
    - Aggregate-only types (HLL, BITMAP, PERCENTILE) as unsupported
  - Update integration test expectations for VARBINARY type

Signed-off-by: Edward Kim <kjjs1996@gmail.com>

* feat(starrocks): add metricsview tests and fix driver issues

  - Add metricsview aggregation, toplist, and comparison tests for StarRocks
  - Implement Stream Load API for test data loading with FE→BE redirect handling
  - Add NULLS LAST for consistent NULL ordering in StarRocks
  - Fix TIMESTAMP handling to use sql.NullTime with parseTime=true
  - Add CONVERT_TZ for timezone support in DateTruncExpr
  - Fix schema resolution using InformationSchema lookup
  - Cast percent_of_total result to DOUBLE for consistent numeric handling
  - Update StarRocks documentation to match driver implementation

Signed-off-by: EdwardKim <kjjs1996@gmail.com>

* Update docs/docs/build/connectors/olap/starrocks.md

---------

Signed-off-by: Edward Kim <kjjs1996@gmail.com>
Signed-off-by: EdwardKim <kjjs1996@gmail.com>
Co-authored-by: Anshul Khandelwal <12948312+k-anshul@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@k-anshul k-anshul k-anshul approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@EdwardArchive @k-anshul