Support Array type in KV table

[platinumhamburg]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Add Array type support for KV table scenarios to enable primary key tables with array columns.

Solution

• Extend KV table storage format to support Array type
• Implement Array type serialization/deserialization for KV tables
• Add support for Array type in KV table read/write operations

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[hemanthsavasere]

Hi @platinumhamburg and @wuchong I would like to work on this issue

[hemanthsavasere]

As per my analysis:

Desired State

1. Tables with array columns can be created and used in KV scenarios
2. Array data can be serialized/deserialized in both CompactedRow and IndexedRow formats
3. Flink integration allows reading/writing array data seamlessly
4. Documentation reflects ARRAY as a supported type

What's not in Scope

1. Arrays cannot be part of primary keys (standard database constraint)
2. MAP type support is out of scope for this implementation
3. ROW type support is out of scope for this implementation
4. Paimon/Iceberg array support is out of scope
5. ARRAY<ARRAY<...>> support will be added as extension
6. Arrow serialization for arrays is out of scope

@platinumhamburg and @wuchong. Please let me know if the analysis is correct so that I can proceeed with my implementation

[platinumhamburg]

As per my analysis:

Desired State

1. Tables with array columns can be created and used in KV scenarios
2. Array data can be serialized/deserialized in both CompactedRow and IndexedRow formats
3. Flink integration allows reading/writing array data seamlessly
4. Documentation reflects ARRAY as a supported type

What's not in Scope

1. Arrays cannot be part of primary keys (standard database constraint)
2. MAP type support is out of scope for this implementation
3. ROW type support is out of scope for this implementation
4. Paimon/Iceberg array support is out of scope
5. ARRAY<ARRAY<...>> support will be added as extension
6. Arrow serialization for arrays is out of scope

@platinumhamburg and @wuchong. Please let me know if the analysis is correct so that I can proceeed with my implementation

Hi @hemanthsavasere,
Thank you so much for your interest in this issue! I noticed that PR #1987 appears to have already implemented the ArrayType encoding/decoding functionality for both CompactedRow and IndexedRow. Based on this, it looks like ArrayType support for PrimaryKey Tables may already be in place.
To ensure we're on the right track, I think it would be great to have @XuQianJin-Stars review and confirm this. Depending on his validation, we might be able to close this issue.

[XuQianJin-Stars]

This PR #1972 implements this logic.