[fix][io][kca] kafka headers silently dropped

[efcasado]

Motivation

The current implementation of the Kafka Connect adaptor silently discards headers from the Kafka Connect SourceRecord instead of mapping them to Pulsar message properties. This is particularly impactful when using Debezium's Outbox Event Router SMT with the table.fields.additional.placement option, which allows outbox table columns to be placed in the header, envelope, or partition of the resulting record. Columns configured with header placement are silently ignored because the adaptor never propagates Connect record headers to Pulsar message properties. Since Kafka Connect headers and Pulsar message properties are semantically equivalent (string key/value metadata outside the payload), the expected behavior is a direct mapping.

A similar issue was previously identified for pulsar-io-kafka and addressed in #17829. However, kafka-connect-adaptor was not covered by that fix and still silently discards Kafka Connect SourceRecord headers.

Modifications

AbstractKafkaConnectSource previously returned a hard-coded empty map from getProperties(), causing all Kafka Connect headers to be silently dropped. This change maps headers from the transformed SourceRecord to Pulsar message properties, preserving metadata that would otherwise be lost in transit (eg. columns placed in header position via Debezium's Outbox Event Router SMT and table.fields.additional.placement).

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Extended the existing KafkaConnectSourceTest test suite with an additional test to confirm Kafka headers are correctly propagated to the resulting record.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: efcasado#2

[lhotari]

LGTM, thanks for the contribution @efcasado

[codecov-commenter]

Codecov Report

❌ Patch coverage is 66.66667% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.75%. Comparing base (735b429) to head (1ce5781).
⚠️ Report is 124 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/pulsar/io/kafka/connect/KafkaConnectSource.java	60.00%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #25325      +/-   ##
============================================
- Coverage     72.97%   72.75%   -0.22%     
+ Complexity    34347    34232     -115     
============================================
  Files          1954     1954              
  Lines        154715   154725      +10     
  Branches      17704    17708       +4     
============================================
- Hits         112898   112575     -323     
- Misses        32816    33072     +256     
- Partials       9001     9078      +77     

Flag	Coverage Δ
inttests	25.74% <ø> (-0.49%)	⬇️
systests	22.56% <ø> (-0.06%)	⬇️
unittests	73.73% <66.66%> (-0.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r/io/kafka/connect/AbstractKafkaConnectSource.java	69.73% <100.00%> (+0.65%)	⬆️
...he/pulsar/io/kafka/connect/KafkaConnectSource.java	70.00% <60.00%> (-0.67%)	⬇️

... and 89 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.