fix: auto-convert DateType/TimestampType to ISO8601 in Azure Search

[Denniz Svens (DennizSvens)]

Related Issues/PRs

Fixes #2380

What changes are proposed in this pull request?

Fixes inconsistent behavior in Azure Search where index creation succeeded with DateType/TimestampType columns but data writing failed due to schema parity check expecting StringType.

Changes:

• Add automatic date/timestamp to ISO8601 conversion in prepareDF method
• Handle TimestampType with UTC conversion before formatting
• Handle DateType as midnight UTC in ISO8601 format
• Update edmTypeToSparkType mapping: Edm.DateTimeOffset → DateType (was StringType)
• Add tests for date handling including null values and timezone scenarios

This eliminates the previous two-step workflow where users had to manually convert dates to strings after initial index creation.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Tests added:

• test("Handle date fields correctly") - Basic date/timestamp handling with null values
• test("Date field conversion handles different timezones correctly") - Timezone conversion scenarios
• Tests verify both DateType and TimestampType conversion to ISO8601 format
• Tests confirm Azure Search index maintains correct Edm.DateTimeOffset field types

Does this PR change any dependencies?

• No. You can skip this section.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.

[Mark Hamilton (mhamilton723)]

Nice work!

[Mark Hamilton (mhamilton723)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

Attention: Patch coverage is 80.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 84.58%. Comparing base (5bbbbc7) to head (9ed0bbe).

Files with missing lines	Patch %	Lines
...azure/synapse/ml/services/search/AzureSearch.scala	80.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2381      +/-   ##
==========================================
+ Coverage   84.57%   84.58%   +0.01%     
==========================================
  Files         331      331              
  Lines       17205    17221      +16     
  Branches     1522     1523       +1     
==========================================
+ Hits        14551    14567      +16     
  Misses       2654     2654              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Mark Hamilton (mhamilton723)]

Denniz Svens (@DennizSvens) likewise style issues here and added unit tests tests are failing.

https://microsoft.github.io/SynapseML/docs/Reference/Developer%20Setup/

Will help you set up a local build to make your life easier. You seem like a great dev and we are happy to take whatever fixes you would like to make to this code so long as we can get the builds green!

[Mark Hamilton (mhamilton723)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Mark Hamilton (mhamilton723)]

LGTM and build has passed, please megre your changes and well make sure it stays green Denniz Svens (@DennizSvens) before final merges. Thanks for your work on this!

[Mark Hamilton (mhamilton723)]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).