Skip to content

Comments

feat: support batch read for openmldb-spark-connector#3070

Merged
tobegit3hub merged 3 commits into4paradigm:mainfrom
tobegit3hub:feat/spark_connector_support_batch_read
Feb 28, 2023
Merged

feat: support batch read for openmldb-spark-connector#3070
tobegit3hub merged 3 commits into4paradigm:mainfrom
tobegit3hub:feat/spark_connector_support_batch_read

Conversation

@tobegit3hub
Copy link
Collaborator

Relate issue:

Usage:

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").getOrCreate()

    val zkCluster ="127.0.0.1:2181"
    val zkPath = "/openmldb"
    val db = "db"
    val table = "spark_read_test"
    val options = Map("db" -> db, "table" -> table, "zkCluster" -> zkCluster, "zkPath" -> zkPath)
    
    val df = spark.read
      .format("openmldb")
      .options(options)
      .load()
    df.show()
  }

Known issues:

  • Do not support partitions
  • Fail to handle null data in InternalRow

@tobegit3hub tobegit3hub self-assigned this Feb 17, 2023
@github-actions github-actions bot added the sdk openmldb java/python sdk label Feb 17, 2023
@codecov
Copy link

codecov bot commented Feb 17, 2023

Codecov Report

Base: 36.46% // Head: 36.47% // Increases project coverage by +0.01% 🎉

Coverage data is based on head (6a2f961) compared to base (f178654).
Patch coverage: 37.93% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3070      +/-   ##
============================================
+ Coverage     36.46%   36.47%   +0.01%     
  Complexity      393      393              
============================================
  Files           160      166       +6     
  Lines          9624     9680      +56     
  Branches       1132     1135       +3     
============================================
+ Hits           3509     3531      +22     
- Misses         5879     5913      +34     
  Partials        236      236              
Impacted Files Coverage Δ
...ava/com/_4paradigm/openmldb/jdbc/SQLResultSet.java 14.38% <0.00%> (-0.06%) ⬇️
...a/com/_4paradigm/openmldb/spark/OpenmldbTable.java 0.00% <0.00%> (ø)
...ldb/spark/read/OpenmldbPartitionReaderFactory.java 0.00% <0.00%> (ø)
...radigm/openmldb/spark/read/OpenmldbReadConfig.java 0.00% <0.00%> (ø)
...m/_4paradigm/openmldb/spark/read/OpenmldbScan.java 0.00% <0.00%> (ø)
...adigm/openmldb/spark/read/OpenmldbScanBuilder.java 0.00% <0.00%> (ø)
...paradigm/openmldb/spark/read/SimplePartition.scala 0.00% <0.00%> (ø)
.../openmldb/spark/read/OpenmldbPartitionReader.scala 75.86% <75.86%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 17, 2023

Linux Test Report

       56 files       230 suites   1h 17m 21s ⏱️
11 923 tests 11 916 ✔️ 7 💤 0
16 924 runs  16 917 ✔️ 7 💤 0

Results for commit 6a2f961.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 17, 2023

SDK Test Report

  91 files    91 suites   1m 40s ⏱️
182 tests 173 ✔️ 9 💤 0
228 runs  219 ✔️ 9 💤 0

Results for commit 6a2f961.

♻️ This comment has been updated with latest results.

@tobegit3hub
Copy link
Collaborator Author

Known issue, we may not get all data with select *.

@tobegit3hub tobegit3hub enabled auto-merge (squash) February 28, 2023 03:27
@tobegit3hub tobegit3hub merged commit 7fefe21 into 4paradigm:main Feb 28, 2023
@lumianph lumianph added the alpha this feature is not fully tested or functional label Apr 25, 2023
@lqy222 lqy222 mentioned this pull request May 8, 2023
@lumianph lumianph removed the alpha this feature is not fully tested or functional label May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sdk openmldb java/python sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support batch read for openmldb-spark-connector

4 participants