[connector] Support spark catalog and introduce some basic classes #544
Conversation
|
@wuchong cc |
abe3482 to
be8528e
Compare
|
hi @Alibaba-HZY This PR looks good overall, I will review it later. Thanks. |
...connector-spark-common/src/main/java/com/alibaba/fluss/connector/spark/SparkInternalRow.java
Outdated
Show resolved
Hide resolved
...connector-spark-common/src/main/java/com/alibaba/fluss/connector/spark/SparkInternalRow.java
Outdated
Show resolved
Hide resolved
...uss-connector-spark-common/src/main/scala/com/alibaba/fluss/connector/spark/SparkTable.scala
Outdated
Show resolved
Hide resolved
...spark-common/src/test/java/com/alibaba/fluss/connector/spark/catalog/SparkCatalogITCase.java
Outdated
Show resolved
Hide resolved
...-connector-spark-common/src/main/java/com/alibaba/fluss/connector/spark/RichTableBucket.java
Outdated
Show resolved
Hide resolved
...ctor-spark-common/src/main/scala/com/alibaba/fluss/connector/spark/FlussInputPartition.scala
Outdated
Show resolved
Hide resolved
f2996f4 to
f10f1f0
Compare
|
Sorry, I just merged #597 and causes some conflicts in this PR. The main change of #594 is moving Therefore, we can also move |
ok i will handle this |
down |
95a317e to
e121544
Compare
c8f1067 to
623b75c
Compare
|
@wuchong @Alibaba-HZY @polyzos Do you mind if I pick this up and make some improvements for v0.8? |
|
Since there hasn't been activity for a while now, I'm assigning to you @MaggieSullivan. Thank you for your interest 🥇 |
|
@Alibaba-HZY Apologies, I tried to rebase this branch as it had way too many conflicts to take a look at and somehow I messed things up big time. I don't know how I managed to merge from my cli, and now seems like changed file tracking is lost .. I'm still trying to figure out if there is something I can do to fix this, but no luck so far 😰 |
|
|
@Alibaba-HZY The changes from main were reverted immediately. However, I'm not sure how I managed to mess up your PR-branch, and nothing I tried worked. |
|
ok i will open a new pr
…---- Replied Message ----
| From | Giannis ***@***.***> |
| Date | 09/01/2025 00:58 |
| To | apache/fluss ***@***.***> |
| Cc | HZY ***@***.***>,
Mention ***@***.***> |
| Subject | Re: [apache/fluss] [connector] Support spark catalog and introduce some basic classes (PR #544) |
polyzos left a comment (apache/fluss#544)
@Alibaba-HZY The changes from main were reverted immediately. However, I'm not sure how I managed to mess up your PR-branch, and nothing I tried worked.
Would you mind opening a new PR? Again, I'm really sorry for all this inconvenience.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@Alibaba-HZY, just a reminder, if you can re-open this PR 🙇 |
|
ok sorry because my work is too busy ， i will do this today
…---- Replied Message ----
| From | Giannis ***@***.***> |
| Date | 09/03/2025 14:38 |
| To | apache/fluss ***@***.***> |
| Cc | HZY ***@***.***>,
Mention ***@***.***> |
| Subject | Re: [apache/fluss] [connector] Support spark catalog and introduce some basic classes (PR #544) |
polyzos left a comment (apache/fluss#544)
@Alibaba-HZY, just a reminder, if you can re-open this PR 🙇
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Purpose
Linked issue: open #228
Linked issue: open #155
Support spark catalog and introduce some basic classes to support spark read and write
Brief change log
1.Based on #258 and discussion with @XuQianJin-Stars , I resubmitted [support spakr catalog ] and introduce some basic classes(SparkInternalRow,SparkRow,FlussInputPartition) to support spark read and write.
2.support SupportsPartitionManagement in spark table
Tests
com.alibaba.fluss.connector.spark.catalog.SparkCatalogITCase
com.alibaba.fluss.connector.spark.utils.SparkConversionsTest
com.alibaba.fluss.connector.spark.utils.SparkTypeUtilsTest
com.alibaba.fluss.connector.spark.SparkInternalRowTest
API and Format
no
Documentation
no