[AMORO-3973] Support spark3.4 for mixed format #4013
Conversation
github-actions
bot
added
module:mixed-spark
| val schema = createSchema(transformClause.colTypeList) | ||
| val replacedSchema = CharVarcharUtils.replaceCharVarcharWithStringInSchema(schema) | ||
| (DataTypeUtils.toAttributes(replacedSchema), false) | ||
| (schema.toAttributes, false) |
There was a problem hiding this comment.
should be replacedSchema.toAttributes?
There was a problem hiding this comment.
I fix the code by deleting 'replacedSchema'
| } | ||
|
|
||
| def visitCommonSelectQueryClausePlan( | ||
| private def visitCommonSelectQueryClausePlan( |
| <include>org.apache.iceberg:iceberg-orc</include> | ||
| <include>org.apache.iceberg:iceberg-parquet</include> | ||
| <include>org.apache.iceberg:iceberg-aws</include> | ||
| <include>org.apache.iceberg:iceberg-aliyun</include> |
There was a problem hiding this comment.
is it necessary for spark-3.5 as well?
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
from
e2d8b8a to
c9082d4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4013 +/- ##
============================================
+ Coverage 22.12% 22.85% +0.72%
- Complexity 2461 2531 +70
============================================
Files 445 449 +4
Lines 40897 41048 +151
Branches 5767 5784 +17
============================================
+ Hits 9050 9382 +332
+ Misses 31089 30857 -232
- Partials 758 809 +51
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
zhangwl9
force-pushed
the
AMORO-add-Spark3.4-for-mixedFormat-dev
branch
from
c9082d4 to
c549de1
Compare
| <artifactId>scala-compiler</artifactId> | ||
| <version>${scala.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> |
There was a problem hiding this comment.
why remove it?
If needed, you can remove it in separate PR for all spark-3.5, spark-3.4, spark-3.3
There was a problem hiding this comment.
do not remove the <scope>provided</scope> for spark-3.4, please keep it consistent for spark-3.5, spark-3.4, spark-3.3, and change it in separate PR if necessary.
3 participants
Why are the changes needed?
Close #3973.
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation