Skip to content

[AURON #1850] Add FlinkArrowUtils for Flink-Arrow type conversion #1959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
x-tong wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[AURON #1850] Add FlinkArrowUtils for Flink-Arrow type conversion #1959

x-tong wants to merge 5 commits into from

Conversation

@x-tong
Copy link

@x-tong x-tong commented Jan 26, 2026 •
edited
Loading

Summary

Part 1/3 of Flink RowData to Arrow conversion implementation (split from #1930).

This PR adds the foundational type conversion utilities:

  • FlinkArrowUtils: Conversion form Flink RowData to Arrow types
  • Support for all common Flink types including primitives, temporal, and complex types
  • Comprehensive unit tests for type conversion

Follow-up PRs

  • Part 2: FlinkArrowFieldWriter + FlinkArrowWriter
  • Part 3: FlinkArrowFFIExporter

Test plan

  • Unit tests for FlinkArrowUtils type conversion
  • Build passes with ./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -Pflink-1.18 -DskipBuildNative
  • Code formatted with ./dev/reformat

Related: #1850

@x-tong
[AURON apache#1850] Add FlinkArrowUtils for Flink-Arrow type conversion
b33bea0 
Part 1 of Flink RowData to Arrow conversion implementation.

This PR adds the foundational type conversion utilities:
- FlinkArrowUtils: Bidirectional conversion between Flink LogicalType and Arrow types
- Support for all common Flink types including primitives, temporal, and complex types
- Comprehensive unit tests for type conversion
@x-tong x-tong mentioned this pull request Jan 26, 2026
Open
5 tasks
@cxzl25 cxzl25 requested a review from Copilot January 26, 2026 12:34
Copilot AI reviewed Jan 26, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces foundational type conversion utilities for Flink-Arrow integration as part 1 of a 3-part implementation series. It adds FlinkArrowUtils with methods to convert Flink LogicalTypes to Arrow types, supporting primitives, temporal types, and complex structures (arrays, maps, rows).

Changes:

  • Added FlinkArrowUtils class with one-way conversion from Flink LogicalType to Arrow types
  • Added comprehensive unit tests covering all supported type conversions
  • Added Arrow and Flink table dependencies to support the conversion utilities

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File Description
auron-flink-extension/auron-flink-runtime/src/main/java/org/apache/auron/flink/arrow/FlinkArrowUtils.java Core utility class implementing Flink-to-Arrow type conversion with methods for types, fields, and schemas, plus a shared ROOT_ALLOCATOR for Arrow memory management
auron-flink-extension/auron-flink-runtime/src/test/java/org/apache/auron/flink/arrow/FlinkArrowUtilsTest.java Comprehensive test suite covering basic types, complex types (arrays, rows, maps), temporal types, and error handling
auron-flink-extension/auron-flink-runtime/pom.xml Added Arrow dependencies (arrow-c-data, arrow-memory-unsafe, arrow-vector) and test dependencies (junit-jupiter-api)
.gitignore Added LSP-related files (*.prefs) to ignore list

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

x-tong and others added 2 commits January 26, 2026 22:11
@x-tong
Update auron-flink-extension/auron-flink-runtime/src/main/java/org/ap…
7de5c88 
…ache/auron/flink/arrow/FlinkArrowUtils.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@x-tong
Fix code formatting
666fc85
ShreyeshArangath
ShreyeshArangath reviewed Jan 27, 2026
View reviewed changes
Copy link
Contributor

@ShreyeshArangath ShreyeshArangath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just a couple minor comments

Also, from the description, should this not be built with flink instead of Spark?

./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -DskipBuildNative

@x-tong
Address PR review comments for FlinkArrowUtils
9f6cfc6 
- Mark class as final to prevent subclassing
- Add null check for logicalType parameter to throw IllegalArgumentException instead of NPE
@x-tong x-tong force-pushed the feature/issue-1850-flink-arrow-part1 branch from e6be0b8 to 9f6cfc6 Compare January 27, 2026 18:09
@x-tong
Copy link
Author

x-tong commented Jan 27, 2026

Overall LGTM, just a couple minor comments

Also, from the description, should this not be built with flink instead of Spark?

./auron-build.sh --pre --sparkver 3.5 --scalaver 2.12 -DskipBuildNative

Thank you for your review, yes, that's the way it should be done, I have made the changes.

@cxzl25 cxzl25 requested a review from Copilot January 28, 2026 05:59
Copilot AI reviewed Jan 28, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@x-tong
Add test coverage for NullType and Decimal bitWidth
54875fe 
- Add test case for NullType to Arrow conversion
- Add assertion to verify Decimal bitWidth is 128
ShreyeshArangath
ShreyeshArangath approved these changes Jan 28, 2026
View reviewed changes
Copy link
Contributor

@ShreyeshArangath ShreyeshArangath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

+1 more reviewer

@ShreyeshArangath ShreyeshArangath ShreyeshArangath approved these changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

build flink infra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@x-tong @ShreyeshArangath