DX-114802: Upgrade Java JAR build workflow with modern improvements (dremio_26.1_18.1.0)#126
Draft
timhurskidremio wants to merge 18 commits intodremio:dremio_26.1_18.1.0from
Conversation
- Update GitHub Actions to v4+ with SHA pinning
- Add ARM64 support for Ubuntu and macOS
- Use vcpkg for LLVM to remove z3 runtime dependency
- Add disk space cleanup for macOS runners
- Add disk space monitoring throughout build
- Enhance testing for ARM64 libraries
Ref: DX-103340
timhurskidremio
force-pushed
the
upgrade-java-jars-workflow-dremio_26.1_18.1.0
branch
from
b5cb6b6 to
2681868
Compare
- Remove redundant 'virtual' keyword from lvalue.h:123 (already has 'override') - Replace 'unsigned long' with 'auto' in encrypt_utils_common.cc to avoid C type lint warning
Change from deprecated 'macos-13' to 'macos-13-large' for x86_64 builds. The 'macos-13-us-default' configuration is no longer supported by GitHub Actions.
Replace deprecated 'macos-13' with 'macos-13-large' for AMD64 builds in: - C++ workflow (.github/workflows/cpp.yml) - Python workflow (.github/workflows/python.yml) - MATLAB workflow (.github/workflows/matlab.yml) The 'macos-13-us-default' configuration is no longer supported by GitHub Actions. Using explicit runs-on matrix values to specify macos-13-large for x86_64 and macos-14 for ARM64.
…13-large According to GitHub Actions runner-images documentation, macos-13 and macos-13-large do not exist. The available x86_64 (AMD64) runners are: - macos-14-large (oldest available) - macos-15-large - macos-26-large (newest) Updated all workflows to use macos-14-large for AMD64 builds: - C++ workflow (.github/workflows/cpp.yml) - Python workflow (.github/workflows/python.yml) - MATLAB workflow (.github/workflows/matlab.yml) - Java JARs workflow (dev/tasks/java-jars/github.yml)
Update to match upstream Apache Arrow main branch: - Use correct URL format: /releases/download/v.../googletest-... instead of /archive/release-... - Use correct SHA256 checksum: 65fab701... (for googletest-1.17.0.tar.gz from releases) instead of d5558cd4... (for archive/release-1.17.0.tar.gz) This fixes the 404 error when downloading GoogleTest from Apache JFrog Artifactory, as the mirror expects the same file format as the upstream GitHub release.
The CMake build was using hardcoded old GoogleTest URL format in ThirdpartyToolchain.cmake, which overrode the fix in versions.txt. Changed from: - /archive/release-1.17.0.tar.gz (404 error) - Chromium mirror (also 404) To: - /releases/download/v1.17.0/googletest-1.17.0.tar.gz (works!) Also removed the Chromium mirror as it doesn't have GoogleTest 1.17.0. Tested locally on macOS ARM64 - build completes successfully.
The FindLLVMAlt.cmake script was failing when trying to find old LLVM versions (llvm@13, llvm@11, llvm@10, llvm@9, llvm@8, llvm@7) that no longer exist in Homebrew. The 'brew --prefix llvm@XX' command would error out and cause CMake configuration to fail. Changes: - Added ERROR_QUIET to execute_process() to suppress brew errors - Added check to only append LLVM_BREW_PREFIX if it's not empty This allows CMake to gracefully try multiple LLVM versions and use the first one that exists (e.g., llvm@14, llvm@15, etc.).
Add LLVM path detection logic that: - Determines vcpkg triplet based on architecture - Sets LLVM_DIR to point to vcpkg-installed LLVM when VCPKG_ROOT_LOCAL is set - Configures CMAKE_OSX_SYSROOT for proper SDK detection - Sets ARROW_GANDIVA_PC_CXX_FLAGS for C++ headers - Uses vcpkg's RE2 for ABI compatibility with LLVM/Abseil This matches the build script from dremio-arrow-java.
vcpkg installs LLVM 21.1.1 via the overlay, but the ARROW_LLVM_VERSIONS list only went up to 19.1, causing CMake's FindLLVMAlt.cmake to reject the installed LLVM as incompatible. Use 21.1 (not just 21) to match CMake's version compatibility rules which require MAJOR.MINOR format for proper matching.
timhurskidremio
force-pushed
the
upgrade-java-jars-workflow-dremio_26.1_18.1.0
branch
from
27f7912 to
9a691ec
Compare
The macOS CI runner has Homebrew's Boost 1.90.0 installed, but it's missing the 'system' component. This causes Arrow's ThirdpartyToolchain to detect the partial installation and then attempt to build Boost from source as a fallback. However, this creates a conflict because some Boost targets (like Boost::filesystem) were already imported from the system Boost before the fallback was triggered. Setting Boost_SOURCE=BUNDLED tells Arrow to ignore the system Boost entirely and build all components from source, avoiding the conflict.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.