Skip to content

sync cibuildwheel test env with uv #2913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kevinjqliu wants to merge 8 commits into
base: main
Choose a base branch
from
+7 −4
Open

sync cibuildwheel test env with uv #2913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c7d6ac3
fix quoting
kevinjqliu Jan 14, 2026
e9fd5c4
do we need sqlalchemy?
kevinjqliu Jan 14, 2026
01f5690
lets try uv
kevinjqliu Jan 14, 2026
070e536
include sqlalchemy in dev dep
kevinjqliu Jan 14, 2026
2d44374
update uv lock
kevinjqliu Jan 14, 2026
42f72a0
try uv sync
kevinjqliu Jan 14, 2026
b33abd6
uv run pytest
kevinjqliu Jan 14, 2026
f1409e1
relative path
kevinjqliu Jan 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pypi-build-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ jobs:
# Ignore 32 bit architectures
CIBW_ARCHS: "auto64"
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
CIBW_TEST_REQUIRES: "pytest==7.4.2 moto==5.0.1 pytest-lazy-fixture==0.6.3 sqlalchemy>=2.0.18,<3"
CIBW_TEST_COMMAND: "pytest {project}/tests/avro/test_decoder.py"
CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"
CIBW_TEST_COMMAND: "uv run --directory {project} pytest tests/avro/test_decoder.py"
Copy link
Contributor

@geruh geruh Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uv usage in the build wheels is a great addition! Now the CIBW dependencies won't need to be manually updated and can be kept in sync with the pyproject versions.

I believe this command will use the entire list of dependencies in the dev group. I think here or as a followup we could create a minimal wheel-test dependency group in the pyproject file with just pytest, moto, pytest-lazy-fixture, and sqlalchemy. WDYT?

Copy link
Contributor Author

@kevinjqliu kevinjqliu Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea good point! on the flip side, we're not currently using the dev group for running unit tests 😆 we install all deps

iceberg-python/Makefile

Lines 72 to 73 in 9ab4f04

install-dependencies: setup-venv ## Install all dependencies including extras
uv sync $(PYTHON_ARG) --all-extras --reinstall

Copy link
Contributor Author

@kevinjqliu kevinjqliu Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id prefer to sync more deps than less, esp since any deps can be added to conftest.py that will be used here

Copy link
Contributor

@jayceslesar jayceslesar Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to use all deps for the make command.

Looking at it more closely there is sort of gray area of what is covered in the Makefile vs what is covered in CI. Should build logic be moved to the Makefile and then used in CI instead of being defined separately? (not a blocker for this PR)

Copy link
Contributor

@geruh geruh Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, with our current infra today we won't need to figure out the CI deps is missing an entry with nightly if we just build all-extras. If CI runtime ever becomes a concern we can try out something like this.

As for the Makefile, I think the current setup is mostly helpful on the development and release side of things

# Ignore tests for pypy since not all dependencies are compiled for it
# and would require a local rust build chain
CIBW_TEST_SKIP: "pp*"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/svn-build-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ jobs:
# Ignore 32 bit architectures
CIBW_ARCHS: "auto64"
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
CIBW_TEST_REQUIRES: "pytest==7.4.2 moto==5.0.1 pytest-lazy-fixture==0.6.3 sqlalchemy>=2.0.18,<3"
CIBW_TEST_COMMAND: "pytest {project}/tests/avro/test_decoder.py"
CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"
CIBW_TEST_COMMAND: "uv run --directory {project} pytest tests/avro/test_decoder.py"
# Ignore tests for pypy since not all dependencies are compiled for it
# and would require a local rust build chain
CIBW_TEST_SKIP: "pp*"
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ dev = [
"mypy-boto3-glue>=1.28.18",
"mypy-boto3-dynamodb>=1.28.18",
"pyarrow-stubs>=20.0.0.20251107", # Remove when pyarrow >= 23.0.0 https://github.com/apache/arrow/pull/47609
"sqlalchemy>=2.0.18,<3",
]
# for mkdocs
docs = [
Expand Down
2 changes: 2 additions & 0 deletions uv.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading