Skip to content

[AUTO-MERGE] Adding lance-to-parquet conversion notebook.#754

Merged
drewoldag merged 5 commits intomainfrom
issue/721/lance-to-parquet-notebook
Mar 6, 2026
Merged

[AUTO-MERGE] Adding lance-to-parquet conversion notebook.#754
drewoldag merged 5 commits intomainfrom
issue/721/lance-to-parquet-notebook

Conversation

@drewoldag
Copy link
Collaborator

Change Description

Added a notebook showing how to convert lance output files to parquet files.
Closes #721

@drewoldag drewoldag self-assigned this Mar 5, 2026
Copilot AI review requested due to automatic review settings March 5, 2026 01:06
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds documentation demonstrating how to convert Hyrax inference results stored on disk in Lance format into Parquet files, addressing the workflow requested in #721.

Changes:

  • Added a new notebook walking through Lance → Parquet conversion (in-memory and batched).
  • Linked the new workflow from the “Common workflows” docs index.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
docs/notebooks/convert_lance_to_parquet.ipynb New notebook demonstrating simple and batched conversion of Lance results to Parquet.
docs/common_workflows.rst Adds a toctree entry intended to surface the new conversion notebook in the docs.

You can also share your feedback on Copilot code review. Take the survey.

@codecov
Copy link

codecov bot commented Mar 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.66%. Comparing base (56f06cc) to head (84a661e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #754   +/-   ##
=======================================
  Coverage   64.66%   64.66%           
=======================================
  Files          61       61           
  Lines        5881     5881           
=======================================
  Hits         3803     3803           
  Misses       2078     2078           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Before [56f06cc] After [e280aa4] Ratio Benchmark (Parameter)
failed failed n/a data_cache_benchmarks.DataCacheBenchmarks.time_preload_cache_hsc1k
failed failed n/a data_cache_benchmarks.DataCacheBenchmarks.track_cache_hsc1k_hyrax_size_undercount
failed failed n/a data_request_benchmarks.DatasetRequestBenchmarks.time_request_all_data
failed 54.8±0s n/a vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(16384, 'qdrant')
9.65±0.03ms 10.5±0.2ms 1.09 vector_db_benchmarks.VectorDBSearchBenchmarks.time_search_by_vector_many_shards(128, 'chromadb')
1.96±0.07s 2.05±0.02s 1.05 benchmarks.time_visualize_help
1.92±0.02s 2.00±0.04s 1.04 benchmarks.time_database_connection_help
1.93±0.02s 1.98±0.06s 1.03 benchmarks.time_save_to_database_help
1.95±0s 2.02±0.04s 1.03 benchmarks.time_umap_help
1.96±0.03s 2.00±0.08s 1.02 benchmarks.time_lookup_help

Click here to view all benchmarks.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Collaborator

@aritraghsh09 aritraghsh09 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like these byte-sized notebooks!

@drewoldag drewoldag enabled auto-merge (squash) March 6, 2026 04:38
@drewoldag drewoldag changed the title Adding lance-to-parquet conversion notebook. [AUTO-MERGE] Adding lance-to-parquet conversion notebook. Mar 6, 2026
@drewoldag drewoldag merged commit 56390c8 into main Mar 6, 2026
6 of 7 checks passed
@drewoldag drewoldag deleted the issue/721/lance-to-parquet-notebook branch March 6, 2026 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Demo how to convert lance to parquet

3 participants