Add more metrics to DataSourceExec for vortex format in datafusion
#5851
Replies: 3 comments 1 reply
-
|
Yeah this is definitely on my list, I'm planning on pushing some "completeness" datafusion stuff like this now that we finished merging the 51 upgrade which took longer than expected. |
Beta Was this translation helpful? Give feedback.
-
|
Its also related to some deeper changes in Vortex - I think the whole metrics things would be pretty nice if its a pluggable facade, but that's a bigger conversation that probably needs its own thread. |
Beta Was this translation helpful? Give feedback.
-
|
Starting to experiment with a potential Parquet -> Vortex migration, and indeed performance metrics would be very useful, notably pruning statistics / compute ressource consumption, which are core metrics we monitor in production |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
current only a few metrics show in DataSourceExec when search vortex format, for example

the parquet format have many useful metrics for debug the performance issue

Beta Was this translation helpful? Give feedback.
All reactions