add documentation for pydefix in virtualenv

[glesur]

No description provided.

[neutrinoceros]

This has much more consequences and can easily create a worse situation for a user, especially if they see the export as a suggestion to hardcode this line into a .bashrc or similar config file.
I'd strongly advise against this.

[glesur]

I don't see where it says this should be in .bashrc. My question would be: do you have a better proposition?

[neutrinoceros]

to be more constructive, I would suggest making pydefix a proper Python package if it makes sense (and it doesn't, there is a bigger problem to address here)

[neutrinoceros]

I don't see where it says this should be in .bashrc.

I'm just saying this is typically where export statements from documentation / modifications to PYTHONPATH end up, and it's never¹ the right solution.

Footnotes

1. from the limited scope of my own experience ↩

[glesur]

the problem is not due to pydefix (a c++ class in idefix), but to pybind11, which is already a python package. On Idefix startup, pybind11 does not necessarily pick up the python interpreter from the current venv.

[neutrinoceros]

I think I need to study how this is implemented for a couple minutes (possibly longer) before I can meaningfully contribute to this discussion, but I strongly feel like I should. Can I get back to you later this week ?

[glesur]

for future reference:
the proposed fix comes from:
https://stackoverflow.com/questions/53480120/embedding-pybind11-with-virtual-environment

Similar issues may be found on the web, such as:
https://gitlab.kitware.com/autopybind11/autopybind11/-/issues/97

[svalat]

Hi,
I gave a short look and at least on my setup the virtualenv do not provide the libpython.so in venv/lib so for sure the embedded version of pybind11 load the one from the system. As a consequence it doesn't have the pythonpath set.

Can be confirmed by making

ldd idefix

Which gives :

libpython3.12.so.1.0 => /lib/x86_64-linux-gnu/libpython3.12.so.1.0 (0x000072d526c00000)

I think the idea from @glesur is a first step. But I would not try to guess what should be set by ourselves.

I would patch idefix to first make a call to the python executable to extract sys.path so we automatically get the same than the venv accounting the PYTHONPATH from the user, and then set it before starting pybind11 interpretor.

But to be clean, this patch should in principle go directly in pybind11, this is a miss from their side.

[neutrinoceros]

sorry for dropping the ball here (I'm swamped).

this patch should in principle go directly in pybind11, this is a miss from their side

Without understanding the details yet, this sounds like the right approach to me. Dirty patches on our side are okay if there's a path forwards that leads to removing them after the correct change was upstreamed.