Fix SEGV w/ python and conda on macOS

[olivier-roussel]

Tackles #393
Clean some CMake files to fix multiple links to libpython which causes a SEGV at module import on macOS with some python versions.
I could not find a way to not distinguish the pybind dependency propagation between macOS (which need PRIVATE) and other OS (which needs PUBLIC).

[olivier-roussel]

Reset as draft as still not solved. The following python code is failing

import SofaRuntime
SofaRuntime.importPlugin("SofaPython3") 
[INFO]    [SofaPython3] Initializing with python version 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:46:07) [Clang 15.0.7 ]
[INFO]    [SofaPython3] Initializing python
[INFO]    [SofaPython3] Added '/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages' to sys.path
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages/SofaRuntime/__init__.py", line 7, in <module>
    from .SofaRuntime import *
SystemError: initialization of SofaRuntime failed without raising an exception
python3.10(62417,0x1193bddc0) malloc: *** error for object 0x10f2f60a0: pointer being freed was not allocated
python3.10(62417,0x1193bddc0) malloc: *** set a breakpoint in malloc_error_break to debug

########## SIG 6 - SIGABRT: usually caused by an abort() or assert() ##########

[olivier-roussel]

It looks like libSofaPython3 may not use the same python than other modules (such as Core or SofaRuntime). libSofaPython3 looks like to use the python from the base conda environment (here a v3.10.11) whereas all other modules are using the v3.10.13 installed in active conda environment. Current running python is the one from active environment and cmake was configured to use this python.
Looks like something is using the python directly available from opening a new shell (like executing a shell command), without the conda env activation.

Python 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:59) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import SofaRuntime
---------------------------------------
Checking SOFA_ROOT and SOFAPYTHON3_ROOT
Using environment variable SOFA_ROOT: /Users/ci/miniforge3/envs/sofa-dev-3
---------------------------------------
PYBIND11_MODULE Core Initializing with python version 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:59) [Clang 16.0.6 ]
PYBIND11_MODULE Core Initializing with python executable path: /Users/ci/miniforge3/envs/sofa-dev-3/bin/python
>>> SofaRuntime.importPlugin("SofaPython3")
[INFO]    [SofaPython3] Initializing with python version 3.10.11 | packaged by conda-forge | (main, May 10 2023, 19:16:08) [Clang 14.0.6 ]
[INFO]    [SofaPython3] Initializing python
----------- Initializing with python executable path: /Users/ci/miniforge3/envs/sofa-dev-3/bin/python
[INFO]    [SofaPython3] Added '/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages' to sys.path
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages/SofaRuntime/__init__.py", line 7, in <module>
    from .SofaRuntime import *
SystemError: initialization of SofaRuntime did not return an extension module
python(77144,0x1035cbdc0) malloc: *** error for object 0x1030fd0a0: pointer being freed was not allocated
python(77144,0x1035cbdc0) malloc: *** set a breakpoint in malloc_error_break to debug

Edit: This behavior of using two different python versions only happen on macOS. On a Linux, everything was fine.

[olivier-roussel]

Some updates...
It looks like libSofaPython3.dylib has an invalid rpath configured (or more exactly, too many rpath with some that are invalid, so basically looking at too many places). This lead to link at runtime with an invalid libpython:
dyld: loaded: <39D5CB93-185D-36D6-8E7B-7D0B0F194459> /Users/ci/miniforge3/envs/sofa-dev-3/lib/../../../lib/libpython3.10.dylib
where the multiple parent directory points one level too high (i.e. /Users/ci/miniforge3/lib/libpython3.10.dylib) which corresponds in conda to the lib directory of the base environment.
Patching the rpath of libSofaPython3.dylib manually to the right place ( dyld: loaded: <1E9707C0-19DB-36BD-8870-96645764AF16> /Users/ci/miniforge3/envs/sofa-dev-3/lib/libpython3.10.dylib ) indeed fix the problem of loading an inconsistent version of libpython when importing SofaPython3 plugin:

>>> import SofaRuntime
---------------------------------------
Checking SOFA_ROOT and SOFAPYTHON3_ROOT
Using environment variable SOFA_ROOT: /Users/ci/miniforge3/envs/sofa-dev-3
---------------------------------------
PYBIND11_MODULE Core Initializing with python version 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:59) [Clang 16.0.6 ]
PYBIND11_MODULE Core Initializing with python prefix /Users/ci/miniforge3/envs/sofa-dev-3/bin/python
>>> SofaRuntime.importPlugin("SofaPython3")
[INFO]    [SofaPython3] Initializing with python version 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:43:44) [Clang 16.0.6 ]
[INFO]    [SofaPython3] Initializing python
[INFO]    [SofaPython3] Added '/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages' to sys.path
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ci/miniforge3/envs/sofa-dev-3/lib/python3.10/site-packages/SofaRuntime/__init__.py", line 7, in <module>
    from .SofaRuntime import *
SystemError: initialization of SofaRuntime did not return an extension module
python(13377,0x10deb3dc0) malloc: *** error for object 0x10d0e30a0: pointer being freed was not allocated
python(13377,0x10deb3dc0) malloc: *** set a breakpoint in malloc_error_break to debug

########## SIG 6 - SIGABRT: usually caused by an abort() or assert() ##########

So, python versions are now correct, but this does fix the crash...

[olivier-roussel]

In any case, there is a fix to do in Sofa code arround here:
https://github.com/sofa-framework/sofa/blob/c469ffc0bc975bfdd2f874cb9d3482de08ad5c30/Sofa/framework/Config/cmake/SofaMacrosInstall.cmake#L558 .
Some rpath are apparently not correct.
Most probably there might be also some cleaning to do in SofaPython3 python modules as they are configured with a lot of rpath (arround 21 !), see here

[olivier-roussel]

To avoid duplicated libpython symbols (the ones embedded in statically linked python with libpython such as conda's one and other brought by dynamically linking with libpython), current SofaPython3 code is now, for the macOS case only :

• not linking dynamically python bindings modules with libpython (this is not required by pybind11, see https://pybind11.readthedocs.io/en/stable/compiling.html#building-with-cmake)
• not linking dynamically anymore libSofaPython3 with libpython. The link is normally required and done by linking the pybind11::embed target, which is expected as libSofaPython3 uses the python interpreter. Unless we split the current libSofaPython3 plugin to avoid the use case where we import this plugin into a python interpreter (leading to have a python interpreter within a python interpreter), the workaround here is to remove the dynamic link with libpython. This is done by customizing the linked pybind11 target. It uses pybind11::pybind11 (for python + pybind11 headers) + pybind11:python_link_helper to have the macOS specific linker option -unused dynamic_linking enabled.
  For the case where libSofaPython3 plugin must be loaded outside a python environment (so where no libpython symbols are loaded yet), such as importing it through the plugin manager in runSofa, python libraries must be preloaded first. This can be done in the plugin initialization C++ code (but requires a test to detect if symbols are loaded first), or manually by pre-loading libpython for runSofa, i.e. something like :
  DYLD_INSERT_LIBRARIES=$CONDA_PREFIX/lib/libpython3.10.dylib ./runSofa (path and python version should be adapted and automatically set by cmake for example)