Getting PyGLM working on the Web with Pyodide

[szabolcsdombi]

Hello!

I would like to submit PyGLM as a Pyodide package.
This would mean to add a simple meta.yml pointing to the sdist on PyPI.

Previously the sdist was broken, thanks for the recent version it is fixed now.

There is still some changes that would be great to make before submitting.
When compiling for the web it is important to have all the internal functions / globals declared as static.

I tried to do this for glm but it seems not to be possible.
I have tweaked the setup.hpp to declare functions as either static or static inline.
It did not work due to operators cannot be declared static and there is no other define I could change.
Luckyily glm is behind a namespace, so name collision with other libraries are unlikely, (unless they also provide glm).

Here is the rest of the symbols that are not static:

_ZNKSt3__24hashIN3glm3matILi2ELi2EdLNS1_9qualifierE0EEEEclERKS4_
PTI1
_ZN13PyGLMTypeInfo4initEiP7_object
sourceType1
_Z16PyGLM_TestNumberP7_object
PTI0
sourceType0
_Z21PyGLM_Number_AsDoubleP7_object
_ZNKSt3__24hashIN3glm3vecILi3EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3vecILi4EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi3ELi2EdLNS1_9qualifierE0EEEEclERKS4_
PTI2
sourceType2
_ZNKSt3__24hashIN3glm3matILi4ELi2EdLNS1_9qualifierE0EEEEclERKS4_
PTI3
sourceType3
_Z20PyGLM_Number_AsFloatP7_object
_ZNKSt3__24hashIN3glm3matILi3ELi4EfLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi3EfLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi4EfLNS1_9qualifierE0EEEEclERKS4_
_Z19PyGLM_Number_AsLongP7_object
_ZNKSt3__24hashIN3glm3matILi4ELi4EiLNS1_9qualifierE0EEEEclERKS4_
_Z27PyGLM_Number_AsUnsignedLongP7_object
_ZNKSt3__24hashIN3glm3matILi4ELi4EjLNS1_9qualifierE0EEEEclERKS4_
_Z23PyGLM_Number_AsLongLongP7_object
_Z31PyGLM_Number_AsUnsignedLongLongP7_object
_ZNKSt3__24hashIN3glm3quaIdLNS1_9qualifierE0EEEEclERKS4_
_Z27PyGLM_CtypesVoidP_FromVoidPPv
ctypes_void_p
_Z34PyGLM_UnsignedLongLong_FromCtypesPP7_object
ctypes_cast
_Z15PyGLM_ToCtypesPPf
ctypes_float_p
_Z15PyGLM_ToCtypesPPd
ctypes_double_p
_Z15PyGLM_ToCtypesPPx
ctypes_int64_p
_Z15PyGLM_ToCtypesPPi
ctypes_int32_p
_Z15PyGLM_ToCtypesPPs
ctypes_int16_p
_Z15PyGLM_ToCtypesPPa
ctypes_int8_p
_Z15PyGLM_ToCtypesPPy
ctypes_uint64_p
_Z15PyGLM_ToCtypesPPj
ctypes_uint32_p
_Z15PyGLM_ToCtypesPPt
ctypes_uint16_p
_Z15PyGLM_ToCtypesPPh
ctypes_uint8_p
_Z15PyGLM_ToCtypesPPb
ctypes_bool_p
_Z15PyGLM_ToCtypesPPv
_Z15PyGLM_GetNumberP7_object
_Z32PyLong_AsUnsignedLongAndOverflowP7_objectPi
_Z36PyLong_AsUnsignedLongLongAndOverflowP7_objectPi
_Z11PyLong_SignP7_object
_Z30_PyGLM_Long_As_Number_No_ErrorIdET_P7_object
PyGLM_SHOW_WARNINGS
_Z30_PyGLM_Long_As_Number_No_ErrorImET_P7_object
_Z19PyGLM_Number_AsBoolP7_object
_Z30_PyGLM_Long_As_Number_No_ErrorIfET_P7_object
_Z25GET_PTI_COMPATIBLE_SIMPLEP7_objecti
_Z17vec_releasebufferP7_objectP9Py_buffer
_Z18mvec_releasebufferP7_objectP9Py_buffer
_Z17mat_releasebufferP7_objectP9Py_buffer
_Z17qua_releasebufferP7_objectP9Py_buffer
_Z12glmArray_setP8glmArraylP7_object
ctypes_float
ctypes_double
ctypes_int32
ctypes_uint32
ctypes_int8
ctypes_uint8
ctypes_int16
ctypes_uint16
ctypes_int64
ctypes_uint64
ctypes_bool
_Z12glmArray_getP8glmArrayl
_Z22glmArray_releasebufferP7_objectP9Py_buffer
PyInit_glm
ctypes_dealloc
PyGLM_VERSION_STRING
PyGLM_LICENSE_STRING
_ZN21PyGLMSingleTypeHolderC2EP7_object
_ZN21PyGLMSingleTypeHolder20getMostImportantTypeEiSt16initializer_listINS_5DTypeEE
_ZN21PyGLMSingleTypeHolder7asInt64Ev
_ZN21PyGLMSingleTypeHolder6asBoolEv
_ZN21PyGLMSingleTypeHolder8asDoubleEv
_ZN21PyGLMSingleTypeHolder8asUint64Ev
_ZN21PyGLMSingleTypeHolder7asInt32Ev
_ZN21PyGLMSingleTypeHolder8asUint32Ev
_ZN21PyGLMSingleTypeHolder7asInt16Ev
_ZN21PyGLMSingleTypeHolder8asUint16Ev
_ZN21PyGLMSingleTypeHolder6asInt8Ev
_ZN21PyGLMSingleTypeHolder7asUint8Ev
_ZN21PyGLMSingleTypeHolder7asFloatEv

It would be really nice to mark these symbols static. Except PyInit_glm.
Or at least simple ones like ctypes_float. Such a name might be also present in other libraries.

PS: sorry I could not demagle the names.

---

I have generated this list with the followind Dockerfile added in the root:

FROM python:3.11.3
ENV EMSDK=/opt/emsdk EMSDK_NODE=/opt/emsdk/node/16.20.0_64bit/bin/node \
    PATH=/opt/emsdk:/opt/emsdk/upstream/emscripten:/opt/emsdk/node/16.20.0_64bit/bin:$PATH
RUN git clone https://github.com/emscripten-core/emsdk.git $EMSDK &&\
    emsdk install 3.1.45 && emsdk activate 3.1.45 && pip install pyodide-build==0.24.1 &&\
    python -c "from pyodide_build.build_env import init_environment; init_environment()"

COPY glm /pyglm/glm
COPY glm-stubs /pyglm/glm-stubs
COPY PyGLM /pyglm/PyGLM
COPY setup.py setup.cfg LICENSE MANIFEST.in README.md PyGLM.cpp version.h build_options.h license.h /pyglm/
RUN pyodide build /pyglm -o /web/

RUN python -c "import glob; __file__ = ''; from pyodide_build.pywasmcross import calculate_object_exports_readobj as f; print(*f(glob.glob('/pyglm/build/temp.*/*.o')), sep='\n')" > /web/symbols.txt

# CMD python -m http.server -d web
# CMD cat /web/symbols.txt
CMD grep -vE '^_Z+N3glm' /web/symbols.txt

I can make a PR with the changes.
I first wanted to know if these changes would be welcome.

[Zuzu-Typ]

Hi there (:

I would like to submit PyGLM as a Pyodide package.

That would be great! Thank you for the help!

I can make a PR with the changes.
I first wanted to know if these changes would be welcome.

Thanks a lot for the work you've done already. The changes would indeed be welcome, however, since it's been a while since I've last worked on this project, I would like to take this opportunity myself to get in touch with PyGLM again - if that's alright with you (:
Since PyGLM is built to a single, enormous Object file (which is an issue in itself), there shouldn't be any issues when turning everything static (besides PyInit_glm).

[Zuzu-Typ]

I added the static keyword to any applicable symbol. With the changes in #225, the following symbols remain from the table you've provided:

_ZNKSt3__24hashIN3glm3matILi2ELi2EdLNS1_9qualifierE0EEEEclERKS4_
_ZN13PyGLMTypeInfo4initEiP7_object
_ZNKSt3__24hashIN3glm3vecILi3EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3vecILi4EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi3ELi2EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi2EdLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi3ELi4EfLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi3EfLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi4EfLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi4EiLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3matILi4ELi4EjLNS1_9qualifierE0EEEEclERKS4_
_ZNKSt3__24hashIN3glm3quaIdLNS1_9qualifierE0EEEEclERKS4_
PyInit_glm
_ZN21PyGLMSingleTypeHolderC2EP7_object
_ZN21PyGLMSingleTypeHolder20getMostImportantTypeEiSt16initializer_listINS_5DTypeEE
_ZN21PyGLMSingleTypeHolder7asInt64Ev
_ZN21PyGLMSingleTypeHolder6asBoolEv
_ZN21PyGLMSingleTypeHolder8asDoubleEv
_ZN21PyGLMSingleTypeHolder8asUint64Ev
_ZN21PyGLMSingleTypeHolder7asInt32Ev
_ZN21PyGLMSingleTypeHolder8asUint32Ev
_ZN21PyGLMSingleTypeHolder7asInt16Ev
_ZN21PyGLMSingleTypeHolder8asUint16Ev
_ZN21PyGLMSingleTypeHolder6asInt8Ev
_ZN21PyGLMSingleTypeHolder7asUint8Ev
_ZN21PyGLMSingleTypeHolder7asFloatEv

The remaining ones are mostly member functions of structs. I don't think they will cause any collisions.

[szabolcsdombi]

Since PyGLM is built to a single, enormous Object file (which is an issue in itself), there shouldn't be any issues when turning everything static (besides PyInit_glm).

When we compile for the web the entire project including all libraries and the cpython core itself will end up as "a single, enormous Object file"

I also learned this on a single file projects see

[Zuzu-Typ]

Oops, didn't mean to close this

[szabolcsdombi]

The remaining entries and the entries from glm:: will not cause any issues I think.
I checked and there is no way to hide them, not even with __attribute__((hidden))
I guess this would be the opposite of EMSCRIPTEN_KEEPALIVE.

I just built from the updated master. You can try PyGLM in the browser:

https://pyodide.org/en/stable/console.html

import micropip; await micropip.install('https://f003.backblazeb2.com/file/zengl-data/build/PyGLM-2.7.1-cp311-cp311-emscripten_3_1_45_wasm32.whl'); import glm

I can prepare the meta.yml when this source is also available on PyPI as an sdist.
No rush with a release, I will test some existing projects with this code in the meantime.