-
Notifications
You must be signed in to change notification settings - Fork 8
Uploading Wheels (pip)
Versioning in PyPI is very strict. You need to follow a specific guideline when bumping the version number.
There is absolutely NO do-overs. Once you upload a version, that namespace is occupied forever. Therefore, we will follow post-release version scheme for minor bug fixes. When you wish to upload a new version without changes in version number, add .post<post-release-version-number> at the end. It is not at all unusual to upload several posts for a given release in this manner. Just increment the post number in the setup.py script.
pySBOL supports Microsoft Windows, Apple Mac OS X, and various Linux distros. Check each section for detailed instructions.
Example building 32-bit Python 3.6 library on Windows with VS2017:
cmake -DSBOL_BUILD_PYTHON3=TRUE -G"Visual Studio 15 2017" -DSWIG_EXECUTABLE="/swigwin-3.0.12/swig.exe" -DPYTHON_LIBRARY="C:\Program Files (x86)\Python36-32\libs\python36.lib" -DSBOL_BUILD_SHARED=FALSE
Common mistakes are specifying a dynamically linked library (libpython.a) when static build (python36.lib) should be specified. Also make sure the build config is set to Release not Debug in Visual Studio's toolbar.
Example building 64-bit Python 3.6 library on Windows with VS2017. Note this requires specifying the 64-bit CMake generator and the SBOL_BUILD_64 tag:
cmake -DSBOL_BUILD_PYTHON3=TRUE -G"Visual Studio 15 2017 Win64" -DSWIG_EXECUTABLE="/swigwin-3.0.12/swig.exe" -DPYTHON_LIBRARY="C:\Program Files\Python36\libs\python36.lib" -DSBOL_BUILD_64=TRUE -DSBOL_BUILD_SHARED=FALSE
The 32-bit Python 2.7 library:
cmake -DSBOL_BUILD_PYTHON2=TRUE -G"Visual Studio 15 2017" -DSWIG_EXECUTABLE="/swigwin-3.0.12/swig.exe" -DPYTHON_LIBRARY="C:\Python27\libs\python27.lib" -DSBOL_BUILD_SHARED=FALSE
The 64-bit Python 2.7 library:
PS C:\Users\Bryan Bartley\Dev\libSBOL> cmake -DSBOL_BUILD_PYTHON2=TRUE -G"Visual Studio 15 2017 Win64" -DSWIG_EXECUTABLE="/swigwin-3.0.12/swig.exe" -DPYTHON_LIBRARY="C:\Python27amd64\libs\python27.lib" -DSBOL_BUILD_SHARED=FALSE -DSBOL_BUILD_64=TRUE
To generate wheels and upload it to PyPI, you need Python, twine, and PyPI account with the right authentication. Make sure that the version number in setup.py is correct. For different versions of Python, it is recommended to use miniconda distributions, as linking the command console with different Python executables can become messy.
To build a Python wheel run:
python setup.py bdist_wheel --python-tag=cp27 --plat-name=win_amd64
in command console. Above line generates a Python wheel for 64-bit Python 2.7. Change flags if you want to generate wheels for different Python versions (cp35, cp36, etc.) or different architecture (win32). Once finished, a wheel will be generated under dist folder within sub-folder (Win_64_2, etc.) with name such as:
pySBOL-2.3.0.post1-cp27-none-win_amd64.whl
Make sure to rename ABI tag 'none' to 'cp27m' (or 'cp35m', 'cp36m', etc.), e.g.:
pySBOL-2.3.0.post1-cp27-cp27m-win_amd64.whl
To upload the wheels to PyPI, run:
twine upload <file-location> -u <username> -p <password>
It is also possible to configure a .pypirc file so that you don't have to type in your username and password every time when you upload. See this page for more info.
The process for building PyPI distributions on Mac parallels the process for Windows. The following command lines are useful:
python setup.py bdist_wheel --python-tag=cp27 --plat-name=macosx_10_11_x86_64
python3 setup.py bdist_wheel --python-tag=cp36 --plat-name=macosx_10_11_x86_64
python3 setup.py bdist_wheel --python-tag=cp35 --plat-name=macosx_10_11_x86_64
The wheel will be put in the dist directory. If one of the fields in the wheel directory is "none", change to cp27m (replace 27 with whatever Python version you used).
In order to build wheels for Python 3, it's necessary to configure some environment variables. For Python 3.6:
export PATH=/Library/Frameworks/Python.framework/Versions/3.6/bin:$PATH
export PYTHONPATH=/usr/local/lib/python3.6/site-packages:/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages:$PATH
For Python 3.5, I used my Anaconda interpreter:
export PYTHONPATH=$HOME/Users/bryan/anaconda/lib/python3.5/site-packages/:$PYTHONPATH
export PATH=$HOME/anaconda/bin:$PATH
To upload the package, switch back to a Python 2 interpreter and use twine :
twine upload pySBOL2-2.1.0-cp27-cp27m-macosx_10_11_x86_64.whl
Because there are numerous different Linux distros out there, generating wheels can be quite strenuous. Thus, we utilize manylinux to generate a manylinux-compatible wheel that will work on most of the modern Linux distros. To generate pySBOL wheels for Linux using manylinux container, you will need Docker. That means technically you don't need a Linux machine!
In order for maximum compatibility, current manylinux container is based on ancient CentOS 5. This means we will have to do a lot of tinkering to make this work! Following the commend line below step-by-step will lead you to a functional pySBOL manylinux wheel. Before generating wheels, make sure that the version number in setup.py is correct.
Running manylinux container
After installing Docker, run following lines in your terminal/command console. You will end up in manylinux container.
docker pull quay.io/pypa/manylinux1_x86_64
This will take 64 bit version of container. We will only support 64-bit architecture in Linux because nobody use 32-bit Python in Linux in 2018.
docker run -it quay.io/pypa/manylinux1_x86_64 bash
Now you will be inside the container.
Installing dependencies
Following command will install some of the necessary dependencies.
yum install git libxslt-devel pcre-devel
Building CURL
You need to rebuild CURL with PIC support.
curl --tlsv1.2 -L https://github.com/curl/curl/releases/download/curl-7_57_0/curl-7.57.0.tar.bz2 > curl-7.57.0.tar.bz2
tar xvjf curl-7.57.0.tar.bz2
cd curl-7.57.0
./configure --prefix=/curl-7.57.0/install --with-pic
make
make install
Building jsonCpp
cd ..
curl --tlsv1.2 -L https://github.com/open-source-parsers/jsoncpp/archive/1.7.7.tar.gz > jsoncpp-1.7.7.tar.gz
tar -xzf jsoncpp-1.7.7.tar.gz
cd jsoncpp-1.7.7
cmake28 . -Bbuild -DCMAKE_CXX_FLAGS=-fPIC -DCMAKE_C_FLAGS=-fPIC
cd build
make
make install
Building Raptor
cd ../..
curl --tlsv1.2 -L http://download.librdf.org/source/raptor2-2.0.15.tar.gz > raptor2-2.0.15.tar.gz
tar -xzf raptor2-2.0.15.tar.gz
cd raptor2-2.0.15
./configure --prefix=/usr/local --with-curl-config=/curl-7.57.0/curl-config --with-pic
make
make install
Building SWIG
You also need to build SWIG because the version that comes with CentOS 5 is very old.
cd ../..
curl --tlsv1.2 -L https://downloads.sourceforge.net/swig/swig-3.0.12.tar.gz > swig-3.0.12.tar.gz
tar -xzf swig-3.0.12.tar.gz
cd swig-3.0.12
./configure --prefix=/swig-3.0.12/swig_install
make
make install
Building pySBOL
cd ..
git clone https://github.com/SynBioDex/libSBOL.git
cd libSBOL
git submodule update --init --recursive
Use different Python distributions to generate wheels for various Python versions.
- Python 2.7
cmake28 . -Bbuild_27m -DCMAKE_INSTALL_PREFIX=/libSBOL/install_27m -DSBOL_BUILD_SHARED=FALSE -DSBOL_BUILD_PYTHON2=TRUE -DPYTHON_LIBRARY=/opt/python/cp27-cp27m/bin/python2.7 -DPYTHON_INCLUDE_DIR=/opt/python/cp27-cp27m/include/python2.7 -DSWIG_EXECUTABLE=/swig-3.0.12/swig_install/bin/swig -DCURL_LIBRARY=/curl-7.57.0/install/lib/libcurl.a -DCURL_INCLUDE_DIR=/curl-7.57.0/install/include -DCMAKE_CXX_FLAGS=-fPIC -DCMAKE_C_FLAGS=-fPIC
- Python 2.7 wide unicode
cmake28 . -Bbuild_27mu -DCMAKE_INSTALL_PREFIX=/libSBOL/install_27mu -DSBOL_BUILD_SHARED=FALSE -DSBOL_BUILD_PYTHON2=TRUE -DPYTHON_LIBRARY=/opt/python/cp27-cp27mu/bin/python2.7 -DPYTHON_INCLUDE_DIR=/opt/python/cp27-cp27mu/include/python2.7 -DSWIG_EXECUTABLE=/swig-3.0.12/swig_install/bin/swig -DCURL_LIBRARY=/curl-7.57.0/install/lib/libcurl.a -DCURL_INCLUDE_DIR=/curl-7.57.0/install/include -DCMAKE_CXX_FLAGS=-fPIC -DCMAKE_C_FLAGS=-fPIC
- Python 3.6
cmake28 . -Bbuild_36 -DCMAKE_INSTALL_PREFIX=/libSBOL/install_36m -DSBOL_BUILD_SHARED=FALSE -DSBOL_BUILD_PYTHON3=TRUE -DPYTHON_LIBRARY=/opt/python/cp36-cp36m/bin/python3.6 -DPYTHON_INCLUDE_DIR=/opt/python/cp36-cp36m/include/python3.6 -DSWIG_EXECUTABLE=/swig-3.0.12/swig_install/bin/swig -DCURL_LIBRARY=/curl-7.57.0/install/lib/libcurl.a -DCURL_INCLUDE_DIR=/curl-7.57.0/install/include -DCMAKE_CXX_FLAGS=-fPIC -DCMAKE_C_FLAGS=-fPIC
IMPORTANT: Under /opt/python/, you will find various directories dedicated to different Python distributions, e.g. cp27-cp27m, cp36-cp36m, etc. On top on that, you will also find mu variant of some distribution, e.g. cp27-cp27mu, etc. These are wide-unicode (ucs4) versions of Python. We will need to support both m and mu variants.
Now, run make under each build folder.
cd build_27m
make
make install
Building wheel
Use different Python to generate wheels for various Python versions.
cd ../install_27m/wrapper
Get setup.py script from pySBOL repository.
/opt/python/cp27-cp27m/bin/python2.7 setup.py bdist_wheel --python-tag=cp27
Use different Python to generate wheels for various Python versions. Make sure to change --python-tag=<cp27|cp36>.
cd dist
mv ./pySBOL-2.3.0.post11-cp27-none-any.whl ./pySBOL-2.3.0.post11-cp27-cp27m-manylinux1_x86_64.whl
Renaming the wheel is crucial.
IMPORTANT: Make sure to change ABI tag none to cp27mu, etc. when building wheels for mu distributions.
Checking and uploading wheel
Now, lets check whether our wheel is manylinux compatible. Run auditwheel on the wheel that was just generated.
auditwheel show ./pySBOL-2.3.0.post1-cp27-cp27m-manylinux1_x86_64.whl
The output will look like this:
pySBOL-2.3.0.post11-cp27-cp27m-manylinux1_x86_64.whl is consistent
with the following platform tag: "linux_x86_64".
The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libm.so.6 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.3', 'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.3.4',
'GLIBC_2.3.2'}, libstdc++.so.6 with versions {'GLIBCXX_3.4',
'CXXABI_1.3.1', 'GLIBCXX_3.4.5', 'CXXABI_1.3'}
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libm.so.6": "/lib64/libm-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
"libxml2.so.2": "/usr/lib64/libxml2.so.2.6.26",
"libxslt.so.1": "/usr/lib64/libxslt.so.1.1.17",
"libz.so.1": "/lib64/libz.so.1.2.3"
}
In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:
libxml2.so.2, libxslt.so.1, libz.so.1
Let's remove the last set of dependencies by using
auditwheel repair ./pySBOL-2.3.0.post1-cp27-cp27m-manylinux1_x86_64.whl
There might be an error, but a new wheel will be generated under wheelhouse folder regardless. Checking this wheel by using auditwheel show, you will see:
pySBOL-2.3.0.post11-cp27-cp27m-manylinux1_x86_64.whl is consistent
with the following platform tag: "manylinux1_x86_64".
The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libm.so.6 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.3.4', 'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.3.2',
'GLIBC_2.3'}, libstdc++.so.6 with versions {'CXXABI_1.3.1',
'GLIBCXX_3.4', 'GLIBCXX_3.4.5', 'CXXABI_1.3'}
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libm.so.6": "/lib64/libm-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}
Make sure that the wheel is consistent with the platform tag and there are no other dependencies to be eliminated. The above output indicates that the wheel is ready to be deployed. However, it is always a good idea to test the wheels first.
Next, upload the wheel using twine. You will have to install twine first, through pip under a Python distribution, e.g. /opt/python/cp27-cp27m/bin/pip. After running /opt/python/cp27-cp27m/bin/pip install twine, you can do something like:
/opt/python/cp27-cp27m/bin/twine upload <file-location> -u <username> -p <password>
Repeat this through all Python versions we support (for both m and mu). Right now, we support Python 2.7 and 3.6, so you will need to repeat the steps from building Python libraries few times. Make sure that you are consistent with the Python version you are using.