Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pip install setup docs #2020

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Pip install setup docs #2020

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
204409b
DOC: Suggest installing Cython and numpy before zipline
richafrank Nov 21, 2017
20d9a22
DOC: Removed trailing whitespaces
richafrank Nov 21, 2017
0dc19bd
BLD: Don't even try to deal with setup dependencies ourselves
richafrank Nov 21, 2017
bcddf4b
BLD: Don't include setup_requires since we'll complain earlier
richafrank Nov 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions docs/source/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,17 @@ single tool to install Python and non-Python dependencies, or if you're already
using `Anaconda <http://continuum.io/downloads>`_ as your Python distribution,
you can skip to the :ref:`Installing with Conda <conda>` section.

Since building Zipline's C extensions requires numpy's CPython C API as well as
`Cython <http://cython.org/>`_, pip install these before pip installing zipline
itself:

.. code-block:: bash

$ pip install numpy Cython
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will give people the latest version of numpy. Do we support that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK we do. If we don't, then we should update setup.py's REQ_UPPER_BOUNDS.



Once you've installed the necessary additional dependencies (see below for
your particular platform), you should be able to simply run
your particular platform), you should be able to simply run:

.. code-block:: bash

Expand Down Expand Up @@ -126,10 +135,10 @@ channel:

Managing ``conda`` environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is recommended to install Zipline in an isolated ``conda`` environment.
Installing Zipline in ``conda`` environments will not interfere your default
Python deployment or site-packages, which will prevent any possible conflict
with your global libraries. For more information on ``conda`` environment, see
It is recommended to install Zipline in an isolated ``conda`` environment.
Installing Zipline in ``conda`` environments will not interfere your default
Python deployment or site-packages, which will prevent any possible conflict
with your global libraries. For more information on ``conda`` environment, see
the `Conda User Guide <https://conda.io/docs/user-guide/tasks/manage-environments.html>`_

Assuming ``conda`` has been set up, you can create a ``conda`` environment:
Expand All @@ -148,7 +157,7 @@ Assuming ``conda`` has been set up, you can create a ``conda`` environment:


Now you have set up an isolated environment called ``env_zipline``, a sandbox-like
structure to install Zipline. Then you should activate the conda environment
structure to install Zipline. Then you should activate the conda environment
by using the command

.. code-block:: bash
Expand Down
167 changes: 74 additions & 93 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
join,
)
from distutils.version import StrictVersion
from functools import partial
from setuptools import (
Extension,
find_packages,
Expand All @@ -33,94 +34,76 @@
import versioneer


class LazyBuildExtCommandClass(dict):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check for numpy/Cython being installed and fail with a clear error here if they're not?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the setup_requires for numpy/Cython as well, since they're the main thing that causes breakage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, as it currently is, setup.py develop is broken. Will need to dig into it if we want to remove even more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, updated.

"""
Lazy command class that defers operations requiring Cython and numpy until
they've actually been downloaded and installed by setup_requires.
"""
def __contains__(self, key):
return (
key == 'build_ext'
or super(LazyBuildExtCommandClass, self).__contains__(key)
conda_build = os.path.basename(sys.argv[0]) in ('conda-build', # unix
'conda-build-script.py') # win

if conda_build:
# If conda-build is running this, then we're currently expanding the jinja
# template in conda/zipline/meta.yaml, not actually installing. We don't
# have numpy or Cython yet, but luckily we only need the names from
# install_requires and build_requires.
ext_modules = []
else:
try:
import Cython # noqa
except ImportError:
raise Exception("Install Cython before zipline.")

try:
import numpy as np
except ImportError:
raise Exception("Install numpy before zipline.")

NumpyExtension = partial(Extension, include_dirs=[np.get_include()])

def window_specialization(typename):
"""Make an extension for an AdjustedArrayWindow specialization."""
return NumpyExtension(
'zipline.lib._{name}window'.format(name=typename),
['zipline/lib/_{name}window.pyx'.format(name=typename)],
depends=['zipline/lib/_windowtemplate.pxi'],
)

def __setitem__(self, key, value):
if key == 'build_ext':
raise AssertionError("build_ext overridden!")
super(LazyBuildExtCommandClass, self).__setitem__(key, value)

def __getitem__(self, key):
if key != 'build_ext':
return super(LazyBuildExtCommandClass, self).__getitem__(key)

from Cython.Distutils import build_ext as cython_build_ext
import numpy

# Cython_build_ext isn't a new-style class in Py2.
class build_ext(cython_build_ext, object):
"""
Custom build_ext command that lazily adds numpy's include_dir to
extensions.
"""
def build_extensions(self):
"""
Lazily append numpy's include directory to Extension includes.

This is done here rather than at module scope because setup.py
may be run before numpy has been installed, in which case
importing numpy and calling `numpy.get_include()` will fail.
"""
numpy_incl = numpy.get_include()
for ext in self.extensions:
ext.include_dirs.append(numpy_incl)

super(build_ext, self).build_extensions()
return build_ext


def window_specialization(typename):
"""Make an extension for an AdjustedArrayWindow specialization."""
return Extension(
'zipline.lib._{name}window'.format(name=typename),
['zipline/lib/_{name}window.pyx'.format(name=typename)],
depends=['zipline/lib/_windowtemplate.pxi'],
)


ext_modules = [
Extension('zipline.assets._assets', ['zipline/assets/_assets.pyx']),
Extension('zipline.assets.continuous_futures',
['zipline/assets/continuous_futures.pyx']),
Extension('zipline.lib.adjustment', ['zipline/lib/adjustment.pyx']),
Extension('zipline.lib._factorize', ['zipline/lib/_factorize.pyx']),
window_specialization('float64'),
window_specialization('int64'),
window_specialization('int64'),
window_specialization('uint8'),
window_specialization('label'),
Extension('zipline.lib.rank', ['zipline/lib/rank.pyx']),
Extension('zipline.data._equities', ['zipline/data/_equities.pyx']),
Extension('zipline.data._adjustments', ['zipline/data/_adjustments.pyx']),
Extension('zipline._protocol', ['zipline/_protocol.pyx']),
Extension(
'zipline.finance._finance_ext',
['zipline/finance/_finance_ext.pyx'],
),
Extension('zipline.gens.sim_engine', ['zipline/gens/sim_engine.pyx']),
Extension(
'zipline.data._minute_bar_internal',
['zipline/data/_minute_bar_internal.pyx']
),
Extension(
'zipline.data._resample',
['zipline/data/_resample.pyx']
),
Extension(
'zipline.pipeline.loaders.blaze._core',
['zipline/pipeline/loaders/blaze/_core.pyx'],
depends=['zipline/lib/adjustment.pxd'],
),
]
ext_modules = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, we have a lot of extensions. Probably not for this PR, but I wonder if we could consolidate these somehow. That might speed up some of our CI build times significantly I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, the pip install -e . of the builds takes 90+ seconds currently.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oof

NumpyExtension('zipline.assets._assets',
['zipline/assets/_assets.pyx']),
NumpyExtension('zipline.assets.continuous_futures',
['zipline/assets/continuous_futures.pyx']),
NumpyExtension('zipline.lib.adjustment',
['zipline/lib/adjustment.pyx']),
NumpyExtension('zipline.lib._factorize',
['zipline/lib/_factorize.pyx']),
window_specialization('float64'),
window_specialization('int64'),
window_specialization('int64'),
window_specialization('uint8'),
window_specialization('label'),
NumpyExtension('zipline.lib.rank', ['zipline/lib/rank.pyx']),
NumpyExtension('zipline.data._equities',
['zipline/data/_equities.pyx']),
NumpyExtension('zipline.data._adjustments',
['zipline/data/_adjustments.pyx']),
NumpyExtension('zipline._protocol', ['zipline/_protocol.pyx']),
NumpyExtension(
'zipline.finance._finance_ext',
['zipline/finance/_finance_ext.pyx'],
),
NumpyExtension('zipline.gens.sim_engine',
['zipline/gens/sim_engine.pyx']),
NumpyExtension(
'zipline.data._minute_bar_internal',
['zipline/data/_minute_bar_internal.pyx']
),
NumpyExtension(
'zipline.data._resample',
['zipline/data/_resample.pyx']
),
NumpyExtension(
'zipline.pipeline.loaders.blaze._core',
['zipline/pipeline/loaders/blaze/_core.pyx'],
depends=['zipline/lib/adjustment.pxd'],
),
]


STR_TO_CMP = {
Expand Down Expand Up @@ -271,25 +254,23 @@ def setup_requirements(requirements_path, module_names, strict_bounds,
return module_lines


conda_build = os.path.basename(sys.argv[0]) in ('conda-build', # unix
'conda-build-script.py') # win

setup_requires = setup_requirements(
'etc/requirements.txt',
('Cython', 'numpy'),
strict_bounds=conda_build,
conda_format=conda_build,
)

conditional_arguments = {
'setup_requires' if not conda_build else 'build_requires': setup_requires,
}
conditional_arguments = (
{'build_requires': setup_requires} if conda_build else {}
)


setup(
name='zipline',
url="http://zipline.io",
version=versioneer.get_version(),
cmdclass=LazyBuildExtCommandClass(versioneer.get_cmdclass()),
cmdclass=versioneer.get_cmdclass(),
description='A backtester for financial algorithms.',
entry_points={
'console_scripts': [
Expand Down