Development Processes

This document describes two things:

• Local Builds
• Merging Upstream Changes

Local Builds

Prerequisites:

• git
• cmake >= 2.8.10
• zlib development files
• gcc and g++ >= 4.7

Checkout Repositories

To begin, clone Tokutek/ft-index, Tokutek/jemalloc and Tokutek/tokumxse:

git clone [email protected]:Tokutek/ft-index
git clone [email protected]:Tokutek/jemalloc
git clone [email protected]:Tokutek/tokumxse

Alternatively, use:

git clone https://github.com/Tokutek/ft-index.git
git clone https://github.com/Tokutek/jemalloc.git
git clone https://github.com/Tokutek/tokumxse.git

Now select the relevant git branch or git tag for each repo. Note: Most git branches are expected to be in flux and possibly unstable. Stable builds can be created from Release and Release Candidate tags that should match across the above repositories. To create one of these stable builds check out the most recent stable tag for each repo, for example:

cd tokumxse 
git checkout tokumxse-1.0.0-rc.4 
cd ..
cd ft-index
git checkout tokumxse-1.0.0-rc.4
cd ..
cd jemalloc 
git checkout tokumxse-1.0.0-rc.4 
cd ..

Build the Fractal Tree

First, create a symlink for jemalloc inside the fractal tree directory tree.

cd ft-index
ln -s $PWD/../jemalloc third_party/
cd ..

Now create the build environment for the fractal tree. You can create either a debug or release version. Once the respective build directory is assembled by cmake, you can run make install to build. Note: We want to "install" the resultant binaries somewhere specific using the CMAKE_INSTALL_PREFIX argument.

1. For a debug build:

   cd ft-index
   mkdir dbg
   cd dbg
   cmake \
       -D CMAKE_BUILD_TYPE=Debug \
       -D USE_VALGRIND=ON \
       -D TOKU_DEBUG_PARANOID=ON \
       -D BUILD_TESTING=OFF \
       -D CMAKE_INSTALL_PREFIX=$PWD/install \
       ..
   make install

2. For a release build:

   cd ft-index
   mkdir opt
   cd opt
   cmake \
       -D CMAKE_BUILD_TYPE=Release \
       -D USE_VALGRIND=OFF \
       -D TOKU_DEBUG_PARANOID=OFF \
       -D BUILD_TESTING=OFF \
       -D CMAKE_INSTALL_PREFIX=$PWD/install \
       ..
   make install

###Symlink the Fractal Tree

Next, make a symlink from inside the tokumxse repo to our chosen "install directory" where the recently built ft-index libraries live:

1. For a debug build:

   cd tokumxse
   ln -s $PWD/../ft-index/dbg/install src/third_party/tokuft

2. For a release build:

   cd tokumxse
   ln -s $PWD/../ft-index/opt/install src/third_party/tokuft

Build MXSE with SCons

Finally, invoke scons from inside the tokumxse repo with the following options;

NOTE: For TokuMX-SE release candidates 4 and lower need to use this SCons line to build:

scons --extrapath=$PWD/src/third_party/tokuft --tokuft --allocator=jemalloc mongod mongos mongo

NOTE: Mongo has removed the above option for adding extra directories for linking and header inclusion. The new API, used for newer TokuMX-SE release candidates, requires different arguments. You need to use this SCons line to build:

scons CPPPATH=$PWD/src/third_party/tokuft/include LIBPATH=$PWD/src/third_party/tokuft/lib --tokuft --allocator=jemalloc mongod mongos mongo

If you like, you can add --dbg=[on|off] --opt=[on|off] or -jN to scons. --mute is also quite nice.

Testing MXSE

You can run some C++ unit tests with scons:

scons --extrapath=$PWD/src/third_party/tokuft --tokuft --allocator=jemalloc smokeCppUnittests

You can also run some javascript tests with smoke.py once mongod, mongos, and mongo are built:

python buildscripts/smoke.py --storageEngine=tokuft jsCore

Apart from jsCore you can also run other suites, refer to this map for all of them.

Merging Upstream Changes

Upstream MongoDB manages multi-version development by maintaining separate branches for each stable minor version (v2.4, v2.6, v3.0), as well as an active development branch (master), and they cherry-pick any changes that need to hit multiple branches. We do the same for master and v3.0. About once a day, check for upstream changes (on branches master and v3.0), and if there are some, go ahead and merge them to tokumxse:

1. If you haven't set it up already, create an upstream remote in your git checkout of tokumxse:

   git remote add upstream https://github.com/mongodb/mongo

2. Fetch new commits from upstream:

   git fetch upstream

3. Make sure your master and v3.0 branches are up to date:

   git checkout master
   git pull
   git checkout v3.0
   git pull

4. Into each branch, merge the corresponding upstream branch:

   git checkout master
   git merge upstream/master
   git checkout v3.0
   git merge upstream/v3.0

5. Try building. If they've removed or added part of the API, you should notice with a failed compile. You can also watch the src/mongo/db/storage directory for interesting looking changes that might break things.

   Additionally, there's a mechanism that makes sure that the "assert ids" are unique. You may see a build fail with a message like this:

   DUPLICATE IDS: 28610
     ./src/mongo/db/storage/tokuft/tokuft_init.cpp:75:fassertFailedNoTrace(28610);
     ./src/mongo/db/repl/replication_coordinator_impl.cpp:2275:fassert(28610, cbh.getStatus());
   next id to use:28620
   

   In this case, this means that upstream, they've added a new assert id that conflicts with one we had already. To avoid conflicts, we should change ours, so in the above example, you'd go to tokuft_init.cpp and change 28610 to 28620. Then commit the change on either master or v3.0, and make sure you cherry-pick it to the other branch as well. For example, this particular conflict was fixed by this commit on master and this commit on v3.0.