Skip to content

Home

Jump to bottom
Eric Nawrocki edited this page Feb 20, 2019 · 59 revisions

Installation instructions for users

The file install.sh is an executable shell script that will install ribovore and its dependencies and then output important instructions for updating your environment variables so you can run the ribovore scripts.

More detailed instructions can be found in ribovore's README.txt

Instructions for developers

The instructions below are relevant only to developers who wish to develop ribovore or modify ribovore in some way. For users interested only in running ribovore, see the 'Installation instructions for users' above.

initial checkout

Ribovore depends on several perl modules (epn-options, epn-ofile, and epn-test) and other packages (rRNA_sensor and vecscreen_plus_taxonomy) all of which are also at github. It also depends on the Infernal software package (version 1.1.2) and blastn (version 2.8.1+). If you do not have Infernal installed, the install.sh script will install it. Or you can download it here. If you do not have blastn installed, the install-optional-blastn-linux.sh or install-optional-blastn-macosx.sh will install it, or you can download it here

blastn is only required for the ribosensor.pl and ribodbmaker.pl scripts.

For the ribovore code and its other dependencies, you can clone a github repositories. To do that for the first time, and get it set up:

   $ git clone https://github.com/nawrockie/ribovore.git
   $ git clone https://github.com/nawrockie/epn-options.git
   $ git clone https://github.com/nawrockie/epn-ofile.git
   $ git clone https://github.com/nawrockie/epn-test.git
   $ git clone https://github.com/aaschaffer/rRNA_sensor.git
   $ git clone https://github.com/aaschaffer/vecscreen_plus_taxonomy.git

and to set up your environment:

Add the following lines to your .bashrc file, replacing PATH-TO-X-CLONE with the directories created by the above git clone commands (or to paths where you installed Infernal and blasnt) for package/module X:

export RIBODIR="PATH-TO-RIBOVORE-CLONE"
export RIBOINFERNALDIR="PATH-TO-DIRECTORY-WITH-INFERNAL-BINARIES>"
export RIBOEASELDIR="PATH-TO-DIRECTORY-WITH-EASEL-MINIAPP-BINARIES>"
export RIBOTIMEDIR="PATH-TO-TIME(usually:/usr/bin)>"
export SENSORDIR="<PATH-TO-RRNA-SENSOR-CLONE>"
export EPNOPTDIR="<PATH-TO-EPN-OPTIONS-CLONE>"
export EPNOFILEDIR="<PATH-TO-EPN-OFILE-CLONE>"
export EPNTESTDIR="<PATH-TO-EPN-TEST-CLONE>"
export PERL5LIB="$RIBODIR:$EPNOPTDIR:$EPNOFILEDIR:$EPNTESTDIR:$PERL5LIB"
export PATH="$RIBODIR:$PATH"
export BLASTDB="$SENSORDIR:$BLASTDB"

Or the following lines to your .cshrc file

setenv RIBODIR "PATH-TO-RIBOVORE-CLONE"
setenv RIBOINFERNALDIR "PATH-TO-DIRECTORY-WITH-INFERNAL-BINARIES>"
setenv RIBOEASELDIR "PATH-TO-DIRECTORY-WITH-EASEL-MINIAPP-BINARIES>"
setenv RIBOTIMEDIR "PATH-TO-TIME(usually:/usr/bin)>"
setenv SENSORDIR "<PATH-TO-RRNA-SENSOR-CLONE>"
setenv EPNOPTDIR "<PATH-TO-EPN-OPTIONS-CLONE>"
setenv EPNOFILEDIR "<PATH-TO-EPN-OFILE-CLONE>"
setenv EPNTESTDIR "<PATH-TO-EPN-TEST-CLONE>"
setenv PERL5LIB "$RIBODIR":"$EPNOPTDIR":"$EPNOFILEDIR":"$EPNTESTDIR":"$PERL5LIB"
setenv PATH "$RIBODIR":"$PATH"
setenv BLASTDB "$SENSORDIR":"$BLASTDB"

You're set up on the stable master branches all the github-cloned packages. To do any development, you need to work on the develop branch(es), generally by creating and merging feature branches.

To switch to develop branches:

   $ cd ribovore
   $ git checkout develop
   $ cd ../epn-options
   $ git checkout develop
   $ cd ../epn-ofile
   $ git checkout develop
   $ cd ../epn-test
   $ git checkout develop
   $ cd ../rRNA_sensor
   $ git checkout develop
   $ cd ../vecscreen_plus_taxonomy
   $ git checkout develop

For information about our git workflow, read on.

git workflow

Ribovore uses the popular git workflow that's often just called "git flow". Go read the 2010 blog post by Vincent Driessen that describes it. But we use it with the difference that we don't mind having feature branches on origin.

In what follows, first we'll give concise-ish examples of the flow for normal development, making a release, and making a "hotfix". A summary of the principles and rationale follows the examples.

Normal development

Generally, for any changes you make to our code, you will make on a feature branch, off of develop. So first you create your branch:

   $ git checkout -b myfeature develop

Now you work, for however long it takes. You can make commits on your myfeature branch locally, and/or you can push your branch up to the origin and commit there too, as you see fit.

When you're done, and you've tested your new feature, you merge it to develop (using --no-ff, which makes sure a clean new commit object gets created), and delete your feature branch:

   $ git checkout develop                     
   $ git merge --no-ff -m "Merges myfeature branch into develop" myfeature
   $ git branch -d myfeature
   $ git push origin --delete myfeature
   $ git push origin develop

Small features: single commits can be made to develop

Alternatively, if you're sure your change is going to be a single commit, you can work directly on the develop branch.

   $ git checkout develop                  
     # make your changes
   $ git commit
   $ git push origin develop

Big features: keeping up to date with develop

If your work on a feature is taking a long time (days, weeks...), and if the develop trunk is accumulating changes you want, you might want to periodically merge them in:

   $ git checkout myfeature
   $ git merge --no-ff -m "Merges develop branch into myfeature branch" develop

Making a release

To make a release, you're going to make a release branch of the code, and of the epn-options, epn-ofile, epn-test, rRNA_sensor and vecscreen_plus_taxonomy repos if you made changes there as well. You assign appropriate version numbers to each, test and stabilize. When everything is ready, you merge to master and tag that commit with the version number; then you also merge back to develop, and delete the release branch.

For example, here's the git flow for a ribovore release, depending on epn-options, epn-ofile, epn-test, rRNA_sensor and vecscreen_plus_taxonomy. Suppose ribovore is currently at 0.03, and all of its dependencies (epn-options, epn-ofile, epn-test, rRNA_sensor and vecscreen_plus_taxonomy) are all currently at 0.02. Suppose we decide this release will be ribovore 0.1, and it does not depend on any new features in its dependencies, so we can use the last stable versions of those (this will be the head of the master epn-options, epn-ofile, epn-test, rRNA_sensor, vecscreen_plus_taxonomy git repos, which is what you should be using unless you made changes in any of those repos). To proceed we first go over to epn-options, epn-ofile epn-test, rRNA_sensor, and vecscreen_plus_taxonomy and just make tags:

   $ cd epn-options
   $ git checkout master
   $ git tag -a -m "Tags epn-options 0.02 for ribovore-0.1 release" ribovore-0.1
   $ git push origin ribovore-0.1
   $ cd epn-ofile
   $ git checkout master
   $ git tag -a -m "Tags epn-ofile 0.02 for ribovore-0.1 release" ribovore-0.1
   $ git push origin ribovore-0.1
   $ cd epn-test
   $ git checkout master
   $ git tag -a -m "Tags epn-test 0.02 for ribovore-0.1 release" ribovore-0.1
   $ git push origin ribovore-0.1
   $ cd rRNA_sensor
   $ git checkout master
   $ git tag -a -m "Tags rRNA_sensor 0.02 for ribovore-0.1 release" ribovore-0.1
   $ git push origin ribovore-0.1
   $ cd vecscreen_plus_taxonomy
   $ git checkout master
   $ git tag -a -m "Tags vecscreen_plus_taxonomy 0.02 for ribovore-0.1 release" ribovore-0.1
   $ git push origin ribovore-0.1

then go over and make a new release from ribovore's develop branch:

   $ cd ribovore
   $ git checkout develop # only necessary if you're not already on develop
   $ git checkout -b release-0.1 develop
     # follow the 'Release checklist' at the bottom of this page
     # do and commit any other work needed to test/stabilize ribovore release.
   $ git commit -a -m "Version number bumped to 0.1"

When you're finished merge the ribovore release branch as follows:

   $ git checkout master
   $ git merge --no-ff -m "Merges release-0.1 branch into master" release-0.1
   $ git tag -a -m "Tags 0.1 release" 0.1
   $ git push
   $ git push origin 0.1
      # Now merge release branch back to develop...
   $ git checkout develop
   $ git merge --no-ff -m "Merges release-0.1 branch into develop" release-0.1
   $ git push
   $ git branch -d release-0.1
   $ git push origin --delete release-0.1

Alternatively, what if our new release depends on some new features in epn-options. In this case, we need to make a new epn-options 0.03 release:

   $ cd epn-options
   $ git checkout develop # only necessary if you're not already on develop
   $ git checkout -b release-0.03 develop
     # Update version in epn-options.pm. Update date in README.
   $ git commit -a -m "Version number bumped to 0.03"
     # do and commit any other work needed to test/stabilize epn-options release

then go over and make the ribovore release as explained above in the example that bundled stable epn-options 0.02.

When the release is ready we need to merge the epn-options release branch:

   $ cd epn-options
   $ git checkout master
   $ git merge --no-ff -m "Merges release-0.03 branch into master" release-0.03
   $ git tag -a -m "Tags epn-options 0.03 release"
   $ git push origin 0.03
   $ git tag -a -m "Tags epn-options 0.03 for ribovore-0.1 release" ribovore-0.1  # This records that ribovore-0.1 depends on epn-options 0.03
   $ git push origin ribovore-0.1
       # Now merge release branch back to develop...
   $ git checkout develop
   $ git merge --no-ff -m "Merges release-0.03 branch into develop" release-0.03
   $ git push
   $ git branch -d release-0.03
   $ git push origin --delete release-0.03

and finally, we do the same pattern on ribovore itself (see above 'When you're finished merge the ribovore release branch as follows')

Dependencies always have a tag for their own release (epn-options 0.02), and may have additional tags for packages that depend on them (ribovore 0.1 bundles epn-options 0.02? Then there's a ribovore-0.1 tag pointing to that epn-options commit object).

Fixing bugs: "hotfix" branches

If you need to fix a critical bug and make a new release immediately, you create a hotfix release with an updated version number, and the hotfix release is named accordingly: for example, if we screwed up ribovore 0.03, hotfix-0.04 is the updated 0.04 release.

A hotfix branch comes off master, but otherwise is much like a release branch.

   $ cd ribovore
   $ git checkout -b hotfix-0.04 master                 
     # follow the 'Release checklist' at the bottom of this page
   $ git commit -a -m "Version number bumped to 0.04"

Now you fix the bug(s), in one or more commits. When you're done, the finishing procedure is just like a release:

    $ git checkout master              
    $ git merge --no-ff -m "Merges hotfix-0.04 branch into master" hotfix-0.04
    $ git push
    $ git tag -a -m "Tags 0.04 release" 0.04
    $ git push origin 0.04
    $ git checkout develop              
    $ git merge --no-ff -m "Merges hotfix-0.04 branch into develop" hotfix-0.04
    $ git push
    $ git branch -d hotfix-0.04

And make a tag in epn-options, epn-ofile, epn-test, rRNA_sensor, and vecscreen_plus_taxonomy:

   $ cd epn-options
   $ git checkout master
   $ git tag -a -m "Tags epn-options-0.02 for ribovore-0.04 release" ribovore-0.04
   $ git push origin ribovore-0.04
   $ cd epn-ofile
   $ git checkout master
   $ git tag -a -m "Tags epn-ofile-0.02 for ribovore-0.04 release" ribovore-0.04
   $ git push origin ribovore-0.04
   $ cd epn-test
   $ git checkout master
   $ git tag -a -m "Tags epn-test-0.02 for ribovore-0.04 release" ribovore-0.04
   $ git push origin ribovore-0.04
   $ cd rRNA_sensor
   $ git checkout master
   $ git tag -a -m "Tags rRNA_sensor 0.02 for ribovore-0.04 release" ribovore-0.04
   $ git push origin ribovore-0.04
   $ cd vecscreen_plus_taxonomy
   $ git checkout master
   $ git tag -a -m "Tags vecscreen_plus_taxonomy 0.02 for ribovore-0.04 release" ribovore-0.04
   $ git push origin ribovore-0.04

Summary of main principles

There are two long-lived Ribovore branches: origin/master, and origin/develop. All other branches have limited lifetimes.

master is stable. Every commit object on master is a tagged release, and vice versa.

develop is for ongoing development destined to be in the next release. develop should be in a close-to-release state. Another package (e.g. ribovore) may need to create a release of a downstream dependency (e.g. epn-options) at short notice. Therefore, commit objects on develop are either small features in a single commit, or a merge of a finished feature branch.

We make a feature branch off develop for any nontrivial new work -- anything that you aren't sure will be a single commit on develop. A feature branch:

  • comes from develop
  • is named anything informative (except master, develop, hotfix-* or release-*)
  • is merged back to develop (and deleted) when you're done
  • is deleted once merged

We make a release branch off develop when we're making a release. A release branch:

  • comes from develop
  • is named release-<version>, such as release-1.2
  • first commit on the hotfix branch consists of bumping version/date/copyright
  • is merged to master when you're done, and that new commit gets tagged as a release
  • is then merged back to develop too
  • is deleted once merged

We make a hotfix branch off master for a critical immediate fix to the current release. A hotfix branch:

  • comes from master
  • is named hotfix-<version>, such as hotfix-1.2.1
  • first commit on the hotfix branch consists of bumping version/date/copyright
  • is merged back to master when you're done, and that new commit object gets tagged as a release.
  • is then merged back to develop too
  • is deleted once merged

Release checklist

Optional steps only necessary for releases with significant changes since last release.

  • OPTIONAL: Update the taxonomy tree file:
  $ cd /panfs/pan1/dnaorg/rrna/git-ncbi-rrna-project/taxonomy-files
  $ update-for-ribodbmaker.sh
  $ cp ncbi-taxonomy-tree.ribodbmaker.txt RIBODIR/taxonomy/
  $ git add RIBODIR/taxonomy/ncbi-taxonomy-tree.ribodbmaker.txt
  • Update version and date in .pl scripts (including those in miniscripts/), version and date in README.txt
  • Update install*sh scripts with new version (ribovore, blastn?)
  • Update version (search for all occurrences) in README.txt and README-ncbi-internal.txt
  • OPTIONAL: Go through entire README.txt: rerun sample runs, update output in README.txt and also check via diff output files in testfiles/.
  • OPTIONAL: Attempt an installation with install.sh but you will have to manually modify the install.sh scripts to get the previous tag, or clone the current dir for ribovore and checkout the release branch the tag for the release you are about to do does not yet exist (but the tags for the dependencies should).

Post-release update of NCBI ribovore installation

   $ cd /panfs/pan1/dnaorg/ssudetection/code
   $ mv ribovore old-ribovore
   $ mkdir ribovore
     # if blast version has not changed:
   $ mv old-ribovore/ncbi-blast-2.8.1+ ribovore
   $ cd ribovore
   $ curl -L -O github.com/nawrockie/ribovore/raw/master/install.sh
   $ sh install.sh
     # remove infernal-1.1.2 and bin/ (infernal installation was not necessary)
   $ rm -rf bin; rm -rf lib; rm -rf share; rm -rf include
   $ rm -rf infernal-1.1.2
   $ rm -rf infernal-1.1.2.tar.gz
   # OPTIONALLY:
   $ rm -rf ../old-ribovore
Clone this wiki locally