Setting Up a Local Musicbrainz Mirror

Introduction

This guide will show you how to set up your own local musicbrainz mirror from scratch, optionally using nginx as the frontend, although this is probably not necessary if your only connecting to it from a few machines.

It assumes you'll be using Debian Linux (also tested on Ubuntu Server 11.04), but you can substitute most of the apt-get commands with your package manager of choice.

The official install guide from musicbrainz can be found here: https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL

Prerequisites

1. UNIX-based operating system (here we'll be using Debian 6.0 or Ubuntu Server 11.04)
2. Git
3. Perl (at least v5.0)
4. PostgreSQL (using v8.4 here but replace the version below with the latest available for your distro)
5. Memcached

To grab all the prerequisites in one go, issue this command:

sudo apt-get install build-essential git-core libssl-dev libxml2-dev libpq-dev libexpat1-dev libdb-dev memcached libexpat-dev postgresql-8.4 postgresql-server-dev-8.4 postgresql-contrib liblocal-lib-perl libossp-uuid-perl libicu-dev

Installation and Setup

1.) Set up a musicbrainz user with a home in /home/musicbrainz:

adduser musicbrainz

su musicbrainz

2.) Git clone the latest musicbrainz-server:

cd /home/musicbrainz

git clone git://git.musicbrainz.org/musicbrainz-server.git musicbrainz-server

cd musicbrainz-server

3. Modify DBDefs.pm to run as a slave:

cp lib/DBDefs.pm.default lib/DBDefs.pm

nano lib/DBDefs.pm

Under "Directories", set MB_SERVER_ROOT to /home/musicbrainz/musicbrainz-server

Under "What type of server is this?", set REPLICATION_TYPE to { RT_SLAVE }

Note: To save the edited file simply hit Ctrl + x. Then hit the "y" key and then hit the enter key to save it.

4.) Install carton:

sudo cpan Carton

Note: You are going to get asked a few questions (dozens) along the way.

Just hit the ENTER key to continue. That is the same as answering YES to every question.

You are going to be building some required dependencies and in a moment you will install them.

5.) Install dependencies: sudo carton install --deployment

Note: Don't worry about these messages:

Found conflicting versions for package 'inc::Module::Install' inc/Module/Install.pm (1.04) Module/Install.pm (0)

Found conflicting versions for package 'inc::Module::Install::DSL' inc/Module/Install/DSL.pm (1.04) Module/Install/DSL.pm (0)

! Couldn't find module or a distribution Catalyst::Plugin::ErrorCatcher~v0.0.8.8 ()

! Couldn't find module or a distribution DBD::Pg~v2.18.0 ()

Installing modules failed

Create the database:

1.) Install PostgreSQL Extensions:

cd postgresql-musicbrainz-unaccent

sudo make

sudo make install

cd ..

2.) Build collate extension:

cd postgresql-musicbrainz-collate

sudo make

sudo make install

cd ..

3.) Set up PostgreSQL authentication:

Edit pg_hba.conf to allow all local connections:

# TYPE DATABASE USER CIDR-ADDRESS METHOD local all all trust

For IPv4/IPv6 local connections, you can make the same changes.

4.) Create the database:

carton exec ./admin/InitDb.pl -- --createdb --clean

Grab the latest dumps from: ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/

   "To get going, you need at least the mbdump.tar.bz2,
   mbdump-editor.tar.bz2 and mbdump-derived.tar.bz2 archives, but you can
   grab whichever dumps suit your needs. Assuming the dumps have been
   downloaded to /tmp/dumps/ you can import them with:

       carton exec ./admin/InitDb.pl -- --createdb --import /tmp/dumps/mbdump*.tar.bz2 --echo

5.) Start the server!

Just run:

carton exec -- plackup -Ilib -r

Visiting http://127.0.0.1:5000 should bring up the mirror.

Set up Headphones to access your local mirror

This part is easy :-)

Just select "custom" as your musicbrainz mirror on the config page and enter in "localhost" (if the server is running on your machine), or the ip address if it is on another machine, and the correct port (5000 by default).

You can change the sleep interval between requests - for example musicbrainz.org requires a 1 second sleep between requests, but if it's your local machine you can lower this to 0.

Hit 'Save Changes' and Headphones will start using your local mirror!

Post-Installation Notes

1.) You'll want to grab hourly updates from musicbrainz, which requires setting up a cron job to run, 10 minutes past the hour:

carton exec -- ./admin/replication/LoadReplicationChanges

2.) A handy script by Munger at #musicbrainz-devel on freenode allows you to start/stop the server and run hourly replication changes:

http://paste.pocoo.org/raw/555245/

Save it as /usr/bin/mbcontrol (chmod a+x to make it executable) Usage: mbcontrol start (start the server) mbcontrol stop (stop the server) mbcontrol hourly (load the replication changes)

To have in run as a cronjob every hour, you can run: crontab -e and stick this line in at the end: 10 * * * * /usr/bin/mbcontrol hourly (This will make it run 10 minutes past every hour)

Using nginx as the frontend

Note: Stop the server if it's already running

1. cd '/home/musicbrainz/musicbrainz-server/admin/nginx'

2. copy mbserver-rewrites.conf and nginx.conf into /etc/nginx (in nginx.conf, you can edit worker_processes to match the number of cores you have)

3. copy 001-musicbrainz & 002-beta into /etc/nginx/sites-available

4. Edit either 001-musicbrainz or 002-beta (whichever you prefer - I'm using 002-beta which uses a unix socket, whereas 001-musicbrainz uses a TCP socket to communicate with plack) to point to the correct root directory, since by default 002-beta uses a different musicbrainz-server directory

5. Symlink whichever config you decided to use into /etc/nginx/sites-enabled

6. Restart nginx

7. cd into /home/musicbrainz/musicbrainz-server and start plack:

for 001-musicbrainz:

'carton exec -- plackup -s FCGI --listen :55901'

for 002-beta:

'carton exec -- plackup -Ilib -s FCGI --listen fcgi.socket'

(make sure the port matches 001-musicbrainz or the location of fcgi.socket matches 002-beta, depending on which on you're using)

That's it!