djrandom

DJRandom
========

DJRandom is an online music library application with a web interface,
designed for small groups of people who want to share their media
collections.  On one side, it's a half-joking educational experiment
in designing simple distributed systems (which is why it won't support
millions of users), but it has also proven to be capable of scaling to
a few machines with data in the Terabytes range.


Design
------

The application is designed do be modular, split into a few key
components using minimal interfaces:

* the database, which offers a plain key/value interface, with
  range-based scanning (the current implementation is single-hosted
  but very simple);

* the index, for full-text search on song metadata;

* the storage layer (a "blob store"), which holds the song data, again
  modeled as key/value pairs;

* a queuing system for background processing.

Each of these components can be replaced by clients to other, better,
systems. Many of such systems exists and are open source, and the
interfaces can adapt to the specific semantics a large number of them.

But the default djrandom implementation is self-contained and does not
depend on any external services. See the file services/README.rst for
further details on the various service implementations.

All background jobs are run as mapreduces: again the current
implementation is very simple and does not run workers on multiple
machines.

Building
--------

Base dependencies:

- Go (at least version 1.2)
- LevelDB

Note that the LevelDB version in Debian wheezy is too old with
respect to the Go bindings, so you might need to compile it from
source.

You'll need to check out the source repository at the right place
in your ``GOPATH``::

    $ go get -d git.autistici.org/ale/djrandom

Once this is done, choose whether you want to build the full suite
or just the clients (which have a much smaller set of dependencies).


Building the clients
~~~~~~~~~~~~~~~~~~~~

The DJRandom client suite need the PortAudio library (version 1.9
or higher) to talk to the audio device, and either ``ffmpeg`` or
the ``libav`` suite (``avconv``).

On a Debian-based system, ``apt-get install libav-tools portaudio19-dev``
should do.

* ensure that the GOPATH environment variable is set. For instance,
  the following command will use a directory called ``go`` under
  your $HOME:

    $ export GOPATH=$HOME/go

* install the client binaries::

    $ cd $GOPATH/src/git.autistici.org/ale/djrandom
    $ make
    $ go install ./client/...

You should now have ``djplay``, ``djupload`` and ``djmpd`` in ``$GOPATH/bin``.


Building the service
~~~~~~~~~~~~~~~~~~~~

Server-side components have quite a few dependencies on third-party
libraries, including one that is not packaged in distributions
and has to be built manually (imms):

- a basic C/C++ build environment
- GNU autotools
- GNU libtool
- FFTW (v3)
- PCRE
- IMMS (see below)

On a Debian system, install the required packages with::

    $ apt-get install build-essential autoconf automake libtool \
          pkg-config libfftw3-dev libpcre3-dev

On OSX, using ``brew``::

    $ brew install portaudio leveldb libtool fftw pcre

To build IMMS::

    $ go get -d git.autistici.org/ale/imms
    $ cd $GOPATH/src/git.autistici.org/ale/imms
    $ aclocal ; libtoolize ; automake --foreign --add-missing ; autoconf
    $ ./configure && make && sudo make install

(on OSX, you might have to use ``glibtoolize`` instead of ``libtoolize``).

You should now be able to build all the server-side tools with::

    $ cd ../djrandom
    $ make
    $ go install ./server/...
    $ go install ./mapreduce/...


Running the upload client
-------------------------

Create the ``~/.djrandom.conf`` configuration file with the location
of your media directory and your authentication credentials (an API
key and a secret), in JSON format::

    {
      "music_dir": "/home/user/Music",
      "auth_key": "abcdefghj....",
      "auth_secret": "blahblah...."
    }

Then find a way to start ``djuploader`` in the background on every
login, or whenever your machine starts. It will periodically wake up,
check the ``music_dir``, and upload whatever it finds.

It is possible to limit the bandwidth that the uploader is going to
use by setting the ``bw_limit`` flag to a value in KBytes/sec. For
other options just check ``djuploader --help``.


Running the search client
-------------------------

The ``djplay`` search client will perform a search on the server and
either print out the results in playlist format, or directly attempt
to play them to the audio device. It uses the same configuration file
as above.

It is best used together with a real audio player of some sort, for
instance::

    $ djplay --playlist Frank Zappa | vlc


Running the service
-------------------

A normal deployment usually consists of more than one node (separate
machine). Each service runs as a standalone process. The current
implementation has two different sets of processes:

* ``db_server`` and ``task_server`` must run on a single node (they
  are in fact not distributed services);

* ``index_server``, ``storage_server``, ``djproc`` and ``djfe``
  must run on every node.

Generate two partition tables (one for storage, one for the index)
with a sufficiently large number of partitions, to ensure an
approximately uniform distribution of data, and start the servers
using the ``djrandom.init`` script.

If you modify the partition tables, for instance to add a new node,
you should restart all the processes. Some data will be temporarily
unavailable while the nodes automatically rebalance in the background.