Deploying debci to your own infrastructure
Architecture
The following picture represents the debci architecture:
-
The whole system communicates using an AMQP server, in this case
rabbitmq
, as a message queue. -
Test jobs are injected into a queue by the
debci-enqueue
command line tool. -
Another command-line tool, called
debci-batch
, can be used to periodically inject all test jobs for packages that were recently uploaded, or that had one or more of its dependencies updated. It reusesdebci-enqueue
to inject each test job. -
test jobs are picked up from the test queue by
debci-worker
daemons. They run the tests, and push the results back into a different queue. -
debci-collector
receives the test results, processes them, feeds the database with the results, and produces the HTML pages, atom feeds etc that in the web interface. -
Each component (
debci-collector
,debci-worker
,rabbitmq
, and the command line tools) can live in different hosts if necessary, as long as all of the debci components can connect to therabbitmq
server. -
You can have as many
debci-worker
nodes in the system as you want, but there must be only onedebci-collector
.
Deployment procedure
First install rabbitmq:
$ sudo apt install rabbitmq-server
Install the debci-collector:
$ sudo apt install debci-collector
The installation will prompt for the address of the AMQP server. If
debci-collector
is on the same host as
rabbitmq-server
, that prompt can be left blank.
On each worker node, install apt-cacher-ng
to cache package
downloads, and debci-worker
itself:
$ sudo apt install apt-cacher-ng debci-worker
As with debci-collector
, you will also be prompted for the
address of AMQP server, and can also leave it blank if the worker is on the
same server as rabbitmq-server
.
Note that when first installed, debci-worker
will first build
a testbed (a chroot, container, or a virtual machine image, depening on the
selected backend), and only after that is finished the worker will be able
to start processing test jobs.
Submitting test jobs
On any host that can connect to rabbitmq-server
, first install
debci
:
$ sudo apt install debci
As usual, you will prompted for the address of the AMQP server. If the
rabbitmq-server
is on the same host, just leave it blank.
Say you want to run the tests for the ruby-defauts
package. It
is as easy as
$ debci enqueue ruby-defaults
Scheduling job submission
By default, debci
will not submit any jobs. You need to decide
how, and how often, you want to submit jobs. A simple way of doing that is
using cron
.
Example 1: schedule tests for a set of packages once a day
# /etc/cron.d/debci
0 7 * * * debci debci enqueue rake rubygems-integration ruby-defaults
Example 2: schedule tests for all pending packages evety 4 hours:
# /etc/cron.d/debci
17 */4 * * * debci debci batch
You can also automate calls to debci enqueue
in any other way
you want.
Multiple worker processes per node
If you have worker nodes that have lots of CPUs and a large amount of RAM available you can run multiple worker daemons at once.
TODO
Setting up the LXC backend
TODO