LAVA is written in Python, so you will need to know (or be willing to learn) the language. Likewise, the web interface is a Django application so you will need to use and debug Django if you need to modify the web interface. The pipeline model uses YAML (so you’ll need the YAML Parser) and Jinja2. All LAVA software is maintained in git, as are many of the support scripts, test definitions and job submissions. Some familiarity with Debian is going to be useful, helper scripts are available when preparing packages based on your modifications.
LAVA is complex and works to solve complex problems. This has implications for how LAVA is developed, tested, deployed and used.
The Django backend used with LAVA is PostgreSQL and some postgres-specific support is used. The LAVA UI has some use of Javascript and CSS. LAVA also uses ZMQ and XML-RPC and the LAVA documentation is written with RST.
In addition, test jobs and device support can involve use of U-Boot, GuestFS, ADB, QEMU, Grub, SSH and a variety of other systems and tools to access devices and debug test jobs.
Note
LAVA is developed using Debian packaging to ensure that
daemons and system-wide configuration is correctly updated with changes in
the codebase. There is no support for pypi or python virtual environments
or installing directly from a git directory. python-setuptools
is used
but only with sdist
to create the tarballs to be used for the Debian
packaging, not for install
. Some dependencies of LAVA are not available
with pypi, for example python-guestfs
.
See also
Developing LAVA on Debian and a summary of the Debian LAVA team activity
Developers can update the installed code on their own systems manually (by copying files into the system paths) and/or use symlinks where appropriate but changes need to be tested in a system which is deployed using the Developer package build before being proposed for review. All changes must also pass all the unit tests, unless those tests are already allowed to be skipped using unittest decorators.
Mixing the use of python code in /usr/local/lib
and /usr/lib
on a
single system is known to cause spurious errors and will only waste your
development time. Be very careful when copying files and when using symlinks.
If in doubt, remove /usr/local/lib/python*
and ~/.local/lib/python*
then build a local developer package and install it.
If your change introduces a dependency on a new python module, always ensure
that this module is available in Debian by searching the Debian package lists. If the module
exists but is not in the current stable release of Debian, it can be
backported but be aware that this will delay testing and acceptance of your
change. It is expressly not acceptable to add a dependency on a python
module which is only available using pypi or pip install
. Introducing such
a module to Debian can involve a large amount of work - talk to us before spending time on code which relies on such modules or
which relies on newer versions of the modules than are currently available in
Debian testing.
See also
Certain terms used in LAVA V2 have specific meanings, please be consistent in the use of the following terms:
A database object which collates similar devices into a group for purposes of scheduling. Devices of a single type are often the same vendor model but not all boards of the same model will necessarily be of the same device-type.
See also
lava-dispatcher
source package in
git and in Debian. The dispatcher software for LAVA V2 can be installed
without the server or the scheduler and a machine configured in this way is
also called a dispatcher.The name for the design of LAVA V2, based on how the actions to be executed by the dispatcher are arranged in a unidirectional pipe. The contents of the pipe are validated before the job starts and the description of all elements in the pipe is retained for later reference.
See also
lava-dispatcher
to interact with
external systems and daemons when a shell like environment is not supported.
Protocols need to be supported within the python codebase and currently
include multinode, LXC and vland.A singleton process which is solely responsible for assigning a device to a test job. The scheduler is common to LAVA V1 and LAVA V2 and performs checks on submission restrictions, device availability, device tags and schema compliance.
See also
lava-server
source package in git and
in Debian. It includes components from LAVA V1 and LAVA V2 covering the UI
and the scheduler daemon.LAVA online documentation is written with RST format. You can use the command below to generate html format files for LAVA V2:
$ cd lava-server/
$ make -C doc/v2 clean
$ make -C doc/v2 html
$ firefox doc/v2/_build/html/index.html
(or whatever browser you prefer)
We welcome contributions to improve the documentation. If you are considering adding new features to LAVA or changing current behaviour, ensure that the changes include updates for the documentation.
Wherever possible, all new sections of documentation should come with worked examples.
doc/v2/examples/test-jobs
doc/v2/examples/test-definitions
doc/v2/index.rst
file if you are adding new
pages or altering section headings.The ongoing migration complicates some of the workflow when it comes to finding all of the V2 code. When the V1 code is removed, the organisation of the code will be tidied up.
lava_scheduler_app
, lava_results_app
,
lava_server
, lava
and linaro_django_xmlrpc
components of LAVA V2.lava_dispatcher
and lava_test_shell
components. All LAVA V2 dispatcher code lives in
lava_dispatcher/pipeline
. Some lava_test_shell
scripts remain in the
top level lava_test_shell
directory with overrides in
pipeline/lava_test_shell
.There are also locations which provide device configurations to support the unit tests. Only the Jinja2 support is used by the installed packages,
The Jinja2 templates can be found in lava_scheduler_app/tests/device-types
in the lava-server
codebase. The reason for this is that all template
changes are checked in the unit-tests. When the package is installed, the
device-types
directory is installed into
/etc/lava-server/dispatcher-config/device-types/
. The contents of
lava_scheduler_app/tests/devices
is ignored by the packaging, these files
exist solely to support the unit tests.
Individual instances will each have their own locations for the device
dictionaries of real devices. To allow the unit tests to run, some device
dictionaries are exported into lava_scheduler_app/tests/devices
but there
is no guarantee that any of these would work with any real devices, even of
the declared device-type.
For example, the Cambridge lab stores each device dictionary in git at
https://git.linaro.org/lava/lava-lab.git and you can look at the configuration
of staging
as a reference:
https://git.linaro.org/lava/lava-lab.git/tree/HEAD:/staging.validation.linaro.org/lava/pipeline/devices
The lava-dispatcher
codebase also has local device configuration files in
order to support the dispatcher unit tests. These are not Jinja2 format,
these are YAML - the same YAML as would be sent to the dispatcher by the
relevant master after rendering the Jinja2 templates on that master. There is
no guarantee that any of the device-type or device configurations in the
lava-dispatcher
codebase would work with any real devices, even of the
declared device-type.
The best way to protect your investment on LAVA is to contribute your changes back. This way you don’t have to maintain the changes you need by yourself, and you don’t run the risk of LAVA changed in a way that is incompatible with your changes.
Upstream uses Debian, see Developing LAVA on Debian for more information.
The LAVA software team use git review
to manage contributions. Each review
is automatically tested against all the unit tests. All reviews must pass all
unit tests before being considered for merging into the master branch. The
contributor is responsible for making the changes necessary to allow the unit
tests to pass and to keep the review up to date with other changes in the
master branch.
To setup git review
for the first time, install the package and setup the
local git configuration. (This can take a little time.):
$ apt -y install git-review
$ cd lava-server/
$ git review -s
Important
All changes need to support both Debian unstable and Debian stable - currently Jessie. This often includes multiple versions of django and other supporting packages. Automated unit tests are run on stable (with backports).
The master branch may be significantly ahead of the latest packages available from Debian (unstable or stable backports) which are based on the release branch. Use the LAVA repositories and/or Developer build versions to ensure that your instance is up to date with master.
See also
If you’d like to offer a patch (whether it is a bug fix, documentation update, new feature or even a simple typo fix) it is best to follow this simple check-list:
Clone the master branch of the correct project.
Create a new, clean, local branch based on master:
$ git checkout -b fixupbranch
Add your code, change any existing files as needed.
Commit your changes on the local branch.
Checkout the master branch and git pull
Checkout your existing local branch:
$ git checkout fixupbranch
rebase your local branch against updated master:
$ git rebase master
Fix any merge conficts. #. Send the patch to the Linaro Code Review system (gerrit):
$ git review
If successful, you will get a link to a review.
Login to gerrit and add the lava-team
as reviewers.
The unit tests will automatically start and you will be notified by email of the results and a link to the output which is useful if the tests fail.
See also
Patch Submissions and workflow for detailed information.
You are welcome to use the bug tracker of your chosen distribution. The maintainer for the packages in that distribution should Register with Linaro as a Community contributor with Linaro (or already be part of Linaro) to be able to forward bug reports and patches into the upstream LAVA systems.
If you, or anyone on your team, would like to register with Linaro directly, this will allow you to file an upstream bug, submit code for review by the LAVA team, etc. Register at the following url:
If you are considering large changes, it is best to register and also to subscribe to the lava-devel mailing list and talk to us on IRC:
irc.freenode.net
#linaro-lava
You can use GitHub to fork the LAVA packages and make pull requests
It is worth sending an email to the lava-devel mailing list, so that someone can migrate the pull request to a review.
Note
You will need to respond to comments on the review and the process of updating the review is not linked to the github pull request process.