Writing Tests

Introduction to the LAVA Test Developer Guide

This guide aims to enable users to be able to

  1. Submit desired jobs/tests on targets deployed where the LAVA server is located and report results.
  2. Understand how test job files need to be written so that jobs get submitted properly.
  3. Understand how test shell definitions need to be written.

Ensure you have read through the introduction on Writing a LAVA test definition.

Pay particular attention to sections on:

Assumptions at the start of this guide

  1. The desired board is already configured for use with a LAVA Server instance.
  2. A user account (username, password, email address) is already created by a LAVA administrator on your behalf, with permissions to submit jobs.
  3. lava-tool is already installed on your test system and a suitable token has been added.
  4. You are familiar with submitting jobs written by someone else, including viewing the logs file for a job, viewing the definition used for that job and accessing the complete log.

To install lava-tool, see lava-tool.

To authenticate lava-tool, see Authentication Tokens.

To find out more about submitting tests written by someone else, see Submitting your first job.

To find out more about viewing job details, see Job Submission.

Checking device availability

Use the LAVA scheduler to view available device types and devices. The main scheduler status page shows data for each device type as well as the currently active jobs. Also check the Devices pages:

  • All Devices - includes retired devices to which jobs cannot be submitted.
  • All Active Devices - lists only devices to which jobs can be submitted
  • All Devices Health - limited to just the latest health status of each device.
  • My Devices - available from your profile menu by clicking on your name once signed into the instance.

For a Writing MultiNode tests job, you may need to check more than one device type.

Devices are considered available for new jobs according to the Device status.

  • Idle, Reserved, Offline, Offlining - jobs can be submitted.
  • Restricted - only available for submissions made by declared users.
  • Retired - jobs will be rejected if all remaining devices of this type are retired.

Finding an image to run on the device

Start with an image which is already in use in LAVA. You can find one of these images by checking the device type in LAVA and viewing some of the jobs for devices of this type from the table on that page. e.g. for QEMU devices on validation.linaro.org:

https://validation.linaro.org/scheduler/device_type/qemu

Actions to be run for a LAVA test

There are three important sets of actions that will be run for a LAVA test:

  1. Deploy: The information needed to set up a device to boot a test image. Each device type supports a range of deployment methods.
  2. Boot: The steps to follow to start the test image on the device. Each device type supports a range of boot methods.
  3. Test: Run the lava test shell, running the specified tests.

Examples

Deploying a pre-built QEMU image

actions:
  - deploy:
      timeout:
        minutes: 5
      to: tmpfs
      images:
          rootfs:
            image_arg: -drive format=raw,file={rootfs}
            url: https://images.validation.linaro.org/kvm-debian-wheezy.img.gz
            compression: gz
      os: debian

Using device tags

A device tag marks a specified device as having specific hardware capabilities which other devices of the same device type do not. To test these capabilities, a Test Job can specify a list of tags which the device must support. If no devices exist which match all of the required tags, the job submission will fail. If devices support a wider range of tags than required in the Test Job (or the Test Job requires no tags), any of those devices can be used for the Test Job.

Note

Test jobs which use device tag support can only be submitted to instances which have those tags defined and assigned to the requested boards. Check the device information on the instance to get the correct tag information.

For singlenode jobs, tags are a top level list of strings in the job definition, the same level as job_name, timeouts, metadata and device_type:

# Your first LAVA JOB definition for an x86_64 QEMU
device_type: qemu
job_name: QEMU pipeline, first job

tags:
- tap_device
- virtual_io

timeouts:
  job:
    minutes: 15
  action:
    minutes: 5
priority: medium
visibility: public

# context allows specific values to be overridden or included
context:
  # tell the qemu template which architecture is being tested
  # the template uses that to ensure that qemu-system-x86_64 is executed.
  arch: amd64

metadata:
  # please change these fields when modifying this job for your own tests.
  docs-source: first-job
  docs-filename: qemu-pipeline-first-job.yaml

For multinode test jobs, the tags are defined as part of the Multinode protocol:

protocols:
  lava-multinode:
    roles:
      client:
        device_type: qemu
        context:
          arch: amd64
        count: 1
        # In this example, only one role in the group uses tags
        tags:
        - tap_device
        - virtual_io
      server:
        device_type: qemu
        context:
          arch: amd64
        count: 1
    timeout:
      seconds: 60

Device tags are only relevant during scheduling of the testjob and have no meaning to the dispatcher.

Using LAVA Test Shell

The lava_test_shell action provides a way to employ a black-box style testing approach with the target device. Its format is:

- test:
    failure_retry: 3
    name: kvm-basic-singlenode
    timeout:
      minutes: 5
    definitions:
        - repository:
            metadata:
                format: Lava-Test Test Definition 1.0
                name: smoke-tests-basic
                description: "Basic system test command for Linaro Ubuntu images"
            run:
                steps:
                    - printenv
          from: inline
          name: env-dut-inline
          path: inline/env-dut.yaml
        - repository: git://git.linaro.org/qa/test-definitions.git
          from: git
          path: ubuntu/smoke-tests-basic.yaml
          name: smoke-tests
        - repository: https://git.linaro.org/lava-team/lava-functional-tests.git
          from: git
          path: lava-test-shell/single-node/singlenode03.yaml
          name: singlenode-advanced

The “definitions” list here may contain multiple test definition URLs. These will all be run sequentially in one go; the system will not be rebooted between the definitions.

See also

lava_test_shell developer documentation