Introduction to the LAVA Test Developer Guide

This guide aims to enable engineers to be able to

  1. Submit desired jobs/tests on target deployed where the LAVA server is located and report results.
  2. Understand how .json file need to be written so that jobs get submitted properly.
  3. Understand how test files (.yaml) need to be written.

Ensure you have read through the introduction on Writing a LAVA test definition.

Pay particular attention to sections on:

Assumptions at the start of this guide

  1. The desired board is already deployed at the LAVA Server location.
  2. A bundle stream for your test results is already created by the LAVA administrator on your behalf.
  3. A user account (username, password, email address) is already created by a LAVA administrator on your behalf, with permissions to submit jobs.
  4. You are running Ubuntu 12.04. All steps have been tested on Ubuntu 12.04LTS (precise pangolin) so if you are using some other variant or distribution your results may vary.
  5. lava-tool is already installed on your test system and a suitable token has been added.
  6. You are familiar with submitting jobs written by someone else, including viewing the log file for a job, viewing the definition used for that job and accessing the complete log.

If your desired board is not available in the LAVA instance you want to use, see Deploying a board in LAVA.

If a suitable bundle stream does not exist, see Bundle Stream Overview.

To install lava-tool, see lava-tool overview.

To authenticate lava-tool, see Authentication Tokens.

To find out more about submitting tests written by someone else, see Submitting your first job.

To find out more about viewing job details, see Job Submission.

Checking device availability

Use the LAVA scheduler to view available device types and devices. The main scheduler status page shows data for each device type as well as the currently active jobs. Also check the Devices pages:

  • All Devices - includes retired devices to which jobs cannot be submitted.
  • All Active Devices - lists only devices to which jobs can be submitted
  • All Devices Health - limited to just the latest health status of each device.

Note

One or more of the devices of the type you want could be a restricted device so that even if it is Idle, you may not have permission to submit to that device. See Device Capabilities Support.

For a Multi-Node LAVA job, you may need to check more than one device type.

Devices are considered available for new jobs according to the Device status.

  • Idle, Reserved, Offline, Offlining - jobs can be submitted.
  • restricted - only available for submissions made by declared users.
  • Retired - jobs will be rejected if all remaining devices of this type are retired.

Finding an image to run on the device

Start with an image which is already in use in LAVA. You can find one of these images by checking the device type in LAVA and viewing some of the jobs for devices of this type from the table on that page. e.g. for KVM devices on validation.linaro.org:

https://validation.linaro.org/scheduler/device_type/kvm

The job definition will provide information on the image or the Interactive boot commands which started the job.

Actions to be run for a LAVA test

  1. Deploy an image:
    1. Deploying an image to a device
    2. Deploying a Linaro kernel
    3. Interactive boot commands
  2. Boot the image into a test shell
    1. LAVA Test Shell - supports multiple Test definitions in version control.
  3. Submit results
    1. Initial actions in the JSON

Actions to be run to simply deploy without tests

  1. Deploy an image:
    1. Deploying an image to a device
    2. Deploying a Linaro kernel
    3. Interactive boot commands
  2. Boot the image without tests
    1. Booting a test image - no support for Test definitions in version control.
  3. Submit results
    1. Initial actions in the JSON

Examples

Deploying a pre-built image to a device

{
   "job_name": "panda-prebuilt",
   "target": "panda01",
   "timeout": 18000,
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "image": "http://releases.linaro.org/12.09/ubuntu/leb-panda/lt-panda-x11-base-precise_ubuntu-desktop_20120924-329.img.gz"
           }
       },
       {
           "command": "boot_linaro_image"
       },
       {
           "command": "submit_results",
           "parameters": {
               "server": "http://localhost/lava-server/RPC2/",
               "stream": "/anonymous/test/"
           }
       }
   ]
}

Deploying an image using linaro-media-create

{
   "job_name": "panda-lmc",
   "target": "panda01",
   "timeout": 18000,
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "rootfs": "http://releases.linaro.org/12.09/ubuntu/precise-images/nano/linaro-precise-nano-20120923-417.tar.gz",
               "hwpack": "http://releases.linaro.org/12.09/ubuntu/leb-panda/hwpack_linaro-lt-panda-x11-base_20120924-329_armhf_supported.tar.gz"
           }
       },
       {
           "command": "boot_linaro_image"
       },
       {
           "command": "submit_results",
           "parameters": {
               "server": "http://localhost/lava-server/RPC2/",
               "stream": "/anonymous/test/"
           }
       }
   ]
}

Deploying an Android image

{
    "job_name": "android_test",
    "device_type": "panda",
    "timeout": 900
    "actions": [
        {
            "command": "deploy_linaro_android_image",
            "parameters": {
                "images": [
                    {
                        "partition": "boot",
                        "url": "http://releases.linaro.org/12.07/android/leb-panda/boot.tar.bz2"
                    },
                    {
                        "partition": "userdata",
                        "url": "http://releases.linaro.org/12.07/android/leb-panda/userdata.tar.bz2"
                    },
                    {
                        "partition": "system",
                        "url": "http://releases.linaro.org/12.07/android/leb-panda/system.tar.bz2"
                    }
                ]
            }
        },
        {
            "command": "boot_linaro_android_image",
            "parameters": {
                "options": [ "boot_cmds=boot_cmds_android" ]
            }
        },
        {
            "command": "submit_results",
            "parameters": {
                "server": "http://validation.linaro.org/RPC2/",
                "stream": "/anonymous/lava-android-leb-panda/"
            }
        }
    ]
}

Using device tags

A device tag marks a specified device as having specific hardware capabilities which other devices of the same device type do not. To test these capabilities, a Test Job can specify a list of tags which the device must support. If no devices exist which match all of the required tags, the job submission will fail. If devices support a wider range of tags than required in the Test Job (or the Test Job requires no tags), any of those devices can be used for the Test Job.

Note

Test jobs which use device tag support can only be submitted to instances which have those tags defined and assigned to the requested boards. Check the device information on the instance to get the correct tag information.

Singlenode example

{
   "job_name": "panda-lmc",
   "target": "panda01",
   "timeout": 18000,
   "tags": [
       "hdmi",
       "usbstick"
   ]
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "rootfs": "http://releases.linaro.org/12.09/ubuntu/precise-images/nano/linaro-precise-nano-20120923-417.tar.gz",
               "hwpack": "http://releases.linaro.org/12.09/ubuntu/leb-panda/hwpack_linaro-lt-panda-x11-base_20120924-329_armhf_supported.tar.gz"
           }
       },
       {
           "command": "boot_linaro_image"
       },
       {
           "command": "submit_results",
           "parameters": {
               "server": "http://localhost/RPC2/",
               "stream": "/anonymous/test/"
           }
       }
   ]
}

Multinode example

{
   "health_check": false,
   "logging_level": "DEBUG",
   "timeout": 900,
   "job_name": "multinode-example",
   "device_group": [
       {
           "role": "felix",
           "count": 1,
           "device_type": "kvm",
           "tags": [
               "hdmi",
               "sata"
           ]
       },
       {
           "role": "rex",
           "count": 1,
           "device_type": "kvm",
           "tags": [
               "usbstick"
           ]
       }
   ],
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "image": "http://images.validation.linaro.org/kvm-debian-wheezy.img.gz"
           }
       },
       {
           "command": "boot_linaro_image"
       },
       {
           "command": "submit_results_on_host",
           "parameters": {
               "stream": "/anonymous/codehelp/",
               "server": "http://localhost/RPC2/"
           }
       }
   ]
}

Installing Binary Blobs

Some Android builds for Panda require a binary blob to be installed. This can be done by adding the android_install_binaries after the deploy_linaro_android_image:

{
   "actions": [
       {
           "command": "deploy_linaro_android_image",
           "parameters": {
               "boot": "http://releases.linaro.org/12.09/android/leb-panda/boot.tar.bz2",
               "system": "http://releases.linaro.org/12.09/android/leb-panda/system.tar.bz2",
               "data": "http://releases.linaro.org/12.09/android/leb-panda/userdata.tar.bz2"
           }
       },
       {
           "command": "android_install_binaries"
       }
   ]
}

Executing Tests on Android

Tests are executed on Android by adding a lava_android_test_install and lava_android_test_run action to your base job file:

{
   "actions": [
       {
           "command": "lava_android_test_install",
           "parameters": {
               "tests": [
                   "busybox"
               ]
           }
       },
       {
           "command": "boot_linaro_android_image"
       },
       {
           "command": "lava_android_test_run",
           "parameters": {
               "test_name": "busybox"
           }
       }
   ]
}

Using LAVA Test Shell

The lava_test_shell action provides a way to employ a more black-box style testing approach with the target device. The action only requires that a deploy action (deploy_linaro_image/deploy_linaro_android_image) has been executed. Its format is:

{
   "command": "lava_test_shell",
   "parameters": {
       "testdef_urls": [
           "http://people.linaro.org/~doanac/lava/lava_test_shell/testdefs/lava-test.yaml"
       ],
       "timeout": 1800
   }
}

You can put multiple test definition URLs in “testdef_urls” section. The “testdef_urls” section takes a list of strings which are URLs. These will be run sequentially without reboot. Alternatively, you can specify each URL in a separate lava_test_shell action which will allow for a reboot between each test.

But “testdef_urls” can’t support “parameters” function. To pass parameters to variables in YAML files, see Test definitions in version control.

If your test definitions can be downloaded by a url link then lava_test_shell can automatically download the test definition from the url and execute it. The format is:

{
   "command": "lava_test_shell",
   "parameters": {
       "testdef_repos": [
           {
               "url": "file:///home/tekkamanninja/json_file/repo_parameter/lmp-test-c.yaml",
               "parameters": {
                   "TEST_1": "pass"
               }
           }
       ],
       "timeout": 1800
   }
}

Caution

When using “url” in “testdef_repos”, lava-dispatcher will ignore `revision` and `testdef`.

If your test definitions are available in a git repository then lava_test_shell can automatically pull the test definition from the git repository and execute it. The format is:

{
   "command": "lava_test_shell",
   "parameters": {
       "testdef_repos": [
           {
               "git-repo": "git://git.linaro.org/people/stylesen/sampletestdefs.git",
               "revision": "91df22796f904677c0fe5df787fc04234bf97691",
               "testdef": "testdef.yaml",
               "parameters": {
                   "TEST_1": "pass"
               }
           }
       ],
       "timeout": 1800
   }
}

Alternatively, if your test definitions are available in a bzr repository then lava_test_shell can automatically pull the test definition from the bzr repository and execute it. The format is:

{
   "command": "lava_test_shell",
   "parameters": {
       "testdef_repos": [
           {
               "bzr-repo": "lp:~stylesen/lava-dispatcher/sampletestdefs-bzr",
               "revision": "1",
               "testdef": "testdef.yaml",
               "parameters": {
                   "TEST_1": "pass"
               }
           }
       ],
       "timeout": 1800
   }
}

In both the above formats “revision”, “testdef” and “parameters” are optional. If “revision” is not specified then the latest revision in the repository is cloned. If there is no “testdef” specified, then inside the cloned directory of the repository a file with name “lavatest.yaml” is looked up which is the default name for test definitions. The “testdef” parameter could be used in order to override the default name for test definition file. The “parameters” could be used in order to pass the parameters for those variables, If your test definition file include Shell variables in “install” and “run” sections.

See also

lava_test_shell developer documentation

Passing parameters to test definition variables, the format should be like this:

{
   "parameters": {
       "VARIABLE_NAME_1": "value_1",
       "VARIABLE_NAME_2": "value_2"
   }
}

If you didn’t use “parameters” here, the lava-dispatcher will use the default values that defined in your test definition file.

Adding Meta-Data

Both deploy actions support an optional field, metadata. The value of this option is a set of key-value pairs like:

{
   "command": "deploy_linaro_image",
   "parameters": {
       "image": "http://releases.linaro.org/12.09/ubuntu/leb-panda/lt-panda-x11-base-precise_ubuntu-desktop_20120924-329.img.gz"
   },
   "metadata": {
       "ubuntu.image_type": "ubuntu-desktop",
       "ubuntu.build": "61"
   }
}

This data will be uploaded into the LAVA dashboard when the results are submitted and can then be used as filter criteria for finding data.