Fixed-Function Accelerators

The accel driver can be used for easy integration of custom fixed-function accelerators through a standardized hardware interface and a standardized procedure for enqueuing commands.

Interface

The control register interface for the fixed-function accelerators is quite simple. The address space of the device is split into four regions, the size of which is determined by the largest of the memories in these regions. Therefore, the region is selected with the highest bits of the address space of the accelerator:

High bits	Address Space
00	Control registers
01	Instruction memory
10	Data memory
11	Parameter memory

The size of the memories is read from the control registers, which is sufficient to determine the size of the address space of the accelerator as well as the offsets of each memory. The control registers are also used to control the execution of the accelerator:

Offset	Name	Description
0x000	STATUS	Status of the accelerator. Bit 0 is high when the execution is stalled due to any reason, bit 1 is high when the external stall signal is active, and bit 2 is high when the accelerator reset is active.
0x100	AQL_READ_IDX_LOW	Read index of the AQL queue (low 32 bits). Read only.
0x104	AQL_READ_IDX_HIGH	Read index of the AQL queue (high 32 bits). Read only.
0x108	AQL_WRITE_IDX_LOW	Write index of the AQL queue (low 32 bits). Writing to this register increments the 64-bit value.
0x10C	AQL_WRITE_IDX_HIGH	Write index of the AQL queue (high 32 bits). Read only.
0x200	COMMAND	Command register to control execution. Writing 1 to this register resets the accelerator, writing 2 lifts reset and external stall, and writing 4 enables the external stall signal, pausing execution.
0x300	DEVICE_CLASS	Device class (vendor ID) of the accelerator. Currently unused by the driver.
0x304	DEVICE_ID	Device ID of the accelerator. Currently unused by the driver.
0x308	INTERFACE_TYPE	Version number of the interface. This describes interface version 2.
0x30C	CORE_COUNT	Core count of the accelerator. Multicore devices are currently not supported.
0x310	CTRL_SIZE	Size of control memory (this register space) in bytes. Must be at least 1024.
0x314	DMEM_SIZE	Size of the data memory in bytes
0x318	IMEM_SIZE	Size of the instruction memory in bytes
0x31c	PMEM_SIZE	Size of the parameter memory in bytes.

The instruction memory can be used to configure the accelerator. However, it currently has to be done manually, and is not managed by pocl. The data memory is used to store an AQL Queue, as defined by the HSA Runtime Programmer’s Reference Manual, the write and read indexes of which are exposed by the control registers. The size of the queue is such that it uses all of the data memory. Finally, the parameter memory is used to store data and argument buffers as well as completion signals for the kernels.

As a practical example, enqueuing a kernel dispatch packet proceeds as follows:

• The driver allocates and populates the OpenCL buffers and the argument buffer for the kernel, as well as space for a 32-bit completion signal.

• The driver writes the kernel packet, excluding the header, to the device. Its position depends on the value of the write index. The completion signal address as well as the argument buffer address and pointers to buffer arguments are given as physical addresses in the accelerator’s address space. The kernel object simply corresponds to the kernel IDs shown in the table below.

• The driver sets the packet header and increments the queue write index.

• The device executes the kernel and writes a 1 in case of a success or a 2 in case of a failure to the completion signal address, if it is not 0.

• The driver sees the completion signal change, and can consider the command completed.

Usage

To enable this driver, simply add -DENABLE_ACCEL_DEVICE=1 to the cmake arguments. On small FPGA SoCs and other relatively low performance hosts, you may wish to follow the instructions in LLVM-less build.

The fixed-function accelerators need to be told what kernel to execute. For this, the accel driver has a list of builtin kernels that can be referred to in the clCreateProgramWithBuiltInKernels call:

Kernel name	Kernel ID	Function
pocl.copy	0	Copies from argument 0 to argument 1 as many bytes as there are work items
pocl.add32	1	32-bit element-wise addtion on arrays pointed to by arguments 0 and 1, stored in an array pointed to by argument 3
pocl.mul32	2	As pocl.add32, but with 32-bit multiplication

This list will be expanded in the future.

There is an example program using the accel driver in examples/accel which also includes the VHDL code for synthesizing the accelerator. The accelerator has been developed with the TCE toolset. In order to synthesize the accelerator for a Xilinx FPGA SoC, you can follow the instructions in the TCE manual, in the section titled System-on-a-Chip design with AlmaIF Integrator. Make sure to check the accelerator base address from Vivado.

Driver arguments are used to tell pocl where the accelerator is and what functions it supports. To run this example manually, execute:

POCL_DEVICES=accel POCL_ACCEL0_PARAMETERS=0x43C00000,1,2 ./accel_example

The environment variables define an accelerator with base physical address of 0x43C0_0000 that can execute pocl.add32 and pocl.mul32. When running the example, verify that the address given in the parameter matches the base address of the accelerator.

Note that as the driver requires write access to /dev/mem for memory mapping, you may need to execute the application with elevated privileges. In this case, note that sudo by default overrides your environment variables. You can either assign them in the same command, or use sudo with the --preserve-env switch.