Core vs Runtime libraries

The Core library is a low level collection of algorithms implementations, it is designed to be embedded in existing projects and applications:

It doesn't allocate any memory (All the memory allocations/mappings have to be handled by the caller).
It doesn't perform any kind of multi-threading (but provide information to the caller about how the workload can be split).

The Runtime library is a very basic wrapper around the Core library which can be used for quick prototyping, it is basic in the sense that:

It allocates images and tensors by using standard malloc().
It multi-threads NEON code in a very basic way using a very simple pool of threads.
For OpenCL it uses the default CLScheduler command queue for all mapping operations and kernels.

For maximum performance, it is expected that the users would re-implement an equivalent to the runtime library which suits better their needs (With a more clever multi-threading strategy, load-balancing between NEON and OpenCL, etc.)

Windows, kernels, multi-threading and functions

Windows

A Window represents a workload to execute, it can handle up to Coordinates::num_max_dimensions dimensions. Each dimension is defined by a start, end and step.

It can split into subwindows as long as all the following rules remain true for all the dimensions:

max[n].start() <= sub[n].start() < max[n].end()
sub[n].start() < sub[n].end() <= max[n].end()
max[n].step() == sub[n].step()
(sub[n].start() - max[n].start()) % max[n].step() == 0
(sub[n].end() - sub[n].start()) % max[n].step() == 0

Kernels

Each implementation of the IKernel interface (base class of all the kernels in the core library) works in the same way:

OpenCL kernels:

// Initialize the CLScheduler with the default context and default command queue
// Implicitly initializes the CLKernelLibrary to use ./cl_kernels as location for OpenCL kernels files and sets a default device for which OpenCL programs are built.
CLScheduler::get().default_init();
cl::CommandQueue q = CLScheduler::get().queue();
//Create a kernel object:
MyKernel kernel;
// Initialize the kernel with the input/output and options you want to use:
kernel.configure( input, output, option0, option1);
// Retrieve the execution window of the kernel:
const Window& max_window = kernel.window();
// Run the whole kernel in the current thread:
kernel.run( q, max_window ); // Enqueue the kernel to process the full window on the default queue
// Wait for the processing to complete:
q.finish();

NEON / CPP kernels:

//Create a kernel object:
MyKernel kernel;
// Initialize the kernel with the input/output and options you want to use:
kernel.configure( input, output, option0, option1);
// Retrieve the execution window of the kernel:
const Window& max_window = kernel.window();
// Run the whole kernel in the current thread:
kernel.run( max_window ); // Run the kernel on the full window

Multi-threading

The previous section shows how to run a NEON / CPP kernel in the current thread, however if your system has several CPU cores, you will probably want the kernel to use several cores. Here is how this can be done:

    ThreadInfo info;
    info.cpu_info = &_cpu_info;
    const Window      &max_window     = kernel->window();
    const unsigned int num_iterations = max_window.num_iterations(split_dimension);
    info.num_threads                  = std::min(num_iterations, _num_threads);
    if(num_iterations == 0)
    {
        return;
    }
    if(!kernel->is_parallelisable() || info.num_threads == 1)
    {
        kernel->run(max_window, info);
    }
    else
    {
        int  t         = 0;
        auto thread_it = _threads.begin();
        for(; t < info.num_threads - 1; ++t, ++thread_it)
        {
            Window win     = max_window.split_window(split_dimension, t, info.num_threads);
            info.thread_id = t;
            thread_it->start(kernel, win, info);
        }
        // Run last part on main thread
        Window win     = max_window.split_window(split_dimension, t, info.num_threads);
        info.thread_id = t;
        kernel->run(win, info);
        try
        {
            for(auto &thread : _threads)
            {
                thread.wait();
            }
        }
        catch(const std::system_error &e)
        {
            std::cerr << "Caught system_error with code " << e.code() << " meaning " << e.what() << '\n';
        }
    }

This is the very basic implementation used in the NEON runtime library by all the NEON functions.

See also: CPPScheduler.

Note: Some kernels like for example NEHistogramKernel need some local temporary buffer to perform their calculations. In order to avoid memory corruption between threads, the local buffer must be of size: memory_needed_per_thread * num_threads and a unique thread_id between 0 and num_threads must be assigned to the ThreadInfo object passed to the run function.

Functions

Functions will automatically allocate the temporary buffers mentioned above, and will automatically multi-thread kernels' executions using the very basic scheduler described in the previous section.

Simple functions only call a single kernel (e.g NEConvolution3x3), while more complex ones consist of several kernels pipelined together (e.g NEGaussianPyramid, NEHarrisCorners). Check their documentation to find out which kernels are used by each function.

//Create a function object:
MyFunction function;
// Initialize the function with the input/output and options you want to use:
function.configure( input, output, option0, option1);
// Execute the function:
function.run();

Warning: The Compute Library requires Mali OpenCL DDK r8p0 or higher (OpenCL kernels are compiled using the -cl-arm-non-uniform-work-group-size flag)

Note: All OpenCL functions and objects in the runtime library use the command queue associated with CLScheduler for all operations, a real implementation would be expected to use different queues for mapping operations and kernels in order to reach a better GPU utilization.

OpenCL Scheduler and kernel library

The Compute Library runtime uses a single command queue and context for all the operations.

The user can get / set this context and command queue through CLScheduler's interface.

The user can get / set the target GPU device through the CLScheduler's interface.

Attention: Make sure the application is using the same context as the library as in OpenCL it is forbidden to share objects across contexts. This is done by calling CLScheduler::init() or CLScheduler::default_init() at the beginning of your application.; Make sure the scheduler's target is not changed after function classes are created.

All OpenCL kernels used by the library are built and stored in CLKernelLibrary. If the library is compiled with embed_kernels=0 the application can set the path to the OpenCL kernels by calling CLKernelLibrary::init(), by default the path is set to "./cl_kernels"

OpenCL events and synchronization

In order to block until all the jobs in the CLScheduler's command queue are done executing the user can call CLScheduler::sync() or create a sync event using CLScheduler::enqueue_sync_event()

For example:

        PPMLoader     ppm;
        constexpr int scale_factor = 2;
        CLScheduler::get().default_init();
        if(argc < 2)
        {
            // Print help
            std::cout << "Usage: ./build/cl_events [input_image.ppm]\n\n";
            std::cout << "No input_image provided, creating a dummy 640x480 image\n";
            // Create an empty grayscale 640x480 image
            src.allocator()->init(TensorInfo(640, 480, Format::U8));
        }
        else
        {
            ppm.open(argv[1]);
            ppm.init_image(src, Format::U8);
        }
        TensorInfo dst_info(src.info()->dimension(0) / scale_factor, src.info()->dimension(1) / scale_factor, Format::U8);
        // Configure the temporary and destination images
        dst.allocator()->init(dst_info);
        tmp_scale_median.allocator()->init(dst_info);
        tmp_median_gauss.allocator()->init(dst_info);
        //Configure the functions:
        scale.configure(&src, &tmp_scale_median, InterpolationPolicy::NEAREST_NEIGHBOR, BorderMode::REPLICATE);
        median.configure(&tmp_scale_median, &tmp_median_gauss, BorderMode::REPLICATE);
        gauss.configure(&tmp_median_gauss, &dst, BorderMode::REPLICATE);
        // Allocate all the images
        src.allocator()->allocate();
        dst.allocator()->allocate();
        tmp_scale_median.allocator()->allocate();
        tmp_median_gauss.allocator()->allocate();
        // Fill the input image with the content of the PPM image if a filename was provided:
        if(ppm.is_open())
        {
            ppm.fill_image(src);
            output_filename = std::string(argv[1]) + "_out.ppm";
        }

OpenCL / NEON interoperability

You can mix OpenCL and NEON kernels and functions. However it is the user's responsibility to handle the mapping/unmapping of OpenCL objects, for example:

        PPMLoader ppm;
        CLScheduler::get().default_init();
        if(argc < 2)
        {
            // Print help
            std::cout << "Usage: ./build/cl_convolution [input_image.ppm]\n\n";
            std::cout << "No input_image provided, creating a dummy 640x480 image\n";
            // Create an empty grayscale 640x480 image
            src.allocator()->init(TensorInfo(640, 480, Format::U8));
        }
        else
        {
            ppm.open(argv[1]);
            ppm.init_image(src, Format::U8);
        }
        TensorInfo scale_median_info(TensorInfo(src.info()->dimension(0) / 2, src.info()->dimension(1) / 2, Format::U8));
        // Configure the temporary and destination images
        scale_median.allocator()->init(scale_median_info);
        median_gauss.allocator()->init(scale_median_info);
        dst.allocator()->init(scale_median_info);
        scale.configure(&src, &scale_median, InterpolationPolicy::NEAREST_NEIGHBOR, BorderMode::REPLICATE);
        median.configure(&scale_median, &median_gauss, BorderMode::REPLICATE);
        gauss.configure(&median_gauss, &dst, BorderMode::REPLICATE);
        // Allocate all the images
        src.allocator()->allocate();
        scale_median.allocator()->allocate();
        median_gauss.allocator()->allocate();
        dst.allocator()->allocate();
        // Fill the input image with the content of the PPM image if a filename was provided:
        if(ppm.is_open())
        {
            ppm.fill_image(src);
            const std::string output_filename = std::string(argv[1]) + "_out.ppm";
        }

See also: main_neoncl_scale_median_gaussian

Algorithms

All computer vision algorithms in this library have been implemented following the OpenVX 1.1 specifications. Please refer to the Khronos documentation for more information.

Images, padding, border modes and tensors

Most kernels and functions in the library process images, however, in order to be future proof most of the kernels actually accept tensors. See below for more information about how they are related.

Attention: Each memory object can be written by only one kernel, however it can be read by several kernels. Writing to the same object from several kernels will result in undefined behavior. The kernel writing to an object must be configured before the kernel(s) reading from it.

Padding and border modes

Several algorithms require a neighborhood around the current pixel to compute it's value. This means the algorithm will not be able to process the borders of the image unless you give it more information about how those border pixels should be processed. The BorderMode enum is used for this purpose.

You have 3 types of BorderMode :

BorderMode::UNDEFINED : Neighbor pixels outside of the image are treated as undefined. As a result all the pixels which are on the border will have a value which is undefined.
BorderMode::REPLICATE : Neighbor pixels outside of the image are treated as having the same value as the closest valid pixel.
BorderMode::CONSTANT : Neighbor pixels outside of the image are treated as having the same constant value. (The user can choose what this value should be).

Moreover both OpenCL and NEON use vector loads and stores instructions to access the data in buffers, so in order to avoid having special cases to handle for the borders all the images and tensors used in this library must be padded.

Padding

There are different ways padding can be calculated:

Accurate padding:

        PPMLoader ppm;
        if(argc < 2)
        {
            // Print help
            std::cout << "Usage: ./build/neon_convolution [input_image.ppm]\n\n";
            std::cout << "No input_image provided, creating a dummy 640x480 image\n";
            // Initialize just the dimensions and format of your buffers:
            src.allocator()->init(TensorInfo(640, 480, Format::U8));
        }
        else
        {
            ppm.open(argv[1]);
            // Initialize just the dimensions and format of your buffers:
            ppm.init_image(src, Format::U8);
        }
        // Initialize just the dimensions and format of the temporary and destination images:
        tmp.allocator()->init(*src.info());
        dst.allocator()->init(*src.info());
        // Apply a Gaussian 3x3 filter to the source image followed by a Gaussian 5x5:
        // The function will automatically update the padding information inside input and output to match its requirements
        conv3x3.configure(&src, &tmp, gaussian3x3, 0 /* Let arm_compute calculate the scale */, BorderMode::UNDEFINED);
        conv5x5.configure(&tmp, &dst, gaussian5x5, 0 /* Let arm_compute calculate the scale */, BorderMode::UNDEFINED);
        // Now that the padding requirements are known we can allocate the images:
        src.allocator()->allocate();
        tmp.allocator()->allocate();
        dst.allocator()->allocate();
        // Fill the input image with the content of the PPM image if a filename was provided:
        if(ppm.is_open())
        {
            ppm.fill_image(src);
            output_filename = std::string(argv[1]) + "_out.ppm";
        }

Note: It's important to call allocate after the function is configured: if the image / tensor is already allocated then the function will shrink its execution window instead of increasing the padding. (See below for more details).

Manual padding / no padding / auto padding: You can allocate your images / tensors up front (before configuring your functions). In that case the function will use whatever padding is available and will shrink its execution window if there isn't enough padding available (which translates into a smaller valid region for the output). See also Valid regions). If you don't want to manually set the padding but still want to allocate your objects upfront then you can use auto_padding. It guarantees that the allocation will have enough padding to run any of the provided functions.

Image     src, dst;
// Use auto padding for the input:
src.info()->init_auto_padding(TensorShape(640u,480u), Format::U8);
// Use manual padding for the destination image
dst.info()->init(src.info()->tensor_shape(), Format::U8, strides_in_bytes, offset_first_element_in_bytes, total_size_in_bytes);
// Allocate all the images
src.allocator()->allocate();
dst.allocator()->allocate();
// Fill the input image with the content of the PPM image if a filename was provided:
fill_image(src);
NEGaussian3x3 gauss;
// Apply a Gaussian 3x3 filter to the source image (Note: if the padding provided is not enough then the execution window and valid region of the output will be shrunk)
gauss.configure(&src, &dst, BorderMode::UNDEFINED);
//Execute the functions:
gauss.run();

Warning: Some kernels need up to 3 neighbor values to calculate the value of a given pixel. Therefore, to be safe, we use a 4-pixel padding all around the image. In addition, some kernels read and write up to 32 pixels at the same time. To cover that case as well we add an extra 32 pixels of padding at the end of each row. As a result auto padded buffers waste a lot of memory and are less cache friendly. It is therefore recommended to use accurate padding or manual padding wherever possible.

Valid regions

Some kernels (like edge detectors for example) need to read values of neighboring pixels to calculate the value of a given pixel, it is therefore not possible to calculate the values of the pixels on the edges.

Another case is: if a kernel processes 8 pixels per iteration and the image's dimensions are not a multiple of 8 and not enough padding is available then the kernel will not be able to process the pixels near the right edge. As a result these pixels will be left undefined.

In order to know which pixels have been calculated, each kernel sets a valid region for each output image or tensor. See also TensorInfo::valid_region(), ValidRegion

Tensors

Tensors are multi-dimensional arrays with a maximum of Coordinates::num_max_dimensions dimensions.

Depending on the number of dimensions tensors can be interpreted as various objects. A scalar can be represented as a zero-dimensional tensor and a vector of numbers can be represented as an one-dimensional tensor. Further, an image is actually just a 2D tensor, a 3D tensor can be seen as an array of images and a 4D tensor as a 2D array of images, etc.

Note: Most algorithms process images (i.e a 2D slice of the tensor), therefore only padding along the X and Y axes is required (2D slices can be stored contiguously in memory).

Images and Tensors description conventions

Image objects are defined by a Format and dimensions expressed as [width, height, batch]

Tensors are defined by a DataType plus a number of channels (Always expected to be 1 for now) and their dimensions are expressed as [width, height, feature_maps, batch].

In other words, the lower three dimensions of a tensor specify a single input in [width, height, feature_maps], while any other specified dimension represents a batch in the appropriate dimension space. For example, a tensor with dimensions [128, 128, 64, 16] represents a 1D batch space with 16 batches of 128 elements in width and height and 64 feature maps each. Each kernel specifies the expected layout of each of its tensors in its documentation.

Note: Unless specified otherwise in the kernel's or function's documentation all tensors and images parameters passed must have identical dimensions.; Unless specified otherwise in the kernel's or function's documentation the number of channels for tensors is expected to be 1 (For images, the number of channels is inferred from the Format).

Attention: Regardless of the DataType used by a tensor the ITensor::buffer() method will always return a uint8_t pointer, and all the metadata in TensorInfo will be expressed in bytes. It is the user's responsibility to cast the pointer to the correct type.

For example, to read the element located at the coordinates (x,y) of a float tensor:

float value = *reinterpret_cast<float*>(input.buffer() + input.info()->offset_element_in_bytes(Coordinates(x,y)));

Working with Images and Tensors using iterators

The library provides some iterators to access objects' data. Iterators are created by associating a data object (An image or a tensor for example) with an iteration window.

Iteration windows are defined by an array of dimensions, each of which consists of a start, end and step.

The execute_window_loop function takes an execution window, a lambda function and one or more iterators. It will iterate through every element of the execution window and for each element it will update the iterators accordingly and call the lambda function.

Here are a couple of examples of how to use the iterators to fill / read tensors:

        constexpr unsigned int width  = 4;
        constexpr unsigned int height = 3;
        constexpr unsigned int batch  = 2;
        src_data = new float[width * height * batch];
        dst_data = new float[width * height * batch];
        // Fill src_data with dummy values:
        for(unsigned int b = 0; b < batch; b++)
        {
            for(unsigned int h = 0; h < height; h++)
            {
                for(unsigned int w = 0; w < width; w++)
                {
                    src_data[b * (width * height) + h * width + w] = static_cast<float>(100 * b + 10 * h + w);
                }
            }
        }
        // Initialize the tensors dimensions and type:
        const TensorShape shape(width, height, batch);
        input.allocator()->init(TensorInfo(shape, 1, DataType::F32));
        output.allocator()->init(TensorInfo(shape, 1, DataType::F32));
        // Configure softmax:
        softmax.configure(&input, &output);
        // Allocate the input / output tensors:
        input.allocator()->allocate();
        output.allocator()->allocate();
        // Fill the input tensor:
        // Simplest way: create an iterator to iterate through each element of the input tensor:
        Window input_window;
        input_window.use_tensor_dimensions(input.info()->tensor_shape());
        std::cout << " Dimensions of the input's iterator:\n";
        std::cout << " X = [start=" << input_window.x().start() << ", end=" << input_window.x().end() << ", step=" << input_window.x().step() << "]\n";
        std::cout << " Y = [start=" << input_window.y().start() << ", end=" << input_window.y().end() << ", step=" << input_window.y().step() << "]\n";
        std::cout << " Z = [start=" << input_window.z().start() << ", end=" << input_window.z().end() << ", step=" << input_window.z().step() << "]\n";
        // Create an iterator:
        Iterator input_it(&input, input_window);
        // Iterate through the elements of src_data and copy them one by one to the input tensor:
        // This is equivalent to:
        // for( unsigned int z = 0; z < batch; ++z)
        // {
        //   for( unsigned int y = 0; y < height; ++y)
        //   {
        //     for( unsigned int x = 0; x < width; ++x)
        //     {
        //       *reinterpret_cast<float*>( input.buffer() + input.info()->offset_element_in_bytes(Coordinates(x,y,z))) = src_data[ z * (width*height) + y * width + x];
        //     }
        //   }
        // }
        // Except it works for an arbitrary number of dimensions
        execute_window_loop(input_window, [&](const Coordinates & id)
        {
            std::cout << "Setting item [" << id.x() << "," << id.y() << "," << id.z() << "]\n";
            *reinterpret_cast<float *>(input_it.ptr()) = src_data[id.z() * (width * height) + id.y() * width + id.x()];
        },
        input_it);
        // More efficient way: create an iterator to iterate through each row (instead of each element) of the output tensor:
        Window output_window;
        output_window.use_tensor_dimensions(output.info()->tensor_shape(), /* first_dimension =*/Window::DimY); // Iterate through the rows (not each element)
        std::cout << " Dimensions of the output's iterator:\n";
        std::cout << " X = [start=" << output_window.x().start() << ", end=" << output_window.x().end() << ", step=" << output_window.x().step() << "]\n";
        std::cout << " Y = [start=" << output_window.y().start() << ", end=" << output_window.y().end() << ", step=" << output_window.y().step() << "]\n";
        std::cout << " Z = [start=" << output_window.z().start() << ", end=" << output_window.z().end() << ", step=" << output_window.z().step() << "]\n";
        // Create an iterator:
        Iterator output_it(&output, output_window);
        // Iterate through the rows of the output tensor and copy them to dst_data:
        // This is equivalent to:
        // for( unsigned int z = 0; z < batch; ++z)
        // {
        //   for( unsigned int y = 0; y < height; ++y)
        //   {
        //     memcpy( dst_data + z * (width*height) + y * width, input.buffer() + input.info()->offset_element_in_bytes(Coordinates(0,y,z)), width * sizeof(float));
        //   }
        // }
        // Except it works for an arbitrary number of dimensions
        execute_window_loop(output_window, [&](const Coordinates & id)
        {
            std::cout << "Copying one row starting from [" << id.x() << "," << id.y() << "," << id.z() << "]\n";
            // Copy one whole row:
            memcpy(dst_data + id.z() * (width * height) + id.y() * width, output_it.ptr(), width * sizeof(float));
        },
        output_it);

MemoryManager

IMemoryManager is a memory managing interface that can be used to reduce the memory requirements of a given pipeline by recycling temporary buffers.

MemoryGroup, MemoryPool and MemoryManager Components

MemoryGroup

IMemoryGroup defines the memory managing granularity.

MemoryGroup binds a number of objects to a bucket of memory requirements that need to be fulfilled in order for an operation or list of operations to be executed.

Requesting backing memory for a specific group can be done using IMemoryGroup::acquire and releasing the memory back using IMemoryGroup::release.

Note

Two types of memory groups are currently implemented:

MemoryGroup that manages Tensor objects
CLMemoryGroup that manages CLTensor objects.

MemoryPool

IMemoryPool defines a pool of memory that can be used to provide backing memory to a memory group.

Note: BlobMemoryPool is currently implemented which models the memory requirements as a vector of distinct memory blobs.

MemoryManager Components

IMemoryManager consists of two components:

ILifetimeManager that keeps track of the lifetime of the registered objects of the memory groups and given an IAllocator creates an appropriate memory pool that fulfils the memory requirements of all the registered memory groups.
IPoolManager that safely manages the registered memory pools.

Note: IMemoryManager::finalize should be called once the configuration of all the memory groups, kernels and functions is done, so that the memory manager can allocate the appropriate backing memory.; BlobLifetimeManager is currently implemented which models the memory requirements as a vector of distinct memory blobs.

Working with the Memory Manager

Using a memory manager to reduce the memory requirements of a pipeline can be summed in the following steps:

Initially a memory manager must be set-up:

Allocator  allocator{};                                                               // Create an allocator to use for the backing memory allocation
auto lifetime_mgr  = std::make_shared<BlobLifetimeManager>();                         // Create Lifetime Manager
auto pool_mgr      = std::make_shared<PoolManager>();                                 // Create Pool Manager
auto mm            = std::make_shared<MemoryManagerOnDemand>(lifetime_mgr, pool_mgr); // Create Memory Manager

Once done, memory groups can be registered to use the memory manager:

MemoryGroup memory_group(mm); // Create a memory group and set the memory manager to use

Note: If a memory manager is not specified then all allocation will be immediate instead of deferred through the memory manager.

Next step is to set objects to be managed by the memory group. It is important though to note that the lifetime of an object is tracked from the MemoryGroup::manage() and the TensorAllocator::allocate calls. MemoryGroup::manage flags that the object will be needed starting now and when TensorAllocator::allocate is called it signals the end of the object lifetime.

Tensor tmp1, tmp2, tmp3;            // Create example tensors
memory_group.manage(&tmp1);         // Start managing object tmp1 and start its lifetime
memory_group.manage(&tmp2);         // Start managing object tmp2 and start its lifetime
operation1.configure(&tmp1, &tmp2); // Configure a function/kernel using tmp1 and tmp2
tmp1.allocator()->allocate();       // Flag that the lifetime of object tmp1 has ended
memory_group.manage(&tmp3);         // Start managing object tmp3 and start its lifetime
operation2.configure(&tmp2, &tmp3); // Configure a function/kernel using tmp2 and tmp3
tmp2.allocator()->allocate();       // Flag that the lifetime of object tmp2 has ended
tmp3.allocator()->allocate();       // Flag that the lifetime of object tmp3 has ended

Warning: The configuration step should be done sequentially by a single thread so that all the lifetimes are captured correclty.

When configuration of all the operations is finished then the memory manager have to be finalized:

mm->set_allocator(&allocator); // Set allocator to use
mm->set_set_num_pools(2);      // Set number of pools to create in case parallel operations can be run
mm->finalize();                // Finalize memory manager (Object lifetime check, Memory pool creation etc)

Finally, during execution of the pipeline the memory of the appropriate memory group should be requested before running:

memory_group.acquire(); // Request memory for the group
operation1.run();       // Run operation1
operation2.run();       // Run operation2
memory_group.release(); // Release memory so that it can be reused

Note: Execution of a pipeline can be done in a multi-threading environment as memory acquisition/release are thread safe.

Function support

Most of the library's function have been ported to use IMemoryManager for their internal temporary buffers.

If that is the case, a memory manager can be passed to them during construction to reuse memory among these functions.

// Setup Memory Manager
CLBufferAllocator  allocator{};                                                       // Create an allocator to use for the backing memory allocation
auto lifetime_mgr  = std::make_shared<BlobLifetimeManager>();                         // Create Lifetime Manager
auto pool_mgr      = std::make_shared<PoolManager>();                                 // Create Pool Manager
auto mm            = std::make_shared<MemoryManagerOnDemand>(lifetime_mgr, pool_mgr); // Create Memory Manager
// Create two convolution layers and use the memory manager to manager their internal temporary buffers
CLConvolutionLayer conv1(mm), conv2(mm);
// Configure layers
conv1.configure(...);
conv2.configure(...);
// Finalize memory manager
mm->set_allocator(&allocator); // Set allocator to use
mm->set_set_num_pools(1);      // Set number of pools to create in case parallel operations can be run
mm->finalize();                // Finalize memory manager (Object lifetime check, Memory pool creation etc)
// Run layers (Memory will be recycled for internal buffers for conv1 and conv2
conv1.run();
conv2.run();

OpenCL Tuner

OpenCL kernels when dispatched to the GPU take two arguments:

The Global Workgroup Size (GWS): That's the number of times to run an OpenCL kernel to process all the elements we want to process.
The Local Workgroup Size (LWS): That's the number of elements we want to run in parallel on a GPU core at a given point in time.

The LWS can be required by an algorithm (For example if it contains memory barriers or uses local memory) but it can also be used for performance reasons to tweak the performance of a kernel: the execution time of the overall kernel might vary significantly depending on how the GWS is broken down.

However, there is no universal rule regarding which LWS is best for a given kernel, so instead we created the CLTuner.

When the CLTuner is enabled ( Target = 2 for the graph examples), the first time an OpenCL kernel is executed the Compute Library will try to run it with a variety of LWS values and will remember which one performed best for subsequent runs. At the end of the run the graph::Graph will try to save these tuning parameters to a file.

However this process takes quite a lot of time, which is why it cannot be enabled all the time.

But, when the CLTuner is disabled ( Target = 1 for the graph examples), the graph::Graph will try to reload the file containing the tuning parameters, then for each executed kernel the Compute Library will use the fine tuned LWS if it was present in the file or use a default LWS value if it's not.

Table of Contents