path: root/docs/project/19_NN_Compiler_and_Runtime_STD.rst

=======================
SW System Test Document
=======================

.. contents::

.. section-numbering::

.. raw:: pdf

   PageBreak oneColumn

**Revision history**

+-------+-------------+----------------------------+--------------------------+---------------------+
| Ver.  | Date        | Contents                   | Author                   | Approver            |
+=======+=============+============================+==========================+=====================+
| 0.1   | 2019.04.22  | Initial version            | Rusyaev Roman            | Sung-Jae Lee        |
+-------+-------------+----------------------------+--------------------------+---------------------+
| 0.2   | 2019.04.23  | SE member in-charge review | Aleksei Kondrashov       |                     |
+-------+-------------+----------------------------+--------------------------+---------------------+
| 1.0   | 2019.04.24  | Final Execution DR version | Rusyaev Roman            | Sung-Jae Lee        |
+-------+-------------+----------------------------+--------------------------+---------------------+
| 1.1   | 2019.07.24  | DR1 version                | Efimov Alexander         | Sung-Jae Lee        |
+-------+-------------+----------------------------+--------------------------+---------------------+

.. include:: project_terms_and_abbreviation.rst

|


SW System Test Overview
=======================

Purpose
-------

Software testing is an investigation to provide the quality of the product under test and to reduce risk of its failure
to users or customers. Purpose of testing is to detect software failures so that defects may be discovered and corrected.

Software system test procedure is a collection of processes and methods used to ensure quality. An additional goal is to make sure that the product
follows regulations and meets the quality standards expected by the customer.

SW System Test Items
====================

Functions to be tested
----------------------

**Table 2-1. Test Item**

.. list-table::
   :widths: 20 10 50
   :header-rows: 1

   * - Feature
     - Test Item ID
     - Test Item description

   * - RF-1
     - TST-1
     - Test suite checks NN ops import from Tensorflow format by loading NN model.

   * - RF-2
     - TST-2
     - Test suite checks NN ops import from ONNX format by loading NN model.

   * - RF-3
     - TST-3
     - The test suite should automatically verify the completeness of information that was read from the raw data by comparing it with serialized raw data from Model IR

   * - RF-4
     - TST-4
     - The test should verify successful loading the Inception V3 NN model

   * - RF-5
     - TST-5
     - The test should verify successful loading the MobileNet NN model

   * - RF-6
     - TST-6
     - Test checks that NN Runtime can execute test model and save file with appropriate performance profile

   * - RF-6
     - TST-7
     - Test checks that NN Runtime can import performance data from file with saved performance profile

   * - RF-7
     - TST-8
     - Test checks that NN Runtime uses profile information to choose appropriate backends for operations

   * - RF-8
     - TST-9
     - Test checks that NN Runtime mixes different backends in one model if this helps to minimize expected execution time

   * - RNF-2
     - TST-10
     - The test suite is to verify the correctness of calculations by comparing the result of original NN model and the result of compiled artifact/inference of NN runtime on the same input data

   * - RNF-3, RNF-4
     - TST-11
     - A Linux-based OS should be used while the test environment are built.

   * - RNF-1
     - TST-12
     - The test should verify Inference time should be equal or less then models executed by ML frameworks of original model


Functions not to be tested
--------------------------

The following requirements cannot be tested:

* The source code requirements (RNF-5. RNF-6, RNF-7)


SW System Test Procedure
========================

Test Pass/Fail Criteria
-----------------------

All tests must be executed without any issues at any time for newly implemented, refactored, or changed code.


Regression Test strategy
------------------------

If a new issue is detected and it is not covered by an existing test then a new test will be developed. In other case the issue should be resolved.

Test tools
----------

**Table 3-1. Test Tools**

.. list-table::
   :widths: 20 30
   :stub-columns: 1

   * - Source code static verification
     - SVACE

   * - Source code dynamic analysis
     - Valgrind/AdressSanitizer/LeakSanitizer

   * - Test execution
     - CMake

   * - Defect management
     - Samsung Research GitHub

   * - Continuous Integration system
     - HQ CI (CODE)


SW System Test Schedule Plan
============================

Test task & schedule
--------------------

**Table 4-1. Test Tasks and Schedule**

.. list-table::
   :widths: 20 20 20 30
   :stub-columns: 1

   * - Task
     - Schedule
     - Responsibility
     - Detailed Task

   * - Unit testing
     - 01.05.2019 - 31.12.2019
     - All
     - All unit tests should be carried out

   * - System testing
     - 01.05.2019 - 31.12.2019
     - All
     - All system tests should be carried out


Test Resource organization plan
-------------------------------

Test environment
````````````````

**Table 4-2. Hardware / Operating System**

.. list-table::
   :widths: 10 15 30
   :header-rows: 1

   * - Type/Model
     - Operating System
     - Usage

   * - PC/x86
     - Ubuntu GNU/Linux version >=14.04
     - Build system with unit tests. System tests are performed too.

   * - Odroid XU4
     - Linux based OS (including Tizen)
     - Unit and system testing

**Table 4-3. Software**

.. list-table::
   :widths: 10 15 30
   :header-rows: 1

   * - Type
     - Spec
     - Usage

   * - Library
     - Google test
     - Organize test code and provide utility methods

   * - Source Code Management (SCM)
     - Samsung GitHub
     - The source code version controlling system

   * - CI
     - Jenkins
     - The HQ CI system

   * - Build system
     - CMake
     - Run test and check status

   * - Management tool
     - Samsung Github
     - Source code version control, code review, issue tracker

Risk management plan
--------------------

**Table 4-4. Risk Management**

.. list-table::
   :widths: 20 35 11 19
   :header-rows: 1

   * - Risk
     - Description
     - Probability
     - Countermeasures

   * - HQ CI does not support target testing
     - Some tests required the target devices to be run on it. The provided CI system may not support such type of testing.
     - High
     - Set CI environment on site

   * - Targets for testing/development are not available
     - Full automatic testing may take a long time. It also required target devices to execute the binaries.
     - Medium
     - Request/Buy enough amount of devices

SW configuration management plan
--------------------------------

SW Configuration items identification
`````````````````````````````````````

Baseline
````````

**Table 4-5. Baselines**

.. list-table::
   :widths: 15 40 40 15
   :header-rows: 1

   * - Test Round
     - Baseline Name
     - Configuration Item
     - Schedule

   * - Round 1
     - The nncc v1.3
     - SRR-RAJ0119ZZ-BERF-STD001, SRR-RAJ0119ZZ-BERF-UTR001
     - 09.2019

   * - Round 2
     - The nncc v1.7
     - SRR-RAJ0119ZZ-BERF-STD002, SRR-RAJ0119ZZ-BERF-UTR002
     - 11.2019

   * - Round 3
     - The nncc v2.0
     - SRR-RAJ0119ZZ-BERF-STD003, SRR-RAJ0119ZZ-BERF-UTR003
     - 12.2019



SW System Test Case
===================


**Table 5-1. System Test case**

.. list-table::
   :widths: 10 12 50 20
   :header-rows: 1

   * - TestItem ID
     - Testcase ID
     - Test Procedures
     - Expected Results

   * - TST-1, TST-4
     - T1
     - Import TensorFlow Inception V3 NN model
     - During import no crashes or error messages occurred

   * - TST-1, TST-5
     - T2
     - Import TensorFlow MobileNet NN model
     - During import no crashes or error messages occurred

   * - TST-2, TST-4
     - T3
     - Import ONNX Inception V3 NN model
     - During import no crashes or error messages occurred

   * - TST-2, TST-5
     - T4
     - Import ONNX MobileNet NN model
     - During import no crashes or error messages occurred

   * - TST-3, TST-4
     - T5
     - Import Inception V3 NN model, serialize all model weights, compare serialized data with the initial NN model
     - Test executed successfully, serialized weights are equal to initial model weights

   * - TST-3, TST-5
     - T6
     - Import MobileNet NN model, serialize all model weights, compare serialized data with the initial NN model
     - Test executed successfully, serialized weights are equal to initial model weights

   * - TST-10, TST-4
     - T7
     - Compare the result from original Inception V3 model with the result from compiled artifact/runtime inference
     - Test executed successfully, results are comparable

   * - TST-10, TST-5
     - T8
     - Compare the result from original MobileNet model with the result from compiled artifact/runtime inference
     - Test executed successfully, results are comparable

   * - TST-11
     - T9
     - Check that the OS used during test environment build is Linux-based
     - Test executed successfully

   * - TST-12
     - T10
     - The test should verify Inference time should be equal or less then models executed by ML frameworks of original model
     - Test executed successfully

   * - TST-6
     - T11
     - The test should verify NN Runtime with enabled profiling generates file with profile data
     - Test executed successfully

   * - TST-7, TST-8
     - T12
     - The test should verify NN Runtime assigns one-op networks to different backends and choise is depends on provided specially prepared profile file and properties of operations (for example size of arguemnt tensors)
     - Test executes successfully

   * - TST-7, TST-9
     - T13
     - The test should verify if profile data is provided NN Runtime assigns operations in network with several operations to different backends if expected execution time is less then exepcted time of one backend assignment method
     - Tesy executes successfully

SW System Test Result
=====================

Test Schedule
-------------

**Table 6-1. Test Period**

.. list-table::
   :widths: 40 20 20
   :header-rows: 1

   * -
     - Plan
     - Actual

   * - Entire test period
     - 04/2019 - 12/2019
     -

   * - Reason for change period
     -
     -

|

**Table 6-2. Test plan vs Actual**

+------------+---------------------+--------------------+-----------------+
| Test round |         Plan        |     Actual         | Remarks         |
|            +-----------+---------+-----------+--------+                 |
|            | Schedule  |   MM    | Schedule  |  MM    |                 |
+============+===========+=========+===========+========+=================+
| R.1        |  09/2019  |   1     |           |        |                 |
+------------+-----------+---------+-----------+--------+-----------------+
| R.2        |  11/2019  |   1     |           |        |                 |
+------------+-----------+---------+-----------+--------+-----------------+
| R.3        |  12/2019  |   1     |           |        |                 |
+------------+-----------+---------+-----------+--------+-----------------+