diff options
Diffstat (limited to 'compiler/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst')
| -rw-r--r-- | compiler/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst | 1696 |
1 files changed, 0 insertions, 1696 deletions
diff --git a/compiler/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst b/compiler/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst deleted file mode 100644 index 97aaa711e..000000000 --- a/compiler/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst +++ /dev/null @@ -1,1696 +0,0 @@ -======================= -SW System Test Document -======================= - - -.. contents:: - -.. section-numbering:: - -.. raw:: pdf - - PageBreak oneColumn - -**Revision history** - -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| Ver. | Date | Contents | Author | Approver | -+=======+=============+==========================================+==========================+=====================+ -| 0.1 | 2018.04.12 | Initial version | Vostokov Sergey | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 0.2 | 2018.04.13 | SE member in-charge review | Aleksei Kondrashov | | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 1.0 | 2018.04.17 | Final Execution DR version | Vostokov Sergey | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 1.1 | 2018.06.20 | DR1 version (Test Round 1) | Vostokov Sergey | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 2.0 | 2018.09.24 | DR2 version (Test Round 2) | Vostokov Sergey | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 2.1 | 2018.10.23 | NN Importer Tech Item (Test Round 2.1) | Roman Rusyaev | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 3.0 | 2018.11.30 | Code Emitter Tech Item (Test Round 3) | Roman Rusyaev | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 3.1 | 2018.12.07 | Minor updates for Completion DR | Roman Rusyaev | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 3.2 | 2018.12.07 | SE member in-charge review | Aleksei Kondrashov | | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ -| 3.3 | 2018.12.07 | Final Completion DR version | Roman Rusyaev | Sung-Jae Lee | -+-------+-------------+------------------------------------------+--------------------------+---------------------+ - -| - -**Terminology and Abbreviation** - -.. list-table:: - :widths: 10 30 - :header-rows: 0 - - * - OS - - Operating System - * - OS API - - Application interface of OS - * - HW - - Hardware - * - SW - - Software - * - NN - - Neural Network - * - NN model - - Neural network model (Instance of NN built with ML framework) - * - NN compiler - - The compiler for neural network - * - ML framework - - The machine learning framework - * - TF/TF Lite - - Tensorflow/Tensorflow Lite ML framework - * - IR - - Intermediate representation - * - CI/CI system - - Continuous integration system - * - UI - - The user interface - * - GUI - - The graphical user interface - * - CLI - - The command-line interface - * - CG - - Computational Graph - - -| - -**References** - -[1] Vostokov Sergey, `SW Requirements Specification <requirements_specification.md>`_ - - -SW System Test Overview -======================= - -Purpose -------- - -Software testing is an investigation to provide the quality of the product under test and to reduce risk of its failure -to users or customers. Purpose of testing is to detect software failures so that defects may be discovered and corrected. - -Software system test procedure is a collection of processes and methods used to ensure quality. An additional goal is to make sure that the product -follows regulations and meets the quality standards expected by the customer. - -Scope ------ - -As the number of possible tests for every software is practically infinite, we use some strategy to select tests that -are feasible for the available time and resources. - -Software system tests attempt to cover requirements listed in the `SW Requirement Specification <https://github.sec.samsung.net/STAR/nncc/doc/project/requirements_specification.md>`_. - -Since the project outcome is a compiler then its testing are in different domain than many other kinds of application or -system testing. They are dedicated to find all possible issues that cause the following bugs: - -- Compiler crashes (also known as an ICE or Internal Compiler Error) -- Compiler hangs (kind of infinite loop in the compiler) -- Bad code generation (a result of incorrect compiler output): - - - Bad code generation that leads to a crash in the application - - “Silent” bad code generation - -- Compiler throughput issues (Issues that affect the amount of time the compiler takes to compile code ) -- Code quality issues (Issues that affect the performance of the compiled application) -- Compiler feature correctness issues (This class of bugs involves the compiler generating correct code, but not doing what a particular feature specifies should be done) - - -SW System Test Items -==================== - -Functions to be tested ----------------------- - -**Table 2-1. Test Item** - -.. list-table:: - :widths: 20 10 50 - :header-rows: 1 - - * - Feature - - Test Item ID - - Test Item description - - * - RF-1, RIF-3 - RIF-7 - - TST-1 - - Test suite checks NN ops import from Tensorflow Lite format by loading NN model that consists of a single NN op. One test for each NN op. - - * - RF-2, RIF-3 - RIF-7 - - TST-2 - - Test suite checks NN ops import from Caffe format by loading NN model that consists of a single NN op. One test for each NN op. - - * - RF-3, RIF-3 - RIF-7 - - TST-3 - - Test suite checks NN ops import from Caffe2 format by loading NN model that consists of a single NN op. One test for each NN op. - - * - RF-5, RIF-3 - RIF-7 - - TST-4 - - The test should verify successful loading the Inception V3 NN model - - * - RF-6, RIF-3 - RIF-7 - - TST-5 - - The test should verify successful loading the MobileNet NN model - - * - RF-4 - - TST-6 - - The test suite should automatically verify the completeness of information that was read from the raw data by comparing it with serialized raw data from Model IR - - * - RF-7, RF-18, RIF-13 - - TST-7 - - The test should automatically verify successful execution of binary on ARM CPU - - * - RF-8, RF-17, RIF-14, RIF-15 - - TST-8 - - The unit test should automatically verify successful execution of calculation on GPU - - * - RF-9, RNF-1, RIF-17, RIF-18 - - TST-9 - - Unit test should verify the existence and format of binary (shared or static) in accordance to specified options - - * - RF-10 - - TST-10 - - Unit test should verify that compiler produces a compiled artefact for the Inception V3 NN model (Validity of compiled artefact is checked by other tests) - - * - RF-11 - - TST-11 - - Unit test should verify that compiler produces a compiled artefact for the MobileNet NN model (Validity of compiled artefact is checked by other tests) - - * - RF-12, RF-13, RF-14, RNF-6, RNF-7, RNF-8 - - TST-12 - - The test suite should verify correctness of configuration object by unit testing - - * - RF-15, RNF-1 - - TST-13 - - The test suite is to verify the correctness of calculations by comparing the result of original NN model and the result of compiled artefact on the same input data - - * - RF-16 - - TST-14 - - Unit test should verify that the incorrect input data is processed with an error message without unexpected termination of the application - - * - RNF-4, RNF-5, RIF-8 - - TST-15 - - A Linux-based OS should be used while the test environment are built. - - * - RIF-16 - - TST-16 - - The unit test should verify the existence and validity of generated C/C++ header for compiled artefact - - -**The following requirements can be tested only manually:** - -* Non-functional requirements: RNF-2, RNF-3 (They would be tested during development) -* Interface requirements: RIF-1, RIF-2, RIF-9 - RIF-12, RIF-19 - - -Functions not to be tested --------------------------- - -The following requirements cannot be tested: - -* The source code requirements (RNF-9. RNF-10. RNF-11) - - -SW System Test Procedure -======================== - -Test approaches ---------------- - -While implementation of the project deliverables several kinds of testing are used. All of them are performed automatically by continuous integration system since it is developed. CI system subscribes on source code modification in the version control system. The configuration does not allow any changes to be merged into the main line if these changes do not pass merge mandatory tests. - -- **Code style check** (Merge mandatory test): to verify consistency of coding style - -- **Build test** (Merge mandatory test): to verify the current build - -- **Unit tests**: to verify SW system consistency. All new implemented features, code refactoring, optimizations must not cause unit test failure. Each unit test reflect the exact logic of testing component, thus, it should be adopted any time when program logic changes. - -- **System tests**: to verify the feature quality as well as compliance with its specified requirements. - -- **Manual-based UI testing approach**: for interface requirements, which cannot be automated - - -Test Pass/Fail Criteria ------------------------ - -All tests (unit/system) must be executed without any issues at any time for newly implemented, refactored, or changed code. - -Test Start/Suspension/Resumption criteria ------------------------------------------ - -Two mandatory tests (code style check and build test) are performed for every pool request (PR) before it is merged. The configuration of continuous integration system (CI) does not allow to merge the changes into devel branch if they does not pass the tests. - -Unit and feature testing are performed for the devel branch automatically. The merge to master branch (release) are possible when all these tests passed. - -Regression Test strategy ------------------------- - -If a new issue is detected and it is not covered by an existing -test then a new test will be developed. In other case the issue should be resolved. - -Test tools ----------- - -**Table 3-1. Test Tools** - -.. list-table:: - :widths: 20 30 - :stub-columns: 1 - - * - Source code static verification - - SVACE - - * - Source code dynamic analysis - - Valgrind/AdressSanitizer/LeakSanitizer/MemorySanitizer - - * - Test execution - - CMake - - * - Defect management - - Samsung Research GitHub - - * - Continuous Integration system - - HQ CI (CODE) - - -SW System Test Schedule Plan -============================ - -Test task & schedule --------------------- - -**Table 4-1. Test Tasks and Schedule** - -.. list-table:: - :widths: 20 20 20 30 - :stub-columns: 1 - - * - Task - - Schedule - - Responsibility - - Detailed Task - - * - Unit testing - - 01.04.2018 - 31.12.2018 - - All - - All unit tests should be carried out - - * - System testing - - 01.04.2018 - 31.12.2018 - - All - - All system tests should be carried out - - -Test Resource organization plan -------------------------------- - -Test environment -```````````````` - -**Table 4-2. Hardware / Operating System** - -.. list-table:: - :widths: 10 15 30 - :header-rows: 1 - - * - Type/Model - - Operating System - - Usage - - * - PC/x86 - - Ubuntu GNU/Linux version >=14.04 - - Build system with unit tests. System tests are performed too. - - * - Tizen TM2 - - Tizen - - Unit and system testing - - * - Odroid XU4 - - Tizen - - Unit and system testing - -**Table 4-3. Software** - -.. list-table:: - :widths: 10 15 30 - :header-rows: 1 - - * - Type - - Spec - - Usage - - * - Library - - Google test - - Organize test code and provide utility methods - - * - VCS - - Samsung github - - The source code version controlling system - - * - CI - - CODE - - The HQ CI system - - * - Build system - - CMake - - Run test and check status - - * - Device connectivity - - sdb - - Send tools to the device and provide shell to run it - - * - Management tool - - The CODE (Collaborative Open Development Environment) - - Source code version control, code review, issue tracker, Continuous Integration - -Risk management plan --------------------- - -**Table 4-5. Risk Management** - -.. list-table:: - :widths: 20 35 11 19 - :header-rows: 1 - - * - Risk - - Description - - Probability - - Countermeasures - - * - SmartMachine OS SDK toolchain is not available - - In order to support compilation for SmartMachine OS the SDK is required. The compiler would have dependency of a SmartMachine OS SDK toolchain. - - High - - Suspend support of SmartMachine OS, and make plans when SmartMachine OS SDK is released - - * - SmartMachine OS targets are not available - - To perform testing of executables for SmartMachine OS the specified targets are required. - - High - - Request targets or SW emulator when SmartMachine OS is released - - * - HQ CI does not support target testing - - Some tests required the target devices to be run on it. The provided CI system may not support such type of testing. - - High - - Set CI environment on site - - * - Targets for testing/development are not available - - Full automatic testing may take a long time. It also required target devices to execute the binaries. - - Medium - - Request/Buy enough amount of devices - -SW configuration management plan --------------------------------- - -SW Configuration items identification -````````````````````````````````````` - -**Table 4-6. SW Configuration Items List** - -.. list-table:: - :widths: 5 30 25 30 - :header-rows: 1 - - * - No - - Document number - - SW configuration Item - - File name - - * - 1 - - SRR-RAJ0118ZZ-BWRF-STD001 - - System Test Document (test cases design, round 1) - - 18 NN compiler and Optimizer (STD) v1.0.pdf - - * - 2 - - SRR-RAJ0118ZZ-BWRF-STD002 - - System Test Document (Round 2, 2.1) - - 18 NN compiler and Optimizer (STD) v2.0.pdf - - * - 3 - - SRR-RAJ0118ZZ-BWRF-UTR001 - - Unit Test Report (Round 1) - - 18 NN compiler and Optimizer (UTR) v1.0.pdf - - * - 4 - - SRR-RAJ0118ZZ-BWRF-STD003 - - System Test Document (Round 3) - - 18 NN compiler and Optimizer (STD) v3.0.pdf - - * - 5 - - SRR-RAJ0118ZZ-BWRF-UTR002 - - Unit Test Report (Round 2) - - 18 NN compiler and Optimizer (UTR) v2.0.pdf - - * - 6 - - SRR-RAJ0118ZZ-BWRF-UTR003 - - Unit Test Report (Round 3) - - 18 NN compiler and Optimizer (UTR) v3.0.pdf - - - -Directory Structure -``````````````````` - -**Table 4-7. Directory Structure** - -.. list-table:: - :widths: 20 60 - :header-rows: 1 - - * - Directory - - Description - - * - / - - source codes of the build system, main README file - - * - /contrib - - Incubating projects - - * - /contrib/nnc - - Contains the source code of nnc (NN Compiler) - - * - /doc - - Contains the documentation of the project - - * - /doc/project - - Contains project management documents (SRS, SDD, STD, HLD, DLD, etc) - - * - /libs - - Contains the source of the libraries which are used by the nncc - - * - /libs/core - - Contains the source code of the core library of nncc - - * - /libs/frontend - - Contains the source code of supported frontend's plugins - - * - /libs/frontend/caffe - - The source code for the Caffe frontend - - * - /libs/frontend/caffe2 - - The source code for the Caffe2 frontend - - * - /libs/frontend/tflite - - The source code for the Tensorflow Lite frontend - - * - /libs/backend - - Contains the source code of supported backend plugins - - * - /libs/backend/cpu - - Contains the source code of CPU backend - - * - /libs/backend/gpu - - Contains the source code of GPU backend - - * - /libs/backend/3rd_party - - Contains the source code of backend to utilize 3rd party libraries - - * - /scripts - - Various scripts for building and testing the nncc - - * - /tools - - The source code of the executables - - -Baseline -```````` - -**Table 4-8. Baselines** - -.. list-table:: - :widths: 15 40 40 15 - :header-rows: 1 - - * - Test Round - - Baseline Name - - Configuration Item - - Schedule - - * - Round 1 - - The nncc v0.3 (Commit: **f1d2cca0fa9d22292e47adc82b10d6bcdd9c2097**) - - SRR-RAJ0118ZZ-BWRF-STD001, SRR-RAJ0118ZZ-BWRF-UTR001 - - 09.2018 - - * - Round 2 - - The nncc v0.7 (Commit: **84e16c9d06903ab9fb2094ea40914942e430d20a**) - - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 - - 11.2018 - - * - Round 2.1 - - The nncc v0.9 (Commit: **2d7efd2d0bc8a1ba135cb4996902284078e14eff**) - - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 - - 11.2018 - - * - Round 3 - - The nncc v1.0 (Commit: **85708532881069c6c7b2d26cd1537bf8685a81a7**) - - SRR-RAJ0118ZZ-BWRF-STD003, SRR-RAJ0118ZZ-BWRF-UTR003 - - 12.2018 - - - -SW System Test Case -=================== - - -The following tests are not supported because they are unavailable in ML Frameworks - -**Table 5-1. Unsupported Tests** - -+--------------+----------------------------------------------------+ -| ML Framework | Unsupported Tests | -+==============+====================================================+ -| TFLite | TST-1-9, TST-1-14, TST-1-15, TST-1-23, TST-1-26, | -| | TST-1-27, TST-1-28, TST-1-29, TST-1-30, TST-1-31, | -| | TST-1-32, TST-1-34, TST-1-43, TST-1-45, TST-1-51 | -+--------------+----------------------------------------------------+ -| Caffe | TST-2-15 | -+--------------+----------------------------------------------------+ - -**Table 5-2. System Test case** - -.. list-table:: - :widths: 10 12 50 20 - :header-rows: 1 - - * - TestItem ID - - Testcase ID - - Test Procedures - - Expected Results - - * - TST-1 - - TST-1-1 - - Import a NN consisting of a single Tensorflow Lite ADD operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-2 - - Import a NN consisting of a single Tensorflow Lite AVERAGE_POOL_2D operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-3 - - Import a NN consisting of a single Tensorflow Lite CONCATENATION operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-4 - - Import a NN consisting of a single Tensorflow Lite CONV_2D operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-5 - - Import a NN consisting of a single Tensorflow Lite DEPTHWISE_CONV_2D operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-6 - - Import a NN consisting of a single Tensorflow Lite DEQUANTIZE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-7 - - Import a NN consisting of a single Tensorflow Lite EMBEDDING_LOOKUP operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-8 - - Import a NN consisting of a single Tensorflow Lite FULLY_CONNECTED operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-9 - - Import a NN consisting of a single Tensorflow Lite HASHTABLE_LOOKUP operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-10 - - Import a NN consisting of a single Tensorflow Lite L2_NORMALIZATION operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-11 - - Import a NN consisting of a single Tensorflow Lite L2_POOL_2D operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-12 - - Import a NN consisting of a single Tensorflow Lite LOCAL_RESPONSE_NORMALIZATION operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-13 - - Import a NN consisting of a single Tensorflow Lite LOGISTIC operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-14 - - Import a NN consisting of a single Tensorflow Lite LSH_PROJECTION operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-15 - - Import a NN consisting of a single Tensorflow Lite LSTM operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-16 - - Import a NN consisting of a single Tensorflow Lite MAX_POOL_2D operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-17 - - Import a NN consisting of a single Tensorflow Lite MUL operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-18 - - Import a NN consisting of a single Tensorflow Lite RELU operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-19 - - Import a NN consisting of a single Tensorflow Lite RELU_N1_TO_1 operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-20 - - Import a NN consisting of a single Tensorflow Lite RELU6 operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-21 - - Import a NN consisting of a single Tensorflow Lite RESHAPE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-22 - - Import a NN consisting of a single Tensorflow Lite RESIZE_BILINEAR operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-23 - - Import a NN consisting of a single Tensorflow Lite RNN operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-24 - - Import a NN consisting of a single Tensorflow Lite SOFTMAX operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-25 - - Import a NN consisting of a single Tensorflow Lite SPACE_TO_DEPTH operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-26 - - Import a NN consisting of a single Tensorflow Lite SVDF operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-27 - - Import a NN consisting of a single Tensorflow Lite TANH operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-28 - - Import a NN consisting of a single Tensorflow Lite CONCAT_EMBEDDINGS operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-29 - - Import a NN consisting of a single Tensorflow Lite SKIP_GRAM operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-30 - - Import a NN consisting of a single Tensorflow Lite CALL operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-31 - - Import a NN consisting of a single Tensorflow Lite CUSTOM operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-32 - - Import a NN consisting of a single Tensorflow Lite EMBEDDING_LOOKUP_SPARSE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-33 - - Import a NN consisting of a single Tensorflow Lite PAD operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-34 - - Import a NN consisting of a single Tensorflow Lite UNIDIRECTIONAL_SEQUENCE_RNN operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-35 - - Import a NN consisting of a single Tensorflow Lite GATHER operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-36 - - Import a NN consisting of a single Tensorflow Lite BATCH_TO_SPACE_ND operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-37 - - Import a NN consisting of a single Tensorflow Lite SPACE_TO_BATCH_ND operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-38 - - Import a NN consisting of a single Tensorflow Lite TRANSPOSE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-39 - - Import a NN consisting of a single Tensorflow Lite MEAN operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-40 - - Import a NN consisting of a single Tensorflow Lite SUB operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-41 - - Import a NN consisting of a single Tensorflow Lite DIV operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-42 - - Import a NN consisting of a single Tensorflow Lite SQUEEZE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-43 - - Import a NN consisting of a single Tensorflow Lite UNIDIRECTIONAL_SEQUENCE_LSTM operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-44 - - Import a NN consisting of a single Tensorflow Lite STRIDED_SLICE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-45 - - Import a NN consisting of a single Tensorflow Lite BIDIRECTIONAL_SEQUENCE_RNN operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-46 - - Import a NN consisting of a single Tensorflow Lite EXP operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-47 - - Import a NN consisting of a single Tensorflow Lite TOPK_V2 operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-48 - - Import a NN consisting of a single Tensorflow Lite SPLIT operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-49 - - Import a NN consisting of a single Tensorflow Lite LOG_SOFTMAX operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-50 - - Import a NN consisting of a single Tensorflow Lite DELEGATE operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-51 - - Import a NN consisting of a single Tensorflow Lite BIDIRECTIONAL_SEQUENCE_LSTM operation - - During import no crashes or error messages occurred - - * - TST-1 - - TST-1-52 - - Import a NN consisting of a single Tensorflow Lite CAST operation - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-1 - - Import a NN consisting of Caffe ImageData layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-2 - - Import a NN consisting of Caffe Data layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-3 - - Import a NN consisting of Caffe HDF5Input layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-4 - - Import a NN consisting of two Caffe layers - Input layer and HDF5Output layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-5 - - Import a NN consisting of Caffe Input layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-6 - - Import a NN consisting of Caffe WindowData layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-7 - - Import a NN consisting of Caffe MemoryData layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-8 - - Import a NN consisting of Caffe DummyData layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-9 - - Import a NN consisting of two Caffe layers - Input layer and Convolution layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-10 - - Import a NN consisting of two Caffe layers - Input layer and Pooling layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-11 - - Import a NN consisting of two Caffe layers - Input layer and SPP layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-12 - - Import a NN consisting of two Caffe layers - Input layer and Crop layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-13 - - Import a NN consisting of two Caffe layers - Input layer and Deconvolution layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-14 - - Import a NN consisting of two Caffe layers - Input layer and Im2Col layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-15 - - Import a NN consisting of two Caffe layers - Input layer and Recurrent layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-16 - - Import a NN consisting of two Caffe layers - Input layer and RNN layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-17 - - Import a NN consisting of two Caffe layers - Input layer and LSTM layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-18 - - Import a NN consisting of two Caffe layers - Input layer and InnerProduct layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-19 - - Import a NN consisting of two Caffe layers - Input layer and Dropout layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-20 - - Import a NN consisting of two Caffe layers - Input layer and Embed layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-21 - - Import a NN consisting of two Caffe layers - Input layer and LRN layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-22 - - Import a NN consisting of two Caffe layers - Input layer and MVN layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-23 - - Import a NN consisting of two Caffe layers - Input layer and BatchNorm layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-24 - - Import a NN consisting of two Caffe layers - Input layer and ReLU layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-25 - - Import a NN consisting of two Caffe layers - Input layer and PReLU layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-26 - - Import a NN consisting of two Caffe layers - Input layer and ELU layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-27 - - Import a NN consisting of two Caffe layers - Input layer and Sigmoid layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-28 - - Import a NN consisting of two Caffe layers - Input layer and TanH layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-29 - - Import a NN consisting of two Caffe layers - Input layer and AbsVal layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-30 - - Import a NN consisting of two Caffe layers - Input layer and Power layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-31 - - Import a NN consisting of two Caffe layers - Input layer and Exp layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-32 - - Import a NN consisting of two Caffe layers - Input layer and Log layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-33 - - Import a NN consisting of two Caffe layers - Input layer and BNLL layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-34 - - Import a NN consisting of two Caffe layers - Input layer and Threshold layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-35 - - Import a NN consisting of two Caffe layers - Input layer and Bias layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-36 - - Import a NN consisting of two Caffe layers - Input layer and Scale layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-37 - - Import a NN consisting of two Caffe layers - Input layer and Flatten layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-38 - - Import a NN consisting of two Caffe layers - Input layer and Reshape layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-39 - - Import a NN consisting of two Caffe layers - Input layer and BatchReindex layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-40 - - Import a NN consisting of two Caffe layers - Input layer and Split layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-41 - - Import a NN consisting of two Caffe layers - Input layer and Concat layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-42 - - Import a NN consisting of two Caffe layers - Input layer and Slice layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-43 - - Import a NN consisting of two Caffe layers - Input layer and Eltwise layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-44 - - Import a NN consisting of two Caffe layers - Input layer and Filter layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-45 - - Import a NN consisting of two Caffe layers - Input layer and Parameter layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-46 - - Import a NN consisting of two Caffe layers - Input layer and Reduction layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-47 - - Import a NN consisting of two Caffe layers - Input layer and Silence layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-48 - - Import a NN consisting of two Caffe layers - Input layer and ArgMax layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-49 - - Import a NN consisting of two Caffe layers - Input layer and Softmax layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-50 - - Import a NN consisting of two Caffe layers - Input layer and Python layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-51 - - Import a NN consisting of two Caffe layers - Input layer and MultinomialLogisticLoss layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-52 - - Import a NN consisting of two Caffe layers - Input layer and Infogain layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-53 - - Import a NN consisting of two Caffe layers - Input layer and SoftmaxWithLoss layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-54 - - Import a NN consisting of two Caffe layers - Input layer and EuclideanLoss layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-55 - - Import a NN consisting of two Caffe layers - Input layer and HingeLoss layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-56 - - Import a NN consisting of two Caffe layers - Input layer and SigmoidCrossEntropyLoss layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-57 - - Import a NN consisting of two Caffe layers - Input layer and Accuracy layer - - During import no crashes or error messages occurred - - * - TST-2 - - TST-2-58 - - Import a NN consisting of two Caffe layers - Input layer and ContrastiveLoss layer - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-1 - - Import a NN consisting of a single Caffe2 Add operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-2 - - Import a NN consisting of a single Caffe2 AveragePool2D operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-3 - - Import a NN consisting of a single Caffe2 Concat operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-4 - - Import a NN consisting of a single Caffe2 Conv2D operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-5 - - Import a NN consisting of a single Caffe2 FC operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-6 - - Import a NN consisting of a single Caffe2 LRN operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-7 - - Import a NN consisting of a single Caffe2 Sigmoid operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-8 - - Import a NN consisting of a single Caffe2 MaxPool2D operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-9 - - Import a NN consisting of a single Caffe2 Mul operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-10 - - Import a NN consisting of a single Caffe2 Relu operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-11 - - Import a NN consisting of a single Caffe2 Reshape operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-12 - - Import a NN consisting of a single Caffe2 Softmax operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-13 - - Import a NN consisting of a single Caffe2 Tanh operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-14 - - Import a NN consisting of a single Caffe2 PadImage operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-15 - - Import a NN consisting of a single Caffe2 BatchToSpace operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-16 - - Import a NN consisting of a single Caffe2 SpaceToBatch operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-17 - - Import a NN consisting of a single Caffe2 Transpose operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-18 - - Import a NN consisting of a single Caffe2 Mean operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-19 - - Import a NN consisting of a single Caffe2 Sub operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-20 - - Import a NN consisting of a single Caffe2 Div operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-21 - - Import a NN consisting of a single Caffe2 Squeeze operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-22 - - Import a NN consisting of a single Caffe2 Exp operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-23 - - Import a NN consisting of a single Caffe2 TopK operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-24 - - Import a NN consisting of a single Caffe2 Split operation - - During import no crashes or error messages occurred - - * - TST-3 - - TST-3-25 - - Import a NN consisting of a single Caffe2 Cast operation - - During import no crashes or error messages occurred - - * - TST-4 - - TST-4-1 - - Import Inception V3 NN model - - During import no crashes or error messages occurred - - * - TST-5 - - TST-5-1 - - Import MobileNet NN model - - During import no crashes or error messages occurred - - * - TST-6 - - TST-6-1 - - Import Inception V3 NN model, serialize all model weights, compare serialized data with the initial NN model - - Test executed successfully, serialized weights are equal to initial model weights - - * - TST-6 - - TST-6-2 - - Import MobileNet NN model, serialize all model weigths, compare serialized data with the initial NN model - - Test executed successfully, serialized weights are equal to initial model weights - - * - TST-7 - - TST-7-1 - - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Ubuntu - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-7 - - TST-7-2 - - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Ubuntu - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-7 - - TST-7-3 - - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Tizen - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-7 - - TST-7-4 - - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Tizen - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-7 - - TST-7-5 - - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with SmartMachine OS - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-7 - - TST-7-6 - - Generate binary for the MobileNet NN model and run its inference on ARM CPU with SmartMachine OS - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-8 - - TST-8-1 - - Generate binary for the Inception V3 NN model and run its inference on a GPU-enabled device - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-8 - - TST-8-2 - - Generate binary for the MobileNet V3 NN model and run its inference on a GPU-enabled device - - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs - - * - TST-9 - - TST-9-1 - - Provide correct NN model, compile it as a static library, then check that corresponding binary exists and it is a static library - - Test executed successfully - - * - TST-9 - - TST-9-2 - - Provide correct NN model, compile it as a shared library, then check that corresponding binary exists and it is a shared library - - Test executed successfully - - * - TST-9 - - TST-9-3 - - Provide incorrect model, compile it as a static library, then check that no compiled artifact is produced - - Test executed successfully - - * - TST-9 - - TST-9-4 - - Provide incorrect model, compile it as a shared library, then check that no compiled artifact is produced - - Test executed successfully - - * - TST-10 - - TST-10-1 - - Check that a static library is provided after compiling Inception V3 as a static library - - Test executed successfully - - * - TST-10 - - TST-10-2 - - Check that a shared library is provided after compiling Inception V3 as a shared library - - Test executed successfully - - * - TST-11 - - TST-11-1 - - Check that a static library is provided after compiling MobileNet as a static library - - Test executed successfully - - * - TST-11 - - TST-11-2 - - Check that a shared library is provided after compiling MobileNet as a shared library - - Test executed successfully - - * - TST-12 - - TST-12-1 - - Check that configuration object is constructed correctly when getting configuration parameters from command line - - Test executed successfully - - * - TST-12 - - TST-12-2 - - Check that configuration object is constructed correctly when getting configuration parameters from config file - - Test executed successfully - - * - TST-12 - - TST-12-3 - - Check that configuration object is constructed correctly when getting configuration parameters from environment variables - - Test executed successfully - - * - TST-13 - - TST-13-1 - - Compile Inception V3 as static library for CPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-2 - - Compile Inception V3 as shared library for CPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-3 - - Compile Inception V3 as static library for GPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-4 - - Compile Inception V3 as shared library for GPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-5 - - Compile MobileNet as static library for CPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-6 - - Compile MobileNet as shared library for CPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-7 - - Compile MobileNet as static library for GPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-13 - - TST-13-8 - - Compile MobileNet as shared library for GPU, provide it and the original model with same correct input data, then compare the result from original model with the result from compiled artifact - - Test executed successfully, results are comparable - - * - TST-14 - - TST-14-1 - - Provide compiled Inception V3 artifact with invalid input, check that no unexpected termination occurs - - Test executed successfully - - * - TST-14 - - TST-14-2 - - Provide compiled Inception V3 artifact with invalid input, check that an error message is provided - - Test executed successfully - - * - TST-14 - - TST-14-3 - - Provide compiled MobileNet artifact with invalid input, check that no unexpected termination occurs - - Test executed successfully - - * - TST-14 - - TST-14-4 - - Provide compiled MobileNet artifact with invalid input, check that an error message is provided - - Test executed successfully - - * - TST-15 - - TST-15-1 - - Check that the OS used during test environment build is Linux-based - - Test executed successfully - - * - TST-16 - - TST-16-1 - - Compile a valid NN model, then check that C/C++ header corresponding to compiled artifact exists - - Test executed successfully - - * - TST-16 - - TST-16-2 - - Compile a valid NN model, then if C/C++ header corresponding to compiled artifact exists, verify its validity - - Test executed successfully - - -SW System Test Result -===================== - -Test Schedule -------------- - -**Table 6-1. Test Period** - -.. list-table:: - :widths: 40 20 20 - :header-rows: 1 - - * - - - Plan - - Actual - - * - Entire test period - - 04/2018 - 12/2018 - - 06/2018 - 12/2018 - - * - Reason for change period - - - - - -| - -**Table 6-2. Test plan vs Actual** - -+------------+---------------------+--------------------+-----------------+ -| Test round | Plan | Actual | Remarks | -| +-----------+---------+-----------+--------+ | -| | Schedule | MM | Schedule | MM | | -+============+===========+=========+===========+========+=================+ -| R.1 | 09/2018 | 1 | 09/2018 | 1 | nncc v0.3 | -+------------+-----------+---------+-----------+--------+-----------------+ -| R.2 | 11/2018 | 1 | 10/2018 | 1 | nncc v0.7 | -+------------+-----------+---------+-----------+--------+-----------------+ -| R.2.1 | 11/2018 | 1 | 11/2018 | 1 | nncc v0.9 | -+------------+-----------+---------+-----------+--------+-----------------+ -| R.3 | 12/2018 | 1 | 12/2018 | 1 | nncc v1.0 | -+------------+-----------+---------+-----------+--------+-----------------+ - -Test Output ------------ - -Test Round 1 -```````````` - -:: - - Discovering tests... - Total tests discovered: 80 - Log folder: /home/ivan/GIT/nnc_doc_test/2018_09_25_14_30_35 - Threads quantity: 8 - - ================================= - Tests: 80/80, successes: 80, failures: 0, errors: 0, other: 0 - ================================= - Result: - Total tests: 80 - Successes: 80 - Errors: 0 - Failures: 0 - Skipped: 0 - Expected failures: 0 - Unexpected successes: 0 - -Test Round 2 -```````````` - -:: - - Discovering tests... - Total tests discovered: 118 - Log folder: /home/ivan/GIT/nnc_doc_test/2018_10_23_16_33_19 - Threads quantity: 8 - - ================================= - Tests: 118/118, successes: 118, failures: 0, errors: 0, other: 0 - ================================= - Result: - Total tests: 118 - Successes: 118 - Errors: 0 - Failures: 0 - Skipped: 0 - Expected failures: 0 - Unexpected successes: 0 - -Test Round 2.1 -`````````````` - -:: - - Discovering tests... - Total tests discovered: 122 - Log folder: /home/ivan/GIT/nnc_doc_test/2018_11_09_21_46_03 - Threads quantity: 1 - - ================================= - Tests: 122/122, successes: 122, failures: 0, errors: 0, other: 0 - ================================= - Result: - Total tests: 122 - Successes: 122 - Errors: 0 - Failures: 0 - Skipped: 0 - Expected failures: 0 - Unexpected successes: 0 - -Test Round 3 -```````````` - -:: - - Discovering tests... - Total tests discovered: 128 - Log folder: /home/ivan/GIT/nnc_doc_test/2018_12_03_22_10_26 - Threads quantity: 1 - - ================================= - Tests: 128/128, successes: 128, failures: 0, errors: 0, other: 0 - ================================= - Result: - Total tests: 128 - Successes: 128 - Errors: 0 - Failures: 0 - Skipped: 0 - Expected failures: 0 - Unexpected successes: 0 - - -Test Result ------------ - - -**Table 6-3. System Test Result** - -+--------------------+------------------------------------+------------+----------------+ -| Official release | The number of test case | Pass rate | Result | -| +-------+-------+------+-------------+ | | -| | Total | Pass | Fail | Unsupported | Pass/Total | | -+====================+=======+=======+======+=============+============+================+ -| v0.3 | 80 | 80 | 0 | 1 | 1.0 |**Tests passed**| -+--------------------+-------+-------+------+-------------+------------+----------------+ -| v0.7 | 118 | 118 | 0 | 15 | 1.0 |**Tests passed**| -+--------------------+-------+-------+------+-------------+------------+----------------+ -| v0.9 | 122 | 122 | 0 | 15 | 1.0 |**Tests passed**| -+--------------------+-------+-------+------+-------------+------------+----------------+ -| v1.0 | 128 | 128 | 0 | 15 | 1.0 |**Tests passed**| -+--------------------+-------+-------+------+-------------+------------+----------------+ - -| - -**Table 6-4. Test Problem Result** - -+---------+---------------------------+---------------------------+ -| Status | Unresolved | Resolved | -+---------+----------+--------+-------+----------+--------+-------+ -| Round | Critical | Major | Minor | Critical | Major | Minor | -+=========+==========+========+=======+==========+========+=======+ -| R.1 | 0 | 0 | 0 | 0 | 0 | 0 | -+---------+----------+--------+-------+----------+--------+-------+ -| R.2 | 0 | 0 | 0 | 0 | 0 | 0 | -+---------+----------+--------+-------+----------+--------+-------+ -| R.2.1 | 0 | 0 | 0 | 0 | 0 | 0 | -+---------+----------+--------+-------+----------+--------+-------+ -| R.3 | 0 | 0 | 0 | 0 | 0 | 0 | -+---------+----------+--------+-------+----------+--------+-------+ - -| - -**Table 6-5. Round Test Item** - -+---------+----------------------------------------------------------------------------------------------+ -| Round | Test Item | -+=========+==============================================================================================+ -| R.1 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | -| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | -| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | -| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | -| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | -| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | -| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, | -| | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6, | -| | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2 | -+---------+----------------------------------------------------------------------------------------------+ -| R.2 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | -| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | -| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | -| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | -| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | -| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | -| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, | -| | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6, | -| | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2, TST-1-11, TST-1-25, | -| | TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, TST-1-16, TST-1-17, TST-1-18, TST-1-19, | -| | TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, TST-1-2, TST-1-33, TST-1-35, TST-1-38, | -| | TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, TST-1-44, TST-1-46, TST-1-47, TST-1-48, | -| | TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, TST-1-6, TST-1-7, TST-1-8, TST-12-1 | -+---------+----------------------------------------------------------------------------------------------+ -| R.2.1 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | -| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | -| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | -| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | -| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | -| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | -| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-7-3, TST-7-4, TST-7-5, | -| | TST-7-6, TST-9-1, TST-9-2, TST-9-3, TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, | -| | TST-13-1, TST-13-2, TST-13-5, TST-13-6, TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, | -| | TST-16-1, TST-16-2, TST-1-11, TST-1-25, TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, | -| | TST-1-16, TST-1-17, TST-1-18, TST-1-19, TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, | -| | TST-1-2, TST-1-33, TST-1-35, TST-1-38, TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, | -| | TST-1-44, TST-1-46, TST-1-47, TST-1-48, TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, | -| | TST-1-6, TST-1-7, TST-1-8, TST-12-1 | -+---------+----------------------------------------------------------------------------------------------+ -| R.3 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | -| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | -| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | -| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | -| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | -| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | -| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-7-3, TST-7-4, TST-7-5, | -| | TST-7-6, TST-9-1, TST-9-2, TST-9-3, TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, | -| | TST-13-1, TST-13-2, TST-13-5, TST-13-6, TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, | -| | TST-16-1, TST-16-2, TST-1-11, TST-1-25, TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, | -| | TST-1-16, TST-1-17, TST-1-18, TST-1-19, TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, | -| | TST-1-2, TST-1-33, TST-1-35, TST-1-38, TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, | -| | TST-1-44, TST-1-46, TST-1-47, TST-1-48, TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, | -| | TST-1-6, TST-1-7, TST-1-8, TST-12-1, TST-8-1, TST-8-2, TST-13-3, TST-13-4, TST-13-7, | -| | TST-13-8 | -+---------+----------------------------------------------------------------------------------------------+ - -Defect rates -```````````` -No defects are found - - -Main problem contents ---------------------- - -No problems were found - - -Total Result -============ - -128 of 128 system tests are passed. 15 tests currently unsupported by nncc v1.0. |