diff options
Diffstat (limited to 'docs')
61 files changed, 1991 insertions, 203 deletions
diff --git a/docs/conf.py b/docs/conf.py index 9b870097a..50256719a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -21,7 +21,7 @@ copyright = '2020, Samsung Research & contributors' author = 'Samsung Research & contributors' # The full version, including alpha/beta/rc tags -release = '1.9.0' +release = '1.25.0' # -- General configuration --------------------------------------------------- diff --git a/docs/howto/how-to-add-a-new-operation.md b/docs/howto/how-to-add-a-new-operation.md index 8ea7014a3..241ba6cb1 100644 --- a/docs/howto/how-to-add-a-new-operation.md +++ b/docs/howto/how-to-add-a-new-operation.md @@ -6,4 +6,4 @@ ## Runtime -- [How to introduce a new operatoin into runtime](how-to-introduce-a-new-operation-into-runtime.md) +- [How to introduce a new operation into runtime](how-to-introduce-a-new-operation-into-runtime.md) diff --git a/docs/howto/how-to-build-compiler.md b/docs/howto/how-to-build-compiler.md index e9dcb263a..3a9638677 100644 --- a/docs/howto/how-to-build-compiler.md +++ b/docs/howto/how-to-build-compiler.md @@ -1,7 +1,211 @@ # How to Build Compiler +This document is based on the system where Ubuntu Desktop Linux 18.04 LTS is installed with default +settings, and can be applied in other environments without much difference. For reference, the +development of our project started in the Ubuntu Desktop Linux 16.04 LTS environment. +As of now, to build in 16.04, please use gcc 7.x or above. + ## Build Requires +If you are going to build this project, the following modules must be installed on your system: + +- CMake +- Boost C++ libraries + +In the Ubuntu, you can easily install it with the following command. + +``` +$ sudo apt-get install cmake libboost-all-dev +``` + +If your linux system does not have the basic development configuration, you will need to install +more packages. A list of all packages needed to configure the development environment can be found +in the https://github.com/Samsung/ONE/blob/master/infra/docker/Dockerfile.1804 file. + +Here is a summary of it + +``` +$ sudo apt-get install \ +build-essential \ +clang-format-8 \ +cmake \ +doxygen \ +git \ +hdf5-tools \ +lcov \ +libatlas-base-dev \ +libboost-all-dev \ +libgflags-dev \ +libgoogle-glog-dev \ +libgtest-dev \ +libhdf5-dev \ +libprotobuf-dev \ +protobuf-compiler \ +pylint \ +python3 \ +python3-pip \ +python3-venv \ +scons \ +software-properties-common \ +unzip \ +wget + +$ mkdir /tmp/gtest +$ cd /tmp/gtest +$ cmake /usr/src/gtest +$ make +$ sudo mv *.a /usr/lib + +$ pip install yapf==0.22.0 numpy +``` + +Additional install python3.8 if you are using Ubuntu 18.04. +``` +$ sudo apt-get install \ +python3.8 \ +python3.8-dev \ +python3.8-venv +``` + +If you get `Unable to locate package clang-format-8` then just use `clang-format`. + ## Build for Ubuntu -## Build for windows +In a typical linux development environment, including Ubuntu, you can build the compiler with a +simple command like this: + +``` +$ git clone https://github.com/Samsung/ONE.git one +$ cd one +$ ./nncc configure +$ ./nncc build +``` +Build artifacts will be placed in `build` folder. + +To run unit tests: +``` +$ ./nncc test +``` + +Above steps will build all the modules in the compiler folder. There are modules that are currently +not active. To build only as of now active modules of the compiler, we provide a preset of modules +to build with below command: +``` +$ ./nnas create-package --prefix $HOME/.local +``` + +With this command, `~/.local` folder will contain all files in release. +If you have added `~/.local/bin` in PATH, then you will now have latest compiler binaries. + +### Build for debug and release separately + +Build target folder can be customized by `NNCC_WORKSPACE` environment, as we may want to separate +debug and release builds. + +``` +$ NNCC_WORKSPACE=build/debug ./nncc configure +$ ./nncc build +``` +will build debug version in `build/debug` folder, and + +``` +$ NNCC_WORKSPACE=build/release ./nncc configure -DCMAKE_BUILD_TYPE=Release +$ ./nncc build +``` +will build release version in `build/release` folder. + +### Trouble shooting + +If you are using python3.8, as there is no TensorFlow1.13.2 package for python3.8, build may fail. +Please install python3.7 or lower versions as default python3. + +## Build for Windows + +To build for Windows, we use MinGW(Minimalist GNU for Windows). [Here](https://github.com/git-for-windows/build-extra/releases) you can download a tool that includes it. + +``` +$ git clone https://github.com/Samsung/ONE.git one +$ cd one +$ NNAS_BUILD_PREFIX=build ./nnas create-package --preset 20200731_windows --prefix install +``` + +- `NNAS_BUILD_PREFIX` is the path to directory where compiler-build-artifacts will be stored. +- `--preset` is the one that specifies a version you will install. You can see `infra/packaging/preset/` directory for more details and getting latest version. +- `--prefix` is the install directory. + +## Cross build for Ubuntu/ARM32 (experimental) + +Some modules are availble to run in Ubuntu/ARM32 through cross building. + +While configuring the build, some modules need to execute tools for generating +test materials and they need to execute in the host(x86-64). So some modules +are needed to build the tools for host before cross building. + +Cross build overall steps are like, (1) configure for host +(2) build tools for host (3) configure for ARM32 target (4) and then build +for ARM32 target. + +Unit tests can also run in target device. +But value test needs to run TensorFlow lite to get expected results, +and it would be a task to do this so the data files from host execution +are used instead. + +Thus to run the unit tests in the target, running in host is needed in prior. + +### Prepare root file system + +You should prepare Ubuntu/ARM32 root file system for cross compilation. +Please refer +[how-to-cross-build-runtime-for-arm.md](how-to-cross-build-runtime-for-arm.md) +for preparation. + +You can set `ROOTFS_ARM` environment variable if you have in alternative +folder. + +### Clean existing external source for patches + +Some external projects from source are not "cross compile ready with CMake" +projects. This experimental project prepared some patches for this. +Just remove the source and stamp file like below and the `make` will prepare +patch applied source codes. +``` +rm -rf externals/HDF5 +rm -rf externals/PROTOBUF +rm externals/HDF5.stamp +rm externals/PROTOBUF.stamp +``` + +### Build + +To cross build, `infra/nncc/Makefile.arm32` file is provided as an example to +work with `make` command. +``` +make -f infra/nncc/Makefile.arm32 cfg +make -f infra/nncc/Makefile.arm32 debug +``` +First `make` will run above steps (1), (2) and (3). Second `make` will run (4). + +### Test + +Preprequisite for testing in ARM32 device. +``` +# numpy is required for value match in ARM32 target device +sudo apt-get install python3-pip +python3 -m pip install numpy +``` + +You can also run unit tests in ARM32 Ubuntu device with cross build results. +First you need to run the test in host to prepare files that are currently +complicated in target device. +``` +# run this in x86-64 host +make -f infra/nncc/Makefile.arm32 test_prep + +# run this in ARM32 target device +make -f infra/nncc/Makefile.arm32 test +``` + +NOTE: this assumes +- host and target have same directoy structure +- should copy `build` folder to target or +- mounting `ONE` folder with NFS on the target would be simple diff --git a/docs/howto/how-to-build-package.md b/docs/howto/how-to-build-package.md index 65cc3c753..803466bfe 100644 --- a/docs/howto/how-to-build-package.md +++ b/docs/howto/how-to-build-package.md @@ -1 +1,188 @@ # How to Build Package + +## Overview + +This document describes how to build a Package to run the model in our runtime +_onert_ that consists of model and additional file(s). Users can build a +package through command line tools. + +Steps of building a Package: +1. Import model and convert to circle +1. Optionally, optimize and quantize circle +1. Create package from circle + +NOTE: Examples and options of each command shown below are from the version of +writing this document. They may differ from latest version of commands, 1.9.0. +Please fire an issue or post a PR to correct them if anything needs update. + +## Import model + +Currently TensorFlow and TensorFlow lite models are supported as of writing +this document. + +To import a model, use `one-import` with a model framework key and arguments. +``` +$ one-import FRAMEWORK [arguments] +``` + +Execute `one-import` without any key will show the list of supported frameworks. + +Example of `one-import` command: +``` +$ one-import +Usage: one-import [FRAMEWORK] ... +Available FRAMEWORK drivers: + bcq + tf + tflite +``` + +### Example for TensorFlow + +This is an example to import TensorFlow model: +``` +$ one-import tf --input_path mymodel.pb --output_path mymodel.circle \ +--input_arrays input1,input2 --input_shapes "1,224,224,3:1000" \ +--output_arrays output +``` + +Running with `--help` will show current required/optional arguments: +``` +$ one-import tf --help +Convert TensorFlow model to circle. +Usage: one-import-tf + --version Show version information and exit + --input_path <path/to/tfmodel> + --output_path <path/to/circle> + --input_arrays <names of the input arrays, comma-separated> + --input_shapes <input shapes, colon-separated> + --output_arrays <names of the output arrays, comma-separated> + --v2 Use TensorFlow 2.x interface (default is 1.x interface) +``` + +### Example for TensorFlow lite + +This is an example to import TensorFlow lite model: +``` +$ one-import tflite --input_path mymodel.tflite --output_path mymodel.circle +``` + +Likewise, running with `--help` will show current required/optional arguments: +``` +$ one-import tflite --help +Convert TensorFlow lite model to circle. +Usage: one-import-tflite + --version Show version information and exit + --input_path <path/to/tflitemodel> + --output_path <path/to/circle> +``` + +### Example for TensorFlow Model Including BCQ Information + +This is an example to import TensorFlow model which includes BCQ information. +As a result of this command, BCQ information nodes will be preserved. +``` +$ one-import bcq --input_path bcqmodel.pb --output_path bcqmodel.circle +``` + +Likewise, running with `--help` will show current required/optional arguments: +``` +$ one-import bcq --help +Convert TensorFlow model with BCQ to circle. +Usage: one-import-bcq + --version Show version information and exit + --input_path <path/to/tfmodel/with/BCQ> + --output_path <path/to/circle> + --input_arrays <names of the input arrays, comma-separated> + --input_shapes <input shapes, colon-separated> + --output_arrays <names of the output arrays, comma-separated> + --v2 Use TensorFlow 2.x interface (default is 1.x interface) +``` + +## Optimize circle model + +circle model can be optimized for better performance and smaller size. +Typical optimization algorithm for this is to fuse some patterns of operators +to one fused operator. + +This is an example to optimize circle model: +``` +$ one-optimize --all --input_path mymodel.circle --output_path optmodel.circle +``` + +Run with `--help` will show current optimization options: +``` +$ one-optimize --help +Optimize circle model. +Usage: one-optimize + --version Show version information and exit + --all Enable all optimization algorithms + --fuse_bcq Enable FuseBCQ Pass + --fuse_instnorm Enable FuseInstanceNormalization Pass + --resolve_customop_add + Enable ResolveCustomOpAddPass Pass + --resolve_customop_batchmatmul + Enable ResolveCustomOpBatchMatMulPass Pass + --resolve_customop_matmul + Enable ResolveCustomOpMatMulPass Pass + --input_path <path/to/input/circle> + --output_path <path/to/output/circle> +``` + +## Quantize circle model + +Floating-point circle model can be quantized to lower-precision format +(e.g., uint8 or int16) for faster inference speed and smaller model size, +by reducing the number of bits that represent weights and activations. + +This is an example to quantize circle model: +``` +$ one-quantize --input_path mymodel.circle --output_path quantmodel.circle +``` + +Like wise, `--help` will show current quantization options: +``` +$ one-quantize --help +Quantize circle model. +Usage: one-quantize + --version Show version information and exit + --input_dtype Input data type (supported: float32, default=float32) + --quantized_dtype Output quantized data type (supported: uint8, default=uint8) + --granularity Quantize granularity (supported: layer, channel, default=layer) + --min_percentile Minimum percentile (0.0~100.0, default=1.0) + --max_percentile Maximum percentile (0.0~100.0, default=99.0) + --mode Record mode (supported: percentile/moving_average, default=percentile) + --input_path <path/to/input/circle> + --input_data <path/to/input/data> + --output_path <path/to/output/circle> +``` + +## Pack circle model + +Use `one-pack` command to create package. + +``` +$ one-pack -i mymodel.circle -o nnpackage +``` + +`nnpackage` is a folder containing circle model and addition file(s) + +``` +$ tree nnpackage +nnpackage +└── mymodel + ├── metadata + │ └── MANIFEST + └── mymodel.circle +``` + +Likewise, `--help` will show current package options: + +``` +$ one-pack --help +Package circle to nnpkg +Usage: one-pack + -v, --version Show version information and exit + -i <path/to/circle> + -o <path/to/nnpackage/folder> +``` diff --git a/docs/howto/how-to-build-runtime-tizen-gbs-rpi4.md b/docs/howto/how-to-build-runtime-tizen-gbs-rpi4.md new file mode 100644 index 000000000..5479a342e --- /dev/null +++ b/docs/howto/how-to-build-runtime-tizen-gbs-rpi4.md @@ -0,0 +1,316 @@ +# How to Build Runtime with GBS for Tizen/RPi4 + +This document describes how to build runtime with GBS for Tizen AARCH64. +As a real example, we'll also describe how to prepare Tizen on Raspberry Pi 4 +and show you how to run our test package runner `onert_run`. + +For ARM32, there would be not much difference with some changes. + +Host PC is Ubuntu 18.04 but other versions or distro may work with a little +adjustments. + +Detailed technical informations are not described here so please read referecnce +pages while you go on. + + +## Setting up build environment + +(1) Add Tizen build tools repo +``` +$ sudo vim /etc/apt/sources.list +``` +Add this at the end +``` +deb [trusted=yes] http://download.tizen.org/tools/latest-release/Ubuntu_18.04/ / +``` +Note: There's a slash('/`) at the end. + +For other versions of Ubuntu, please refer +http://download.tizen.org/tools/latest-release/ lists. + +(2) Update package informations and upgrade to latest +``` +$ sudo apt-get update +$ sudo apt-get upgrade +``` + +(3) Install GBS tools +``` +$ sudo apt-get install gbs mic +``` + +To get more informations, please refer [HERE](https://source.tizen.org/ko/documentation/developer-guide/getting-started-guide/installing-development-tools) + +## Build ONERT + +(1) Set `python2` as default python + +Some tools of GBS run in `python2` and won't run with `python3`. +Please check `python` version and set it to 2.x. + +(2) set `TIZEN_BUILD_ROOT` + +You may set `GBS-ROOT` to any place you like. Ususally we use home folder. +``` +$ export TIZEN_BUILD_ROOT=$HOME/GBS-ROOT/ +``` +Adding to `$HOME/.profile` file would be a good thing. + +(3) clone ONE repo + +``` +git clone https://github.com/Samsung/ONE.git +``` + +(4) Build + +``` +$ cd ONE + +$ gbs -c infra/nnfw/config/gbs.conf build --include-all -A aarch64 --define 'test_build 1' +``` +- `-A aarch64` is to set architecture to AARCH64. Use `arm32` for ARM32 target. +- `--define 'test_build 1'` is to enable test build so that we can use `onert_run` + +Now take a cup of coffee. + +(5) Build result RPM packages + +``` +$ ls ~/GBS-ROOT/local/repos/tizen/aarch64/RPMS +nnfw-1.10.0-1.aarch64.rpm +nnfw-debuginfo-1.10.0-1.aarch64.rpm +nnfw-debugsource-1.10.0-1.aarch64.rpm +nnfw-devel-1.10.0-1.aarch64.rpm +nnfw-minimal-app-1.10.0-1.aarch64.rpm +nnfw-minimal-app-debuginfo-1.10.0-1.aarch64.rpm +nnfw-plugin-devel-1.10.0-1.aarch64.rpm +nnfw-test-1.10.0-1.aarch64.rpm +nnfw-test-debuginfo-1.10.0-1.aarch64.rpm +``` + +`-1.10.0-1` may differ as this document was written with under `1.10.0` development. + +## Prepare Tizen on Raspberry Pi 4 + +Please refer https://wiki.tizen.org/Quick_guide_for_RPI4 for detailed descriptions. + +(1) Download flashing tool +``` +$ wget \ +https://git.tizen.org/cgit/platform/kernel/u-boot/plain/scripts/tizen/sd_fusing_rpi3.sh?h=tizen \ +--output-document=sd_fusing_rpi3.sh + +$ chmod 755 sd_fusing_rpi3.sh +``` + +(2) Prepare Micro-SD memory card. + +You first need to find out device name. This document will skip how to find this. +Suppose it's `/dev/sdj`: +``` +$ sudo ./sd_fusing_rpi3.sh -d /dev/sdj --format +``` +You need to change `/dev/sdj` to your configuration. + +Partition table may look like this +``` +Device Boot Start End Sectors Size Id Type +/dev/sdj1 * 8192 139263 131072 64M e W95 FAT16 (LBA) +/dev/sdj2 139264 6430719 6291456 3G 83 Linux +/dev/sdj3 6430720 9183231 2752512 1.3G 83 Linux +/dev/sdj4 9183232 62521343 53338112 25.4G 5 Extended +/dev/sdj5 9185280 61958143 52772864 25.2G 83 Linux +/dev/sdj6 61960192 62025727 65536 32M 83 Linux +/dev/sdj7 62027776 62044159 16384 8M 83 Linux +/dev/sdj8 62046208 62111743 65536 32M 83 Linux +/dev/sdj9 62113792 62130175 16384 8M 83 Linux +/dev/sdj10 62132224 62263295 131072 64M 83 Linux +/dev/sdj11 62265344 62521343 256000 125M 83 Linux +``` + +(3) Download images + +Please visit http://download.tizen.org/snapshots/tizen/unified/latest/images/standard/iot-boot-arm64-rpi4/ +and http://download.tizen.org/snapshots/tizen/unified/latest/images/standard/iot-headed-3parts-aarch64-rpi. + +Please visit `iot-boot-armv7l-rpi4` folder for ARM32 images. + +Get latest file. As of writing this document, name has `20200908.3`. +``` +$ wget http://download.tizen.org/snapshots/tizen/unified/latest/images/standard/iot-boot-arm64-rpi4/tizen-unified_20200908.3_iot-boot-arm64-rpi4.tar.gz + +$ wget http://download.tizen.org/snapshots/tizen/unified/latest/images/standard/iot-headed-3parts-aarch64-rpi/tizen-unified_20200908.3_iot-headed-3parts-aarch64-rpi.tar.gz +``` + +(4) Flash images to memory card + +As like above, suppose memory card is at `/dev/sdj` +``` +$ sudo ./sd_fusing_rpi3.sh -d /dev/sdj \ +-b tizen-unified_20200908.3_iot-boot-arm64-rpi4.tar.gz \ +tizen-unified_20200908.3_iot-headed-3parts-aarch64-rpi.tar.gz +``` +You need to change `/dev/sdj` to your configuration and also `tizen-unified_...` file to your +latest download file name. + +(5) Assign IP address for `sdb` connection + +Here, we provide a way to connect `sdb` tool through TCP/IP. + +Below steps will modify root image and set fixed IP address. + +(5-1) Mount image to host +``` +$ mkdir j2 +$ sudo mount /dev/sdj2 j2 +``` +As like above, please update `/dev/sdj2` to your configuration. + +(5-2) Add a new file +``` +$ vi j2/etc/systemd/system/ip.service +``` +and set as like: +``` +[Service] +Restart=always +RestartSec=1 +User=root +ExecStart=/bin/sh -c "ifconfig eth0 192.168.x.y netmask 255.255.255.0 up" + +[Install] +WantedBy=multi-user.target +``` +Replace 192.168.x.y to your actual ip address. + + +(5-3) Add a symbolic link +``` +$ sudo mkdir -p j2/etc/systemd/system/multi-user.target.wants/ +$ pushd j2/etc/systemd/system/multi-user.target.wants/ +$ sudo ln -s ../../system/ip.service . +$ popd +``` + +(5-4) Now that every thing is ready, unmount and unplug your memory card and plug into +RPi4, turn on the power. +``` +$ sync +$ sudo umount j2 +``` + +## sdb connect to Tizen/RPi4 + +You may need to install Tizen Studio to use `sdb` command. +Please visit https://developer.tizen.org/ if you don't have this. + +We assume `sdb` command is in the PATH. + +(1) Connect + +``` +$ sdb connect 192.168.x.y +connecting to 192.168.x.y:26101 ... +connected to 192.168.x.y:26101 +``` +Please update `192.168.x.y` part to your actual IP address. + +Check with `devices` command: you should see `rpi3` or alike. +``` +$ sdb devices +List of devices attached +192.168.x.y:26101 device rpi3 +``` + +(2) Remount filesystem with R/W + +You need to remount file system with Read/Write so that you can install packages. +``` +$ sdb root on +$ sdb shell +``` +Inside your Tizen/RPi4: +``` +sh-3.2# mount -o rw,remount / +``` + +(3) Download dependent packages + +In your host, maybe with another terminal, download packages from +http://download.tizen.org/releases/daily/tizen/unified/latest/repos/standard/packages/aarch64/ + +``` +$ wget http://download.tizen.org/releases/daily/tizen/unified/latest/repos/standard/packages/aarch64/libarmcl-v21.02-17.5.aarch64.rpm + +$ wget http://download.tizen.org/releases/daily/tizen/unified/latest/repos/standard/packages/aarch64/libhdf5-101-1.10.1-3.85.aarch64.rpm + +$ wget http://download.tizen.org/releases/daily/tizen/unified/latest/repos/standard/packages/aarch64/libhdf5_cpp101-1.10.1-3.85.aarch64.rpm +``` + +(4) Copy to device +``` +$ sdb push libarmcl-v21.02-17.5.aarch64.rpm /opt/usr/home/owner/share/tmp/ +$ sdb push libhdf5-101-1.10.1-3.85.aarch64.rpm /opt/usr/home/owner/share/tmp/ +$ sdb push libhdf5_cpp101-1.10.1-3.85.aarch64.rpm /opt/usr/home/owner/share/tmp/ +``` +And our runtime packages +``` +$ cd ~/GBS-ROOT/local/repos/tizen/aarch64/RPMS +$ sdb push nnfw-1.10.0-1.aarch64.rpm /opt/usr/home/owner/share/tmp/ +$ sdb push nnfw-test-1.10.0-1.aarch64.rpm /opt/usr/home/owner/share/tmp/ +``` + +(5) Install dependent packages + +Within Tizen/RPi4 shell +``` +sh-3.2# cd /opt/usr/home/owner/share/tmp/ + +sh-3.2# rpm -i libarmcl-v21.02-17.5.aarch64.rpm +sh-3.2# rpm -i libhdf5-101-1.10.1-3.85.aarch64.rpm +sh-3.2# rpm -i libhdf5_cpp101-1.10.1-3.85.aarch64.rpm +``` +There may be message like this but it seems OK: +``` +/sbin/ldconfig: Cannot lstat /lib64/libhdf5.so.101.0.0: Permission denied +``` +Continue install +``` +sh-3.2# rpm -i nnfw-1.10.0-1.aarch64.rpm +sh-3.2# rpm -i nnfw-test-1.10.0-1.aarch64.rpm +``` + +Our `Product` binary folder is installed at `/opt/usr/nnfw-test`. +``` +sh-3.2# cd /opt/usr/nnfw-test +sh-3.2# ls -al +total 16 +drwxr-xr-x 4 root root 4096 Jan 1 09:05 . +drwxr-xr-x 14 root root 4096 Jan 1 09:05 .. +drwxr-xr-x 3 root root 4096 Jan 1 09:05 Product +drwxr-xr-x 3 root root 4096 Jan 1 09:05 infra +``` + +(6) Run nnpackage + +Refer `how-to-build-package.md` document to produce nnpackage from a model. + +Assume `mobilenet_v2_1.4_224` nnpackage is already copied to +`/opt/usr/home/owner/media/models` folder with `sdb` command. + +``` +sh-3.2# BACKENDS="cpu" Product/out/bin/onert_run \ +--nnpackage /opt/usr/home/owner/media/models/mobilenet_v2_1.4_224 + +Package Filename /opt/usr/home/owner/media/models/mobilenet_v2_1.4_224 +=================================== +MODEL_LOAD takes 65.403 ms +PREPARE takes 158.716 ms +EXECUTE takes 373.447 ms +- MEAN : 373.447 ms +- MAX : 373.447 ms +- MIN : 373.447 ms +- GEOMEAN : 373.447 ms +=================================== +``` diff --git a/docs/howto/how-to-build-runtime.md b/docs/howto/how-to-build-runtime.md index f4751198e..f4d7b56e0 100644 --- a/docs/howto/how-to-build-runtime.md +++ b/docs/howto/how-to-build-runtime.md @@ -1,11 +1,12 @@ # How to Build Runtime -This document is based on the system where Ubuntu Desktop Linux 18.04 LTS is installed with default settings, and can be applied in other environments without much difference. For reference, the development of our project started in the Ubuntu Desktop Linux 16.04 LTS environment. +This document is based on the system where Ubuntu Desktop Linux 20.04 LTS is installed with default settings, and can be applied in other environments without much difference. For reference, the development of our project started in the Ubuntu Desktop Linux 16.04 LTS environment. ## Build requirements If you are going to build this project, the following modules must be installed on your system: +- C & C++ compiler - CMake - Boost C++ libraries @@ -15,44 +16,30 @@ In the Ubuntu, you can easily install it with the following command. $ sudo apt-get install cmake libboost-all-dev ``` -If your linux system does not have the basic development configuration, you will need to install more packages. A list of all packages needed to configure the development environment can be found in the https://github.com/Samsung/ONE/blob/master/infra/docker/Dockerfile.1804 file. +If your linux system does not have the basic development configuration, you will need to install more packages. A list of all packages needed to configure the development environment can be found in https://github.com/Samsung/ONE/blob/master/infra/docker/focal/Dockerfile. -Here is a summary of it +Here is a summary of it for runtime and related tools ``` $ sudo apt install \ build-essential \ -clang-format-3.9 \ +clang-format-8 \ cmake \ doxygen \ git \ graphviz \ hdf5-tools \ lcov \ -libatlas-base-dev \ libboost-all-dev \ -libgflags-dev \ -libgoogle-glog-dev \ -libgtest-dev \ libhdf5-dev \ -libprotobuf-dev \ -protobuf-compiler \ -pylint \ python3 \ python3-pip \ -python3-venv \ scons \ software-properties-common \ unzip \ wget -$ mkdir /tmp/gtest -$ cd /tmp/gtest -$ cmake /usr/src/gtest -$ make -$ sudo mv *.a /usr/lib - -$ pip install yapf==0.22.0 numpy +$ pip3 install yapf==0.22.0 numpy ``` @@ -63,20 +50,28 @@ In a typical linux development environment, including Ubuntu, you can build the ``` $ git clone https://github.com/Samsung/ONE.git one $ cd one -$ make -f Makefile.template install +$ ./nnfw configure +$ ./nnfw build +$ ./nnfw install +``` + +For easy build process, we provides `Makefile.template` makefile. + +``` +$ make -f Makefile.template ``` -Unfortunately, the debug build on the x86_64 architecture currently has an error. To solve the problem, you must use gcc version 9 or higher. Another workaround is to do a release build rather than a debug build. This is not a suitable method for debugging during development, but it is enough to check the function of the runtime. To release build the runtime, add the environment variable `BUILD_TYPE=release` to the build command as follows. +To release build the runtime, add the environment variable `BUILD_TYPE=release` to the build command as follows. ``` $ export BUILD_TYPE=release -$ make install +$ make -f Makefile.template ``` Or you can simply do something like this: ``` -$ BUILD_TYPE=release make install +$ BUILD_TYPE=release make -f Makefile.template ``` The build method described here is a `native build` in which the build environment and execution environment are same. So, this command creates a runtime binary targeting the current build architecture, probably x86_64, as the execution environment. You can find the build output in the ./Product folder as follows: @@ -84,12 +79,8 @@ The build method described here is a `native build` in which the build environme ``` $ tree -L 2 ./Product ./Product -├── obj -> /home/sjlee/star/one/Product/x86_64-linux.debug/obj -├── out -> /home/sjlee/star/one/Product/x86_64-linux.debug/out -└── x86_64-linux.debug - ├── BUILD - ├── CONFIGURE - ├── INSTALL +├── out -> /home/sjlee/star/one/Product/x86_64-linux.release/out +└── x86_64-linux.release ├── obj └── out @@ -98,62 +89,74 @@ $ tree -L 2 ./Product $ tree -L 3 ./Product/out ./Product/out ├── bin -│ ├── nnapi_test -│ ├── nnpackage_run -│ ├── tflite_loader_test_tool -│ └── tflite_run +│ ├── onert_run +│ ├── tflite_comparator +│ └── tflite_run ├── include -│ ├── nnfw -│ │ ├── NeuralNetworksEx.h -│ │ ├── NeuralNetworksExtensions.h -│ │ ├── NeuralNetworks.h -│ │ ├── nnfw_experimental.h -│ │ └── nnfw.h -│ └── onert -│ ├── backend -│ ├── compiler -│ ├── exec -│ ├── ir -│ ├── misc -│ └── util +│ ├── nnfw +│ │ ├── NeuralNetworksEx.h +│ │ ├── NeuralNetworksExtensions.h +│ │ ├── NeuralNetworks.h +│ │ ├── nnfw_experimental.h +│ │ └── nnfw.h +│ └── onert +│ ├── backend +│ ├── compiler +│ ├── exec +│ ├── ir +│ └── util ├── lib -│ ├── libbackend_cpu.so -│ ├── libcircle_loader.so -│ ├── libneuralnetworks.so -│ ├── libnnfw-dev.so -│ ├── libnnfw_lib_benchmark.so -│ ├── libnnfw_lib_misc.a -│ ├── libonert_core.so -│ └── libtflite_loader.so -├── tests -│ ├── FillFrom_runner -│ ├── nnpkgs -│ │ └── FillFrom -│ └── scripts -│ ├── benchmark_nnapi.sh -│ ├── benchmark_nnpkg.sh -│ ├── common.sh -│ ├── framework -│ ├── list -│ ├── print_to_json.sh -│ ├── test-driver.sh -│ ├── test_framework.sh -│ ├── test_scheduler_with_profiling.sh -│ └── unittest.sh -├── unittest -│ ├── nnapi_gtest -│ ├── nnapi_gtest.skip -│ ├── nnapi_gtest.skip.noarch.interp -│ ├── nnapi_gtest.skip.x86_64-linux.cpu -│ ├── test_compute -│ ├── test_onert -│ ├── test_onert_backend_cpu_common -│ ├── test_onert_frontend_nnapi -│ └── tflite_test -└── unittest_standalone - └── nnfw_api_gtest - -19 directories, 36 files +│ ├── libbackend_cpu.so +│ ├── libbackend_ruy.so +│ ├── libneuralnetworks.so +│ ├── libnnfw-dev.so +│ └── libonert_core.so +├── nnapi-gtest +│ ├── nnapi_gtest +│ ├── nnapi_gtest.skip +│ └── nnapi_gtest.skip.x86_64-linux.cpu +├── test +│ ├── command +│ │ ├── nnpkg-test +│ │ ├── prepare-model +│ │ ├── unittest +│ │ └── verify-tflite +│ ├── FillFrom_runner +│ ├── list +│ │ ├── benchmark_nnpkg_model_list.txt +│ │ ├── nnpkg_test_list.armv7l-linux.acl_cl +│ │ ├── nnpkg_test_list.armv7l-linux.acl_neon +│ │ ├── nnpkg_test_list.armv7l-linux.cpu +│ │ ├── tflite_comparator.aarch64.acl_cl.list +│ │ ├── tflite_comparator.aarch64.acl_neon.list +│ │ ├── tflite_comparator.aarch64.cpu.list +│ │ ├── tflite_comparator.armv7l.acl_cl.list +│ │ ├── tflite_comparator.armv7l.acl_neon.list +│ │ ├── tflite_comparator.armv7l.cpu.list +│ │ └── tflite_comparator.x86_64.cpu.list +│ ├── models +│ │ ├── run_test.sh +│ │ └── tflite +│ ├── nnpkgs +│ │ └── FillFrom +│ └── onert-test +└── unittest + ├── ndarray_test + ├── nnfw_api_gtest + ├── nnfw_api_gtest_models + │ ├── add + │ ├── add_invalid_manifest + │ ├── add_no_manifest + │ ├── if_dynamic + │ ├── mobilenet_v1_1.0_224 + │ └── while_dynamic + ├── nnfw_lib_misc_test + ├── test_cker + ├── test_onert_core + ├── test_onert_frontend_nnapi + └── tflite_test + +26 directories, 42 files ``` @@ -173,26 +176,8 @@ inception_v3.tflite The result of running the inception_v3 model using runtime is as follows. Please consider that this is a test that simply checks execution latency without considering the accuracy of the model. ``` -$ USE_NNAPI=1 LD_LIBRARY_PATH="./Product/out/lib/:$LD_LIBRARY_PATH" ./Product/out -/bin/tflite_run ./inception_v3.tflite -nnapi function 'ANeuralNetworksModel_create' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_addOperand' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_setOperandValue' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_addOperation' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_identifyInputsAndOutputs' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_finish' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksCompilation_create' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksCompilation_finish' is loaded from './Product/out/lib/libneuralnetworks.so' -input tensor indices = [317,] -nnapi function 'ANeuralNetworksExecution_create' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksExecution_setInput' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksExecution_setOutput' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksExecution_startCompute' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksEvent_wait' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksEvent_free' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksExecution_free' is loaded from './Product/out/lib/libneuralnetworks.so' -... run 1 takes 183.895 ms -output tensor indices = [316(max:905),] +$ ./Product/out/bin/onert_run ./inception_v3.tflite +Model Filename ./inception_v3.tflite =================================== MODEL_LOAD takes 1.108 ms PREPARE takes 0.190 ms @@ -202,10 +187,8 @@ EXECUTE takes 183.895 ms - MIN : 183.895 ms - GEOMEAN : 183.895 ms =================================== -nnapi function 'ANeuralNetworksCompilation_free' is loaded from './Product/out/lib/libneuralnetworks.so' -nnapi function 'ANeuralNetworksModel_free' is loaded from './Product/out/lib/libneuralnetworks.so' ``` -Here, `USE_NNAPI=1` means that **ONE** runtime is used for model inference. If omitted, the model will be executed using Tensorflow lite, the basic framework for verification. `LD_LIBRARY_PATH="./Product/out/lib/:$LD_LIBRARY_PATH"` specifies the location of the runtime library to be used for testing. From the previous build result, you can see that it is the path to the directory where `libneuralnetworks.so` and `libonert_core.so` are located. +If you use `tflite_run` instead of `onert_run`, the model will be executed using Tensorflow lite, the basic framework for verification. From the previous build result, you can see that it is the path to the directory where `tflite_run` and `onert_run` are located. If you come here without any problems, you have all of the basic environments for runtime development. diff --git a/docs/howto/how-to-contribute.md b/docs/howto/how-to-contribute.md index fe7b140eb..7c6855427 100644 --- a/docs/howto/how-to-contribute.md +++ b/docs/howto/how-to-contribute.md @@ -34,8 +34,9 @@ This section explains the steps to create a pull request (PR). 1. Create a commit It is time to create a commit for submission once you are convinced that your contribution is - ready to go. Please include signed-off message at the end of commit message. If not, your pull - request will be **rejected** by CI. + ready to go. Please include + [signed-off message](https://github.com/Samsung/ONE/wiki/ONE-Developer's-Certificate-of-Origin) + at the end of commit message. If not, your pull request will be **rejected** by CI. 1. Check code format locally diff --git a/docs/howto/how-to-cross-build-runtime-for-aarch64.md b/docs/howto/how-to-cross-build-runtime-for-aarch64.md index b30ce348a..3e2705762 100644 --- a/docs/howto/how-to-cross-build-runtime-for-aarch64.md +++ b/docs/howto/how-to-cross-build-runtime-for-aarch64.md @@ -16,7 +16,7 @@ Use `install_rootfs.sh` script to prepare Root File System. You should have `sud $ sudo ./tools/cross/install_rootfs.sh aarch64 ``` - supports `arm`(default) and `aarch64` architecutre for now -- supports `xenial`(default), `trusty` and `bionic` release +- supports `bionic`(default) and `focal` release To see the options, ``` @@ -27,7 +27,7 @@ RootFS will be prepared at `tools/cross/rootfs/aarch64` folder. ***\* CAUTION: The OS version of rootfs must match the OS version of execution target device. On the other hand, you need to match the Ubuntu version of the development PC with the Ubuntu version of rootfs to be used for cross-build. Otherwise, unexpected build errors may occur.*** -If you are using Ubuntu 16.04 LTS, select `xenial`, if you are using Ubuntu 18.04 LTS, select `bionic`. You can check your Ubuntu code name in the following way. +If you are using Ubuntu 18.04 LTS, select `bionic`, if you are using Ubuntu 20.04 LTS, select `focal`. You can check your Ubuntu code name in the following way. ``` $ cat /etc/lsb-release @@ -44,7 +44,7 @@ If a build error occurs because the version of the development system and the ta Use `ROOTFS_DIR` to a full path to prepare at alternative path. ``` -$ ROOTFS_DIR=/home/user/rootfs/aarch64-xenial sudo -E ./tools/cross/install_rootfs.sh aarch64 +$ ROOTFS_DIR=/home/user/rootfs/aarch64-bionic sudo -E ./tools/cross/install_rootfs.sh aarch64 ``` ### Using proxy @@ -89,8 +89,8 @@ normal build and cross build as follows. ``` $ export ROOTFS_DIR=xxx ... -$ make all install # do normal build -$ TARGET_ARCH=aarch64 make all install # do cross build +$ make # do normal build +$ TARGET_ARCH=aarch64 make # do cross build ``` ### Run test diff --git a/docs/howto/how-to-cross-build-runtime-for-arm.md b/docs/howto/how-to-cross-build-runtime-for-arm.md index c17c465bf..2db4d564a 100644 --- a/docs/howto/how-to-cross-build-runtime-for-arm.md +++ b/docs/howto/how-to-cross-build-runtime-for-arm.md @@ -13,26 +13,26 @@ Use `install_rootfs.sh` script to prepare Root File System. You should have `sud ``` $ sudo ./tools/cross/install_rootfs.sh arm ``` -- supports `arm`(default) and `aarch` architecutre for now -- supports `xenial`(default) `trusty`, and `bionic` release +- supports `arm`(default) and `aarch64` architecutre for now +- supports `focal`(default), `bionic`, and `jammy` release To see the options, ``` $ ./tools/cross/install_rootfs.sh -h ``` -RootFS will be prepared at `tools/cross/rootfs/arm` folder. +RootFS will be prepared at `tools/cross/rootfs/arm` or `tools/cross/rootfs/aarch64` folder. ***\* CAUTION: The OS version of rootfs must match the OS version of execution target device. On the other hand, you need to match the Ubuntu version of the development PC with the Ubuntu version of rootfs to be used for cross-build. Otherwise, unexpected build errors may occur.*** -If you are using Ubuntu 16.04 LTS, select `xenial`, if you are using Ubuntu 18.04 LTS, select `bionic`. You can check your Ubuntu code name in the following way. +If you are using Ubuntu 20.04 LTS, select `focal`, if you are using Ubuntu 22.04 LTS, select `jammy`. You can check your Ubuntu code name in the following way. ``` $ cat /etc/lsb-release DISTRIB_ID=Ubuntu -DISTRIB_RELEASE=18.04 -DISTRIB_CODENAME=bionic -DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS" +DISTRIB_RELEASE=22.04 +DISTRIB_CODENAME=jammy +DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS" ``` If a build error occurs because the version of the development system and the target system do not match, and if you can't replace your development system for any reason, you can consider [cross-build using the docker image](how-to-build-runtime-using-prebuilt-docker-image.md). @@ -42,7 +42,7 @@ If a build error occurs because the version of the development system and the ta Use `ROOTFS_DIR` to a full path to prepare at alternative path. ``` -$ ROOTFS_DIR=/home/user/rootfs/arm-xenial sudo -E ./tools/cross/install_rootfs.sh arm +$ ROOTFS_DIR=/home/user/rootfs/arm-bionic sudo -E ./tools/cross/install_rootfs.sh arm ``` ### Using proxy @@ -61,55 +61,59 @@ for `http`, `https` and `ftp` protocol. ## Install ARM Cross Toolchain -We recommend you have g++ >= 6 installed on your system because NN generated tests require it. +We recommend you have g++ >= 6.1 installed on your system because NN generated tests require it (c++14). -- On Ubuntu 16.04 or older, follow the next steps: +### Ubuntu 20.04 LTS -``` -$ cd ~/your/path -$ wget https://releases.linaro.org/components/toolchain/binaries/7.2-2017.11/arm-linux-gnueabihf/gcc-linaro-7.2.1-2017.11-x86_64_arm-linux-gnueabihf.tar.xz -$ tar xvf gcc-linaro-7.2.1-2017.11-x86_64_arm-linux-gnueabihf.tar.xz -$ echo 'export PATH=~/your/path/gcc-linaro-7.2.1-2017.11-x86_64_arm-linux-gnueabihf/bin:$PATH' >> ~/.bashrc -``` +On Ubuntu 20.04 LTS, you can install using `apt-get`. -- On Ubuntu 18.04 LTS, you can install using `apt-get`. -Choose g++ version whatever you prefer: 6, 7 or 8. +Choose g++ version whatever you prefer: 9 (default) or 10. We are officially testing on default g++ version, +so we don't confirm build on different version. ``` -$ sudo apt-get install g++-{6,7,8}-arm-linux-gnueabihf +$ sudo apt-get install g++-{9,10}-arm-linux-gnueabihf ``` -Make sure you get `libstdc++.so` updated on your target with your new toolchain's corresponding one. +If you select specific version, update symbolic link for build toolchain. -For example, if you installed gcc-linaro-7.2.1-2017.11 above, do +Otherwise, you should set your custom cmake crossbuild toolchain. You can find cmake toolchain files in `infra/nnfw/cmake/buildtool/cross/`. ``` -$ wget https://releases.linaro.org/components/toolchain/binaries/7.2-2017.11/arm-linux-gnueabihf/runtime-gcc-linaro-7.2.1-2017.11-arm-linux-gnueabihf.tar.xz -$ tar xvf runtime-gcc-linaro-7.2.1-2017.11-arm-linux-gnueabihf.tar.xz +$ update-alternatives --install /usr/bin/arm-linux-gnueabihf-gcc arm-linux-gnueabihf-gcc /usr/bin/arm-linux-gnueabihf-gcc-10 80 \ + --slave /usr/bin/arm-linux-gnueabihf-g++ arm-linux-gnueabihf-g++ /usr/bin/arm-linux-gnueabihf-g++-10 \ + --slave /usr/bin/arm-linux-gnueabihf-gcov arm-linux-gnueabihf-gcov /usr/bin/arm-linux-gnueabihf-gcov-10 ``` -Then, copy `libstdc++.so.6.0.24` into `/usr/lib/arm-linux-gnueabihf`, and update symbolic links on your device. +### Ubuntu 22.04 LTS + +Same with Ubuntu 20.04 LTS. (except g++ version) ## Build and install ARM Compute Library -Mostly you only need once of ACL build. +Mostly you only need once of ACL (ARM Compute Library) build. + +To build ACL, you need to install scons + +``` +$ sudo apt-get install scons +``` -ACL will be automatically installed in `externals/acl` when you build runtime without any changes. +ACL source will be automatically installed in `externals/ARMCOMPUTE` when you build runtime without any changes. -You can check ACL source information in `cmake/packages/ARMComputeSourceConfig.cmake` +You can check ACL source information in `infra/cmake/packages/ARMComputeSourceConfig.cmake` -## Cross build for ARM +## Cross build for ARM by using Makefile.template Give `TARGET_ARCH` variable to set the target architecture. If you used `ROOTFS_DIR` to prepare in alternative folder, you should also give this to makefile. ``` -$ CROSS_BUILD=1 TARGET_ARCH=armv7l make all install +$ CROSS_BUILD=1 TARGET_ARCH=armv7l make -f Makefile.template # If ROOTFS_DIR is in alternative folder $ ROOTFS_DIR=/path/to/your/rootfs/arm \ -CROSS_BUILD=1 TARGET_ARCH=armv7l make all install +CROSS_BUILD=1 TARGET_ARCH=armv7l make ``` You can also omit the `CROSS_BUILD=1` option if you explicitly pass `ROOTFS_DIR`. In that case, if @@ -120,15 +124,23 @@ normal build and cross build as follows. ``` $ export ROOTFS_DIR=xxx ... -$ make all install # do normal build -$ TARGET_ARCH=armv7l make all install # do cross build +$ make -f Makefile.template # do normal build +$ TARGET_ARCH=armv7l make -f Makefile.template # do cross build ``` +Makefile.template will pass crossbuild toolchain setting to cmake automatically by parsing variables. + ### Run test -To run and test the cross-compiled runtime, you need to copy the compiled output to the target device of the architecture in which it is executable. +To run and test the cross-compiled runtime, you need to install library packages and copy the compiled output to the target device of the architecture in which it is executable. + +1. Install hdf5 and boost library package + +``` +$ sudo apt install libhdf5-dev libboost-system-dev libboost-program-options-dev +``` -1. Copy all artifacts under the `./Product` folder to the target device, Odroid-XU4 for example, as a whole. +2. Copy all artifacts under the `./Product/armv7l-linux.<BUILD_TYPE>` folder to the target device, Odroid-XU4 for example, as a whole. ``` $ ssh odroid mkdir -p one/Product @@ -143,19 +155,17 @@ test-driver.sh ... ``` -2. Log in to the target device, go to the copied path, and reestore the symbolic link settings of the `Product` directory. +3. Log in to the target device, go to the copied path, and reestore the symbolic link settings of the `Product` directory. ``` $ ssh odroid sjlee@odroid's password: ... $ cd ~/one/Product -$ ln ${PWD}/armv7l-linux.debug/obj obj $ ln ${PWD}/armv7l-linux.debug/out out $ cd .. $ ls -la Product drwxrwxr-x 5 sjlee sjlee 4096 Jun 4 20:55 armv7l-linux.debug -lrwxrwxrwx 1 sjlee sjlee 51 Jun 4 20:54 obj -> /home/sjlee/one/Product/armv7l-linux.debug/obj lrwxrwxrwx 1 sjlee sjlee 51 Jun 4 20:55 out -> /home/sjlee/one/Product/armv7l-linux.debug/out ``` diff --git a/docs/howto/how-to-introduce-a-new-operation-into-runtime.md b/docs/howto/how-to-introduce-a-new-operation-into-runtime.md index ab449c4be..9ab498783 100644 --- a/docs/howto/how-to-introduce-a-new-operation-into-runtime.md +++ b/docs/howto/how-to-introduce-a-new-operation-into-runtime.md @@ -24,7 +24,6 @@ onert support the operation. - [acl_cl](#acl_cl-1) - [acl_neon](#acl_neon-1) - [cpu](#cpu-1) - - [TensorRegister (in some cases)](#tensorregister-in-some-cases) - [ConstantInitializer (in some cases)](#constantinitializer-in-some-cases) - [cpu](#cpu-2) - [Samples (to be updated)](#samples-to-be-updated) @@ -179,12 +178,12 @@ void Dumper::visit(const Select &node) 5. Add code for shape inference - ONE runtime tries to calculate shapes and allocate memory during compilation time. For some calculations of output shapes that cannot be done during compilation time, ONE runtime will calculate shapes and allocate memory during execution time. - Calculation of shapes during compilation time is called _static shape inference_ and calculation of shapes during execution time is called _dynamic shape inference_. -- [`StaticShapeInference.h`](`/runtime/onert/compiler/StaticShapeInference.h`) +- [`StaticShapeInferer.h`](`/runtime/onert/compiler/StaticShapeInferer.h`) ```CPP void visit(const ir::operation::Select &op) override; ``` -- [`StaticShapeInference.cc`](/runtime/onert/core/src/compiler/StaticShapeInference.cc) +- [`StaticShapeInferer.cc`](/runtime/onert/core/src/compiler/StaticShapeInferer.cc) ```CPP void StaticShapeInferer::visit(const ir::operation::Select &op) { @@ -229,7 +228,7 @@ void DynamicShapeInferer::visit(const ir::operation::Select &op) ir::Shape new_shape = shape_inference::inferSelectShape(input_cond_shape, input_true_shape, input_false_shape); - dynamicTensorManagerOf(output)->applyShape(output_ind, new_shape); + output->applyShape(new_shape); } ``` @@ -420,51 +419,28 @@ void visit(const ir::operation::Select &) override; ```cpp void KernelGenerator::visit(const ir::operation::Select &node) { - const auto output_index{node.getOutputs().at(ir::operation::Select::Output::OUTPUT)}; - const auto cond_index{node.getInputs().at(ir::operation::Select::Input::COND)}; - const auto input1_index{node.getInputs().at(ir::operation::Select::Input::INPUT1)}; - const auto input2_index{node.getInputs().at(ir::operation::Select::Input::INPUT2)}; - - const auto output_backend_descr = ::onert::backend::cpu::kernel::getTensorDescriptor( - _ctx.at(output_index), _current_op_seq_layout); - const auto cond_backend_descr = ::onert::backend::cpu::kernel::getTensorDescriptor( - _ctx.at(cond_index), _current_op_seq_layout); - const auto input1_backend_descr = ::onert::backend::cpu::kernel::getTensorDescriptor( - _ctx.at(input1_index), _current_op_seq_layout); - const auto input2_backend_descr = ::onert::backend::cpu::kernel::getTensorDescriptor( - _ctx.at(input2_index), _current_op_seq_layout); + const auto output_index{node.getOutputs().at(0)}; + const auto condition_index{node.getInputs().at(ir::operation::Select::Input::CONDITION)}; + const auto true_index{node.getInputs().at(ir::operation::Select::Input::INPUT_TRUE)}; + const auto false_index{node.getInputs().at(ir::operation::Select::Input::INPUT_FALSE)}; - auto output_alloc = _tensor_builder->at(output_index).get(); - auto cond_alloc = _tensor_builder->at(cond_index).get(); - auto input1_alloc = _tensor_builder->at(input1_index).get(); - auto input2_alloc = _tensor_builder->at(input2_index).get(); + auto output_tensor = _tensor_reg->getPortableTensor(output_index); + auto condition_tensor = _tensor_reg->getPortableTensor(condition_index); + auto true_tensor = _tensor_reg->getPortableTensor(true_index); + auto false_tensor = _tensor_reg->getPortableTensor(false_index); - auto fn = std::make_unique<::onert::backend::cpu::kernel::SelectLayer>(); + auto fn = std::make_unique<ops::SelectLayer>(); - fn->configure(cond_alloc->buffer(), cond_backend_descr, input1_alloc->buffer(), - input1_backend_descr, input2_alloc->buffer(), input2_backend_descr, - output_alloc->buffer(), output_backend_descr); + fn->configure(condition_tensor, true_tensor, false_tensor, output_tensor); - _execution_builder->append(std::move(fn)); + _return_fn = std::move(fn); } ``` -### TensorRegister (in some cases) - -This component registers tensors. Most tensors will be automatically registered internally. There -are some exceptions, however, where additional implementations are required. It is the case when a -tensor is treated unusually in its backend. - -The kernel of some operation has weights in `HWIO` as layout(data format) in case of that input's -layout is `NHWC`. And, for `NCHW`, weights is `OIHW`. But TFLite model has weigths, `OHWI` for -`NHWC` and `OIHW` for `NCHW`. Therefore, to register the appropriate tensor on the backend, you have -to implement it additionally. - ### ConstantInitializer (in some cases) This component registers function initializing constant tensors and initialize constant tensor -layer. This is similar to TensorRegister. Most tensors will be automatically registered internally. -And there are some exceptions. +layer. Most tensors will be automatically registered internally. And there are some exceptions. #### cpu diff --git a/docs/howto/how-to-use-specific-backend.md b/docs/howto/how-to-use-specific-backend.md new file mode 100644 index 000000000..a839777a0 --- /dev/null +++ b/docs/howto/how-to-use-specific-backend.md @@ -0,0 +1,40 @@ +# How to Use Specific Backend during Inference + +ONE runtime has many ways to use specific backend during inference + +## Using NNFW API + +### [nnfw_set_available_backends](https://github.com/Samsung/ONE/blob/c46ddc04abdb58323fbd38389e6927f003bfaea1/runtime/onert/api/include/nnfw.h#L458) +- Multiple backends can be set and they must be separated by a semicolon (ex: "acl_cl;cpu"). +- For each backend string, `libbackend_{backend}.so` will be dynamically loaded during nnfw_prepare. +- Among the multiple backends, the 1st element is used as the default backend. + +### [nnfw_set_op_backend](https://github.com/Samsung/ONE/blob/c46ddc04abdb58323fbd38389e6927f003bfaea1/runtime/onert/api/include/nnfw.h#L476) +- The backend for op has higher priority than available backends specified by nnfw_set_available_backends. + +## Using Environment Variable + +### 1. BACKENDS +- Same as `nnfw_set_available_backends` +- Example +```bash +BACKENDS=cpu ./Product/out/bin/onert_run ... +``` + +### 2. OP_BACKEND_[OP_TYPE] +- Same as `nnfw_set_op_backend` +- Set backend for specific operator type +- Example + - Execute `Conv2D` operator on ruy backend and others on cpu backend +```bash +OP_BACKEND_Conv2D=ruy BACKENDS="cpu;ruy" ./Product/out/bin/onert_run ... +``` + +### 3. OP_BACKEND_MAP +- Set backend for specific operator by its index +- Format : `<op_id>=<backend>;<op_id>=<backend>...` +- Example + - Execute `operator 10` on `acl_cl` backend and others on `acl_neon` backend +```bash +OP_BACKEND_MAP="10=acl_cl" BACKENDS="acl_neon;acl_cl" ./Product/out/bin/onert_run ... +``` diff --git a/docs/howto/index.rst b/docs/howto/index.rst index d04224ed6..faeedbfaa 100644 --- a/docs/howto/index.rst +++ b/docs/howto/index.rst @@ -10,18 +10,22 @@ How To :maxdepth: 2 :caption: Contents: - ./how-to-add-a-new-operation.md ./how-to-build-compiler.md ./how-to-build-package.md ./how-to-build-runtime.md + ./how-to-build-runtime-tizen-gbs-rpi4.md ./how-to-build-runtime-using-prebuilt-docker-image.md - ./how-to-cross-build-runtime-for-arm.md ./how-to-cross-build-runtime-for-aarch64.md ./how-to-cross-build-runtime-for-android.md - ./how-to-contribute.md - ./how-to-make-an-application-with-runtime.md - ./how-to-remote-debugging-with-visual-studio-code.md + ./how-to-cross-build-runtime-for-arm.md ./how-to-run-package.md + ./how-to-make-an-application-with-runtime.md ./how-to-use-api.md - ./how-to-use-nnfw-api.md ./how-to-use-nnapi-binding.md + ./how-to-use-nnfw-api.md + ./how-to-use-specific-backend.md + ./how-to-contribute.md + ./how-to-remote-debugging-with-visual-studio-code.md + ./how-to-add-a-new-operation.md + ./how-to-introduce-a-new-operation-into-compiler.md + ./how-to-introduce-a-new-operation-into-runtime.md diff --git a/docs/release/1.10/index.rst b/docs/release/1.10/index.rst new file mode 100644 index 000000000..28325e685 --- /dev/null +++ b/docs/release/1.10/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.10 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.10.0.md diff --git a/docs/release/1.10/release-note-1.10.0.md b/docs/release/1.10/release-note-1.10.0.md new file mode 100644 index 000000000..e6423be6f --- /dev/null +++ b/docs/release/1.10/release-note-1.10.0.md @@ -0,0 +1,25 @@ +# Release Note 1.10.0 + +## ONE Compiler + +### Compiler supports more operations + +- Dequantize, UnidirectionalSequenceLSTM + +### Changes + +- New `--fold_dequantize` option in `one-optimize` +- New `--fuse_add_with_tconv` option in `one-optimize` +- Support `int16` quantization in `one-quantize` +- Test scripts are added for basic testing of one-cmds command line tools +- Bug fixes for one-cmds command line tools + + +## ONE Runtime + +### Runtime backend operation support + - ACL-CL backend: OneHot + - CPU backend: FullyConnected for Float32 16x1 Block Sparsity + +### Optimization + - Speed up for ReduceSum, StrideSlice and BinaryArithmetic in CPU backend diff --git a/docs/release/1.11/index.rst b/docs/release/1.11/index.rst new file mode 100644 index 000000000..d08c88b29 --- /dev/null +++ b/docs/release/1.11/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.11 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.11.0.md diff --git a/docs/release/1.11/release-note-1.11.0.md b/docs/release/1.11/release-note-1.11.0.md new file mode 100644 index 000000000..4b0bf4e40 --- /dev/null +++ b/docs/release/1.11/release-note-1.11.0.md @@ -0,0 +1,21 @@ +# Release Note 1.11.0 + +## ONE Compiler + +### Compiler supports more operations +- MaxPoolWithArgMax by CustomOp + +### Changes +- `one-build` command added as representative command +- one-cmds are now revised to python script and supports configuration file as input parameters +- added `rawdata2hdf5` tool to help creating input datasets for calibration +- added more optimization passes in `one-optimize`; `fuse_preactivation_batchnorm`, `make_batchnorm_gamma_positive` and `fuse_activation_function` + +## ONE Runtime + +### Runtime backend operation supports more operations and types + +- CPU backend + - float: AddN, Floor, UniDirectionalSequenceLSTM + - uint8: Dequantize, Rank + - int8: Dequantize, Rank, Shape diff --git a/docs/release/1.12/index.rst b/docs/release/1.12/index.rst new file mode 100644 index 000000000..61465ee7b --- /dev/null +++ b/docs/release/1.12/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.12 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.12.0.md diff --git a/docs/release/1.12/release-note-1.12.0.md b/docs/release/1.12/release-note-1.12.0.md new file mode 100644 index 000000000..1f13bc4ce --- /dev/null +++ b/docs/release/1.12/release-note-1.12.0.md @@ -0,0 +1,28 @@ +# Release Note 1.12.0 + +## ONE Compiler + +### Compiler Frontend + +- Add optimization pass: ReplaceMulAddWithDepthwiseConvPass, SubstitutePackToReshape, RemoveRedundantTranspose, ShuffleWeightTo16x1Float32Pass +- Add quantization for InstanceNorm. +- Fix bug of `one-import-bcq` command for `--v1`, `--v2` arguments. +- Fix FuseBCQPass to work with inter-subgraphs in the model file and minor BCQ related optimizations. + +## ONE Runtime + +### Runtime backend operation supports more operations and types + +- CPU backend + - Concat: int8 + - DepthToSpace: float, uint8, int8 + - LeakyRelu: float +- ACL-CL backend + - ArgMin: float, uint8, int8 +- ACL-NEON backend + - ArgMax: int8 + - ArgMin: float, uint8, int8 + +### nnpackage defines configuration file + +- Allow users to set configuration variable via conf file. For more information, See [nnpackage spec](../../../nnpackage/spec) diff --git a/docs/release/1.13/index.rst b/docs/release/1.13/index.rst new file mode 100644 index 000000000..5601f743b --- /dev/null +++ b/docs/release/1.13/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Wed Jan 14 16:48:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.13 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.13.0.md diff --git a/docs/release/1.13/release-note-1.13.0.md b/docs/release/1.13/release-note-1.13.0.md new file mode 100644 index 000000000..31e3a0d77 --- /dev/null +++ b/docs/release/1.13/release-note-1.13.0.md @@ -0,0 +1,12 @@ +# Release Note 1.13.0 + +## ONE Compiler + +### Compiler Frontend + +- Add optimization pass: ConvertNCHWToNHWC, FoldSparseToDensePass, FuseBatchNormWithConvPass, ForwardReshapeToUnaryOpPass, RemoveUnnecessarySlicePass, RemoveUnnecessarySplitPass, RemoveUnnecessaryReshapePass, RemoveRedundantReshape, SubstituteTransposeToReshapePass, SubstituteSqueezeToReshapePass, +- Support more operators: FAKE_QUANT +- Enhancements: Support auto generated random input for record-minmax (for better quantization testing) +- Changes: `--all` option to `--O1` in circle2circle(and one-optimize) +- Fixes: `tf2tfliteV2` accept input shapes `--v2` option, lots of fixes for increase test coverage +- Experimental: Compile ONNX models to circle diff --git a/docs/release/1.14/index.rst b/docs/release/1.14/index.rst new file mode 100644 index 000000000..558229e7f --- /dev/null +++ b/docs/release/1.14/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu Mar 18 16:47:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.14 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.14.0.md diff --git a/docs/release/1.14/release-note-1.14.0.md b/docs/release/1.14/release-note-1.14.0.md new file mode 100644 index 000000000..7c567b0fe --- /dev/null +++ b/docs/release/1.14/release-note-1.14.0.md @@ -0,0 +1,10 @@ +# Release Note 1.14.0 + +## ONE Compiler + +### Compiler Frontend + +- `one-codegen` interface now distinguishes own arguments from backend's. +- Adds `RemoveUnnecessaryStridedSlice` optimization pass. +- Introduces experimental support for generating profile data. + - Adds `--generate_profile_data` option to `one-optimize`, `one-quantize`. diff --git a/docs/release/1.15/index.rst b/docs/release/1.15/index.rst new file mode 100644 index 000000000..de2194742 --- /dev/null +++ b/docs/release/1.15/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu Mar 18 16:47:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.15 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.15.0.md diff --git a/docs/release/1.15/release-note-1.15.0.md b/docs/release/1.15/release-note-1.15.0.md new file mode 100644 index 000000000..106cefd42 --- /dev/null +++ b/docs/release/1.15/release-note-1.15.0.md @@ -0,0 +1,42 @@ +# Release Note 1.15.0 + +## ONE Compiler + +### Compiler Frontend + +- Support more Ops for quantization +- Fix `record-minmax` tool for bool type, NaN values +- Fix `one-cmds` test scripts +- Remove `stdex` module +- `arser` supports short option + + +## ONE Runtime + +### Runtime backend supports more operations and types + +- CPU backend + - Add: int8 + - AvgPool2d: int8 + - Conv2D: int8 + - DepthwiseConv2D: int8 + - Div: uint8 + - Elu: float + - ExpandDims: int8 + - LogicalAnd: boolean + - Maximum: uint8 + - MaxPool2D: int8 + - Minimum: uint8 + - Mul: int8 + - Pad: int8 + - PadV2: int8 + - Quantize: uint8, int8 + - Reshape: int8 + - Resizebiliear: int8 + - Softmax: int8 + - Squeeze: int8 + - Sub: int8 + +### ARM Compute Library Update + +- ONERT uses Compute Library v21.02 diff --git a/docs/release/1.16/index.rst b/docs/release/1.16/index.rst new file mode 100644 index 000000000..08f4c3e65 --- /dev/null +++ b/docs/release/1.16/index.rst @@ -0,0 +1,14 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 20 12:56:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.16 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.16.0.md + ./release-note-1.16.1.md diff --git a/docs/release/1.16/release-note-1.16.0.md b/docs/release/1.16/release-note-1.16.0.md new file mode 100644 index 000000000..be492033c --- /dev/null +++ b/docs/release/1.16/release-note-1.16.0.md @@ -0,0 +1,17 @@ +# Release Note 1.16.0 + +## ONE Compiler + +### Compiler Frontend + +- Enable `PadV2` in luci-interpreter and quantization +- Provide `circle-tensordump`, `circledump` as a development tool +- Provide `luci-eval-driver` as test tool +- Enable `STRING` type as constant values in CircleConst +- Fix CircleCustom may have 0 input, 0 output +- Enable debian package generation +- More optimization pass + - Min(6)+ReLU to ReLU6 + - Remove FakeQuant Op +- Experimental support of ONNX upgraded to version 1.8.0 with additional patch +- Fix bugs where one-cmds' config file didn't evaluate boolean properly diff --git a/docs/release/1.16/release-note-1.16.1.md b/docs/release/1.16/release-note-1.16.1.md new file mode 100644 index 000000000..e357b9589 --- /dev/null +++ b/docs/release/1.16/release-note-1.16.1.md @@ -0,0 +1,7 @@ +# Release Note 1.16.1 + +## ONE Compiler + +### Compiler Frontend + +- Extends the point where `one-codegen` finds backends. diff --git a/docs/release/1.17/index.rst b/docs/release/1.17/index.rst new file mode 100644 index 000000000..c2eda6020 --- /dev/null +++ b/docs/release/1.17/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 20 12:56:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.17 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.17.0.md diff --git a/docs/release/1.17/release-note-1.17.0.md b/docs/release/1.17/release-note-1.17.0.md new file mode 100644 index 000000000..20f8f7ec5 --- /dev/null +++ b/docs/release/1.17/release-note-1.17.0.md @@ -0,0 +1,24 @@ +# Release Note 1.17.0 + +## ONE Compiler + +### Compiler Frontend + +- More optimization pass + - Remove Quant-Dequant sequence + - Replace Sub with Add + - Substitute StridedSlice to Reshape + - Fuse Mean with Mean + - Fuse Transpose with Mean + - Substitute PadV2 to Pad +- Add new InstanceNorm pattern in `FuseInstanceNormPass` +- Add verbose option +- Introduce `onecc` driver to `one-cmds` +- Introduce `one-profile` driver to `one-cmds` + +## ONE Runtime + +### gpu_cl backend added + +- New backend(gpu_cl) added. This backend exploits tensorflow lite's gpu delegate. +- This backend supports the following operations : Add, Convolution, Depthwise Convolution, Pooling, Reshape, Relu, Softmax diff --git a/docs/release/1.18/index.rst b/docs/release/1.18/index.rst new file mode 100644 index 000000000..71c46585a --- /dev/null +++ b/docs/release/1.18/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Fri Oct 20 15:20:12 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.18 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.18.0.md diff --git a/docs/release/1.18/release-note-1.18.0.md b/docs/release/1.18/release-note-1.18.0.md new file mode 100644 index 000000000..a10f10e37 --- /dev/null +++ b/docs/release/1.18/release-note-1.18.0.md @@ -0,0 +1,11 @@ +# Release Note 1.18.0 + +## ONE Compiler + +### Compiler Frontend + +- More optimization pass + - Fold DepthwiseConv2D + - Substitute SplitV to Split + - Expand BroadCast Const + - Force QuantParam diff --git a/docs/release/1.19/index.rst b/docs/release/1.19/index.rst new file mode 100644 index 000000000..c80782ce8 --- /dev/null +++ b/docs/release/1.19/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Wed Nov 10 15:21:13 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.19 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.19.0.md diff --git a/docs/release/1.19/release-note-1.19.0.md b/docs/release/1.19/release-note-1.19.0.md new file mode 100644 index 000000000..e63d8706c --- /dev/null +++ b/docs/release/1.19/release-note-1.19.0.md @@ -0,0 +1,8 @@ +# Release Note 1.19.0 + +## ONE Compiler + +### Compiler Frontend + +- `circle-quantizer` supports input/output type option +- Introduce configuration file for optimization options diff --git a/docs/release/1.20/index.rst b/docs/release/1.20/index.rst new file mode 100644 index 000000000..082d867f3 --- /dev/null +++ b/docs/release/1.20/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Tue Apr 26 10:18:12 2022. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.20 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.20.0.md diff --git a/docs/release/1.20/release-note-1.20.0.md b/docs/release/1.20/release-note-1.20.0.md new file mode 100644 index 000000000..2c75e06af --- /dev/null +++ b/docs/release/1.20/release-note-1.20.0.md @@ -0,0 +1,34 @@ +# Release Note 1.20.0 + +## ONE Compiler + +### Compiler Frontend + +- luci-interpreter supports multiple kernels with PAL layer including Cortext-M +- luci-interpreter supports integer tensor for partly kernels +- luci import support constant without coping to reduce memory for luci-interpreter +- Reduce duplicate codes to package released modules +- Limited support for ONNX LSTM/RNN unrolling while importing +- Limited support for ARM32 cross build +- Support new operator: SVDF +- New virtual CircleVariable to support tensor with variable +- Support quantization of BatchMatMul Op +- Support mixed(UINT8 + INT16) quantization +- Support backward propagation of quantization parameters +- Upgrade default python to version 3.8 +- Support TensorFlow 2.8.0, ONNX-TF 1.10.0, ONNX 1.11.0 +- Upgrade circle schema to follow tflite schema v3b +- Refactor to mio-tflite280, mio-circle04 with version and helpers methods +- Use one flatbuffers 2.0 version +- Drop support for TensorFlow 1.x +- Fix for several bugs, performance enhancements, and typos + +## ONE Runtime + +### Introduce TRIX backend +- TRIX backend supports trix binary with NHWC layout +- TRIX backend supports trix binary with input/output of Q8 and Q16 type + +### API supports new data type +- Symmetric Quantized int16 type named "NNFW_TYPE_TENSOR_QUANT16_SYMM_SIGNED" + diff --git a/docs/release/1.21/index.rst b/docs/release/1.21/index.rst new file mode 100644 index 000000000..587065f56 --- /dev/null +++ b/docs/release/1.21/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Wed Sep 06 12:18:12 2022. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.21 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.21.0.md diff --git a/docs/release/1.21/release-note_1.21.0.md b/docs/release/1.21/release-note_1.21.0.md new file mode 100644 index 000000000..49bf074b6 --- /dev/null +++ b/docs/release/1.21/release-note_1.21.0.md @@ -0,0 +1,35 @@ +# Release Note 1.21.0 + +## ONE Compiler + +- Support unrolling of LSTM and RNN Ops in `one-import-onnx` tool +- Introduced new tools `one-infer`, `circle-operator`, `circle-interpreter` +- Introduced `Workflow`(WIP) in `one-cmds` +- New option `quant_config` in `one-quantize` +- New option `fake_quantize` in `one-quantize` +- More Ops supported: Densify +- More Ops for quantization: ReduceMax +- More Ops for mixed-precision quantization (MPQ): LeakyRelu, Neg, Relu6, Squeeze +- More Ops for `convert_nchw_to_nhwc` option: LogSoftmax, ReduceMax, SplitV, Softmax +- New optimization options in `one-optimize`: `replace_non_const_fc_with_bmm`, `resolve_customop_splitv`, `fold_densify` +- Improved reshape elimination in `convert_nchw_to_nhwc` option. +- Support fusion of Channel-wise Add + Relu with TConv +- Support negative axis in ArgMin/Max +- Show errors for unrecognized options in `one-optimize` +- Fix shape inference for `StridedSlice` +- Fix FuseBatchNormWithTConvPass to support TConv with bias +- Deprecate `--O1` option in `circle2circle` +- Support gcc-11 +- Support limited Float16 for kernels constants with dequantization to Float32 + +## ONE Runtime + +### Basic Multimodel nnpackage +- Runtime supports to run nnpackage with two models + +### Channel Wise Quantization on Conv2D and Depthwise Conv2D +- Conv2D and Depthwise Conv2D supports per-channel quantization of uint8 type. + +### Batch Execution with TRIX backend +- TRIX backend supports batch execution which run in parallel with multicore + diff --git a/docs/release/1.22/index.rst b/docs/release/1.22/index.rst new file mode 100644 index 000000000..e3c330df6 --- /dev/null +++ b/docs/release/1.22/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Fri Mar 24 14:03:12 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.22 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.22.0.md diff --git a/docs/release/1.22/release-note-1.22.0.md b/docs/release/1.22/release-note-1.22.0.md new file mode 100644 index 000000000..2e6dee74a --- /dev/null +++ b/docs/release/1.22/release-note-1.22.0.md @@ -0,0 +1,12 @@ +# Release Note 1.22.0 + +## ONE Compiler + +- Introduce new optimization options: `unroll_unidirseqlstm`, `forward_transpose_op`, `fold_fully_connected`, `fuse_prelu` +- Support more Ops for fake quantization: `Depth2Space`, `Space2Depth`, `Pack`, `Unpack`, `Abs` +- Support more Ops for quantization: `Abs`, `ReduceProd` +- Introduce _visq_ tool for quantization error visualization +- Introduce _Environment_ section into configuration file +- Improve speed of `convert_nchw_to_nhwc` option +- Support `Add`, `Mul` of index-type (int32, int64) tensors in _one-quantize_ +- Support ubuntu 20.04 diff --git a/docs/release/1.22/release-note-1.22.1.md b/docs/release/1.22/release-note-1.22.1.md new file mode 100644 index 000000000..c76eb2688 --- /dev/null +++ b/docs/release/1.22/release-note-1.22.1.md @@ -0,0 +1,9 @@ +# Release Note 1.22.1 + +## ONE Runtime + +### Multimodel nnpackage + +- Runtime supports to run nnpackage with 3 or more models +- Runtime supports to run multimodel nnpackage with multiple subgraphs +- Runtime supports type casting when tensor's data type between edge is different diff --git a/docs/release/1.23/index.rst b/docs/release/1.23/index.rst new file mode 100644 index 000000000..ae29d5226 --- /dev/null +++ b/docs/release/1.23/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 18 19:07:17 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.23 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.23.0.md diff --git a/docs/release/1.23/release-note-1.23.0.md b/docs/release/1.23/release-note-1.23.0.md new file mode 100644 index 000000000..b5a3d1b9b --- /dev/null +++ b/docs/release/1.23/release-note-1.23.0.md @@ -0,0 +1,8 @@ +# Release Note 1.23.0 + +## ONE Compiler + +- Support more Op(s): GeLU +- Support more option(s): `--fuse-gelu` +- Support multiple backends compilation with a single configuration file +- Upgrade Circle schema to 0.5 diff --git a/docs/release/1.24/index.rst b/docs/release/1.24/index.rst new file mode 100644 index 000000000..fa1698820 --- /dev/null +++ b/docs/release/1.24/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu Jul 18 14:08:15 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.24 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.24.0.md diff --git a/docs/release/1.24/release-note-1.24.0.md b/docs/release/1.24/release-note-1.24.0.md new file mode 100644 index 000000000..9020da8aa --- /dev/null +++ b/docs/release/1.24/release-note-1.24.0.md @@ -0,0 +1,9 @@ +# Release Note 1.24.0 + +## ONE Compiler + +- Introduce _one-import-onnx_ extension interface +- _onecc_ supports profiling of multiple backends with a single cfg file +- Enable more Quantize operator: FloorMod, Squeeze +- _visq_ supports multi-out nodes +- _onecc_ introduces `dynamic_batch_to_single_batch option` option. diff --git a/docs/release/1.25/index.rst b/docs/release/1.25/index.rst new file mode 100644 index 000000000..1965b42e8 --- /dev/null +++ b/docs/release/1.25/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Wed Sep 06 12:18:12 2022. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.25 +==== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.25.0.md diff --git a/docs/release/1.25/release-note_1.25.0.md b/docs/release/1.25/release-note_1.25.0.md new file mode 100644 index 000000000..3d62c6bde --- /dev/null +++ b/docs/release/1.25/release-note_1.25.0.md @@ -0,0 +1,17 @@ +# Release Note 1.25.0 + +## ONE Runtime + +- Support ubuntu 20.04 + +### CPU Backend Operation +- CPU backend supports per-channel hybrid quantization of int8 type weight and float activation. (TFLite's dynamic range quantization) + +### On-device Quantization +- _onert_ supports new experimental API for on-device quantization. +- As the 1st step, _onert_ supports per-channel hybrid quantization of int8/int16 type weight and float activation. +- API requires file path to export quantized model. + +### Minmax Recorder +- _onert_` support minmax recording of each layer as experimental feature. It is not supported by API yet. +- Output file format is HDF5. (File format may change later). diff --git a/docs/release/1.5/index.rst b/docs/release/1.5/index.rst new file mode 100644 index 000000000..6e7810280 --- /dev/null +++ b/docs/release/1.5/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.5 +=== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.5.0.md diff --git a/docs/release/1.6/index.rst b/docs/release/1.6/index.rst new file mode 100644 index 000000000..3f380b5f8 --- /dev/null +++ b/docs/release/1.6/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.6 +=== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.6.0.md diff --git a/docs/release/1.7/index.rst b/docs/release/1.7/index.rst new file mode 100644 index 000000000..234e4b6ec --- /dev/null +++ b/docs/release/1.7/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.7 +=== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.7.0.md diff --git a/docs/release/1.7/release-note-1.7.0.md b/docs/release/1.7/release-note-1.7.0.md new file mode 100644 index 000000000..c1a4f50b2 --- /dev/null +++ b/docs/release/1.7/release-note-1.7.0.md @@ -0,0 +1,46 @@ +## Feature Highlights + +- **ONE** Compiler + - Compiler supports more operations + - New command line interface for user interface consistancy +- **ONE** Runtime + - Runtime CPU backend supports more operations + - Runtime CPU backend supports more quant8 operations + - API changes + - New optimization + +## ONE Compiler + +### Compiler supports more operations + +- MatrixDiag, MatrixSetDiag, ReverseSequence, ReverseV2, SegmentSum, SelectV2, SparseToDense, Where + +### New command line interface for user interface consistancy + +- one-import: imports conventional model files to circle + - one-import-tf: imports TensorFlow model to circle + - one-import-tflite: imports TensorFlow lite model to circle +- one-optimize: circle optimize command +- one-quantize: circle quantize command + - supports float32 to uint8, layer wise (for Conv series) +- one-pack: package command +- one-prepare-venv: prepares python virtual environment for importing TensorFlow model +- one-codegen: backend(if available) code generator + +## ONE Runtime + +### Runtime CPU backend supports more operations + +- LogSoftmax, SpaceToBatchND + +### Runtime CPU backend supports more quant8 operations + +- Logistic, Mul, Tanh, SpaceToBatchND, Transpose, Sub, Max, Min, Less, Greater, GreaterEqual, LessEqual, Equal, NotEqual + +### API changes + +- Introduce basic asynchronous execution API + +### New optimization + +- Remove dynamic tensor overhead from static models diff --git a/docs/release/1.8/index.rst b/docs/release/1.8/index.rst new file mode 100644 index 000000000..022e00ba5 --- /dev/null +++ b/docs/release/1.8/index.rst @@ -0,0 +1,13 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.8 +=== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.8.0.md diff --git a/docs/release/1.9/index.rst b/docs/release/1.9/index.rst new file mode 100644 index 000000000..82f0e1210 --- /dev/null +++ b/docs/release/1.9/index.rst @@ -0,0 +1,14 @@ +.. ONE documentation master file, created by + sphinx-quickstart on Thu May 14 18:13:12 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +1.9 +=== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + ./release-note-1.9.0.md + ./release-note-1.9.1.md diff --git a/docs/release/1.9/release-note-1.9.1.md b/docs/release/1.9/release-note-1.9.1.md new file mode 100644 index 000000000..55290cfa1 --- /dev/null +++ b/docs/release/1.9/release-note-1.9.1.md @@ -0,0 +1,9 @@ +# Release Note 1.9.1 + +## ONE Compiler + +### Changes + +- `tf2nnpkg` now supports to import TensorFlow model which includes BCQ information. +- Minor change for preserving BCQ information. +- Fix invalid input arguments and add more error handles for one-cmds diff --git a/docs/release/index.rst b/docs/release/index.rst index bb542bc1c..f49d3eac4 100644 --- a/docs/release/index.rst +++ b/docs/release/index.rst @@ -15,3 +15,15 @@ Release ./1.2/index ./1.3/index ./1.4/index + ./1.5/index + ./1.6/index + ./1.7/index + ./1.8/index + ./1.9/index + ./1.10/index + ./1.11/index + ./1.12/index + ./1.13/index + ./1.14/index + ./1.15/index + ./1.16/index diff --git a/docs/release/onert-micro/0.1/release-note-0.1.0.md b/docs/release/onert-micro/0.1/release-note-0.1.0.md new file mode 100644 index 000000000..5a4692b4f --- /dev/null +++ b/docs/release/onert-micro/0.1/release-note-0.1.0.md @@ -0,0 +1,72 @@ +## Release Notes for onert-micro 0.1.0 + +_onert-micro_ is tiny runtime specialized for running NN model in MCU boards. Note that onert-micro is under active development and is subject to change. + +### Supported operations + +For MCU board, we support 22 operations as follows : + + ADD, FULLY_CONNECTED, CONV_2D, LOGISTIC ,GATHER, EXPAND_DIMS, PACK, RESHAPE, REDUCE_PROD, LESS, MUL, MAX_POOL_2D, CONCATENATION, SHAPE, SLICE, SUB, SPLIT, STRIDED_SLICE, TANH, SOFTMAX, WHILE, UNIDIRECTIONAL_SEQUENCE_LSTM + +### RNN Model + +#### LSTM + +onert-micro supports Keras model with LSTM operations. But, it should be converted to UNIDIRECTIONAL_SEQUENCE_LSTM operation in circle format. + +#### GRU + +onert-micro supports model with GRU Operations, which is converted from Keras Model. Please refer to https://github.com/Samsung/ONE/issues/10465 to see GRU operation supported by onert-micro. + +### Benchmark + +onert-micro shows better performance than tflite-micro especially in memory consumption, binary size. + +The measurement is done on TizenRT running reference models on the development board with the following spec : + +- 32-bit Arm Cortex-M33 200MHz +- 4MB RAM, 8MB Flash + +Commit for measurement : +- tflite-micro commit: https://github.com/tensorflow/tflite-micro/commit/4e62ea7b821c1e6af004912132395fb81922ea8d + +- onert-micro commit: https://github.com/Samsung/ONE/commit/c763867500fe3d80bfd1ef834990d34a81640d17 +#### L model + +| Params | Tflite micro | Onert-micro | +|-----------------------------------|---------------|-------------| +| Execution time(us)* | **2 912 700** | 2 953 000 | +| RAM consumption(bytes) | 126 800 | **93 376** | +| Binary file size overhead (bytes) | 57 676 | **32 248** | + + +#### T1 model + +Params | Tflite micro | Onert-micro | +--- | --- | --- +Execution time(us)* | **1 340** | 1 510 | +RAM consumption(bytes) | 1 640 | **1 152** | +Binary file size overhead (bytes) | 35 040 | **19 432** | + +#### T2 model + +Params | Tflite micro** | Onert-micro | +--- | --- | --- +Execution time(us)* | N/A | 5 090 | +RAM consumption(bytes) | N/A | 3 360 | +Binary file size overhead (bytes) | N/A | 30 488 | + +#### Model with GRU operations + +- model link : https://github.com/Samsung/ONE/files/8368702/gru.zip + +Params | Tflite micro** | Onert-micro | +--- | --- | --- +Execution time(us)* | N/A | 335 000 | +RAM consumption(bytes) | N/A | 14 816 | +Binary file size overhead (bytes) | N/A | 43 444 | + + +(*) Average for 100 inferences +(**) Tflite-micro has not launched this model + diff --git a/docs/release/onert-micro/1.0/release-note-1.0.0.md b/docs/release/onert-micro/1.0/release-note-1.0.0.md new file mode 100644 index 000000000..a18d55724 --- /dev/null +++ b/docs/release/onert-micro/1.0/release-note-1.0.0.md @@ -0,0 +1,12 @@ +## Release Notes for onert-micro 1.0 + +### Supported operations + +More operations are supported as follows: + +- AveragePool2D, Elu, Exp, Abs, Neg, Div, AddN, Relu, Relu6, Leak_Relu, Pad, PadV2, ArgMin, ArgMax, Resize_Bilinear, LogicalAnd, LogicalOr, Equal, NotEqual, Greater, GreaterEqual, LessEqual + +### Etc + +- Address sanitizer build option(ENABLE_SANITIZER) is added +- Fix buffer overflow in static analyzer's defect diff --git a/docs/runtime/api.md b/docs/runtime/api.md index 3ff9ff056..9dacd2868 100644 --- a/docs/runtime/api.md +++ b/docs/runtime/api.md @@ -28,8 +28,6 @@ For usage, refer to [Howto : NNFW API](../howto/how-to-use-nnfw-api.md). ## Backend API -Backend API is defined by One Runtime. - -Backend API is about actual computation of operations and memory management for operands. In order to allow different kinds of computation units or computation libraries, One Runtime defines Backend API to support user defined operation kernels and memory manager. It contains a lot of C++ headers which are subject to change. +Backend API enables anyone to extend the runtime in terms of operation computation and memory management. For detailed descriptions, refer to [Backend API](../runtime/backend-api.md). diff --git a/docs/runtime/backend-api.md b/docs/runtime/backend-api.md index b291badcf..54ef87090 100644 --- a/docs/runtime/backend-api.md +++ b/docs/runtime/backend-api.md @@ -1 +1,65 @@ # Backend API + +Backend API is defined by One Runtime. It is about actual computation of operations and memory management for operands. In order to allow different kinds of computation units or libraries, Backend API is exposed to support user defined operation kernels and memory manager. It contains several C++ interface classes which are **subject to change**. + +## How backends are loaded + +When a backend ID is given to a session, the compiler module tries to load `libbackend_{BACKEND_ID}.so`. If it is successful, the runtime looks up for C API functions in it, and make use of those. + +## C and C++ API + +### C API + +We have 2 C API functions which are used as the entrypoint and the exitpoint. Here are the definitions of those. + +```c +onert::backend::Backend *onert_backend_create(); +void onert_backend_destroy(onert::backend::Backend *backend); +``` + +What they do is creating a C++ object and destroying it, respectively. These two functions are the only ones that are dynamically resolved at runtime. + +### C++ API + +> **NOTE** C++ API is subject to change so it may change in every release + +C API above is just an entrypoint and it delegates core stuff to C++ API. + +Here are major classes are described below. One must implement these classes(and some more classes) to create a backend. + +- `Backend` : Responsible to create a backend context which is a set of backend components +- `BackendContext` : Holds data for the current session and also responsible to create tensor objects and kernels + - `BackendContext::genTensors` : Create tensor objects + - `BackendContext::genKernels` : Create kernels +- `IConfig` : Configurations and miscellaneous stuff (not session based, global) +- `ITensorRegistry` : A set of tensor(`ITensor`) objects that are used by the current backend + +Please refer to each class document for details. You may refer to [Bundle Backends](#bundle-backends) for actual implementation samples. + +## Provided Backend Implementations + +We provide some backends along with the runtime. There is the special backend `builtin` which is part of runtime core, and some bundle backends which are baseline backends and samples of backend implementation. + +## `builtin` Backend + +`builtin` is a special backend that is always loaded(statically linked, part of runtime core). It is implemented just like other backends, but there are some things that it does exclusively. + +- Has kernels for If, While and Permute operations (Kernels from other backends are never be used) +- The runtime core directly creates `builtin`'s tensor objects to accept user-given input and output buffers +- The runtime core gives the executor context to `builtin` backend which allows control flow ops can change execution flow properly + +## Bundle Backends + +Without actual implmentation of backends, we cannot run any models. So we provide 3 bundle backends which support dozens of operations. + +### cpu + +This backend is written in C++ and all the computation is done with CPU only. + +### acl_neon + +`acl_neon` is a backend that is an adaptation layer of [ARM ComputeLibrary](https://github.com/ARM-software/ComputeLibrary) NE(NEON) part. So it basically only uses CPU too, but worksonly on ARM. + +### acl_cl + +`acl_cl` is a backend that is an adaptation layer of [ARM ComputeLibrary](https://github.com/ARM-software/ComputeLibrary) CL(OpenCL) part. OpenCL support(`libOpenCL.so`) is also necessary in the running environment to be able to use this backend. Also, it works only on ARM. diff --git a/docs/runtime/controlflow-operations.md b/docs/runtime/controlflow-operations.md new file mode 100644 index 000000000..8c456b23a --- /dev/null +++ b/docs/runtime/controlflow-operations.md @@ -0,0 +1,40 @@ +# Controlflow Operations + +We call `If` and `While` operations "Controlflow operations". These operations are different from the others. They are not for computing data, they are used to invoke another subgraph and return back which is to make conditional/iterations work in dataflow models. + +## Defining controlflow operations + +As we use Tensorflow Lite schema(or Circle which is based on TF Lite), the runtime follows the way TF Lite does. The details are stated in [Control Flow in TensorFlow Lite](https://github.com/tensorflow/community/blob/master/rfcs/20190315-tflite-control-flow.md) RFC document. + +Controlflow operations from NN API is not yet supported. But we expect that it can be enabled in the similar way. + +## Implementation + +### Graph representation + +`onert` internally has its representation for controlflow operations and subgraphs. It is pretty much straightforward as it is pretty much isomorphic with the schema. The `onert`'s in-memory model contains multiple subgraphs and the controlflow operations have same parameters(subgraph indices) just like TF Lite schema has. + +### Execution + +`controlflow` backend is a built-in backend to support these controlflow operations. This backend is special as it has access to `onert` core's executor manager(`ExecutorMap`) so it can invoke/return a subgraph. This backend has implementation for `If` and `While` operation and they make use of the access to executor manager. + +An `Executor` has two different ways to execute depending on if it is the initial execution or invoking a subgraph from a controlflow operation. + +- Executing the primary subgraph + - Pass user-given tensors as the subgraph inputs and outputs +- Executing a subgraph for controlflow operations + - Pass controlflow operation inputs tensors as the subgraph inputs + - Pass the subgraph outputs as controlflow operation outputs + +#### Kernel Implementation + +Here is brief explanation what the kernels do, which is quoted from [Control Flow in TensorFlow Lite](https://github.com/tensorflow/community/blob/master/rfcs/20190315-tflite-control-flow.md). + +> * `If` : Check the condition input and invoke one of the 2 subgraphs. +> * `While` : +> * Invoke the condition subgraph. Break out the loop if result is false. +> * Invoke the body subgraph, use the output as the input of the next iteration. + +Invoking a subgraph needs to pass the operation's inputs to the subgraph inputs. And Returning back needs to pass the subgraph outputs to the operation outputs. + +When invoking a subgraph and returning back, the current kernel implementation performs literally copy all the subgraph inputs and outputs. This is going to be optimized to minimize redundant copies. diff --git a/docs/runtime/index.rst b/docs/runtime/index.rst index 4d12dc37b..e80dfc81e 100644 --- a/docs/runtime/index.rst +++ b/docs/runtime/index.rst @@ -12,7 +12,9 @@ Runtime ./api.md ./core.md - ./compute.md + ./controlflow-operations.md ./executors.md - ./backend-api.md ./heterogeneous-execution.md + ./backend-api.md + ./compute.md + ./supported-operations-backend.md diff --git a/docs/runtime/supported-operations-backend.md b/docs/runtime/supported-operations-backend.md new file mode 100644 index 000000000..b5b5c6268 --- /dev/null +++ b/docs/runtime/supported-operations-backend.md @@ -0,0 +1,208 @@ +# Supported Operations and backend + +As of 2021-03-08 + +### Raw-data format (float32, int32, boolean, etc) + +Operation | CPU | ACL-CL | ACL-NEON +-- | -- | -- | -- +Abs | O | O | O +Add | O | O | O +AddN | O | | +ArgMax | O | O | O +ArgMin | O | O | O +AvgPool2D | O | O | O +BatchMatmul | O | | +BatchToSpaceND | O | O | O +BroadcastTo | O | | +Cast | O | O | O +Concat | O | O | O +Conv2D | O | O | O +Cos | O | | +Custom | O | | +DepthToSpace | O | O | O +DepthwiseConv2D | O | O | O +Div | O | O | O +Einsum | O | | +Elu | O | | +EmbeddingLookup | | O | O +Equal | O | O | O +Exp | O | O | O +ExpandDims | O | O | O +Fill | O | | +Floor | O | O | O +FullyConnected | O | O | O +FusedBatchNorm | O | | +Gather | O | O | O +Greater | O | O | O +GreaterEqual | O | O | O +HashtableLookup | | O | O +If | O | | +InstanceNormalize | | O | O +L2Normalization | O | O | O +L2Pool | | O | O +LeakyRelu | O | O | O +Less | O | O | O +LessEqual | O | O | O +LocalResponseNormalize | | O | O +Log | O | | +LogicalAnd | O | O | O +LogicalNot | O | O | O +LogicalOr | O | O | O +Logistic | O | O | O +LogSoftmax | O | | +LSTM | | O | O +MatrixBandPart | O | | +Maximum | O | O | O +MaxPool2D | O | O | O +Mean | O | O | O +Minimum | O | O | O +Mul | O | O | O +Neg | O | O | O +NotEqual | O | O | O +OneHot | O | O | +Pack | O | O | O +Pad | O | O | O +PadV2 | O | O | O +Pow | O | | +PReLU | | O | O +Quantize | O | | +Range | O | | +Rank | O | | +ReduceAny(All) | O | | +ReduceAny(Any) | O | | +ReduceMax(Max) | O | O | O +ReduceMin(Min) | O | O | O +ReduceProd | O | | +ReduceSum(Sum) | O | O | O +ReLU | O | O | O +ReLU6 | O | O | O +Reshape | O | O | O +ResizeBilinear | O | O | O +ResizeNearestNeighbor | | O | O +ReverseV2 | O | O | O +RNN | | O | O +Round | O | | +Rsqrt | O | O | O +Select | O | | +SelectV2 | O | | +Shape | O | | +Sin | O | | +Slice | O | O | O +Softmax | O | O | O +SpaceToBatchND | O | O | O +SpaceToDepth | O | O | O +Split | O | O | O +SplitV | O | O | +Sqrt | O | O | O +Square | O | | | +SquaredDifference | O | O | O +Squeeze | O | O | O +StridedSlice | O | O | O +Sub | O | O | O +Tanh | O | O | O +Tile | O | | +TopKV2 | | | O +Transpose | O | O | O +TransposeConv | | O | O +Unpack(Unstack) | O | O | O +UniDirectionalSequenceLSTM | O | | +While | O | | +ZerosLike | O | | + +### Quantization format (uint8 asymmetric) + +Operation | CPU | ACL-CL | ACL-NEON +-- | -- | -- | -- +Add | O | O | O +ArgMax | O | O | O +ArgMin | O | O | O +AvgPool2D | O | O | O +BatchToSpaceND | O | O | O +Cast | O | O | +Concat | O | O | O +Conv2D | O | O | O +Custom | O | | +DepthToSpace | O | O | O +DepthwiseConv2D | O | O | O +Dequantize | O | O | O +Div | O | | +EmbeddingLookup | | O | O +Equal | O | O | O +Erf | O | | +ExpandDims | O | O | O +FullyConnected | O | O | O +Gather | O | O | O +Greater | O | O | O +GreaterEqual | O | O | O +HashtableLookup | | O | O +L2Normalization | O | | +Less | O | O | O +LessEqual | O | O | O +Logistic | O | O | O +LogSoftmax | O | | +Maximum | O | O | O +MaxPool2D | O | O | O +Mean | O | O | O +Minimum | O | O | O +Mul | O | O | +NotEqual | O | O | O +Pack | | O | O +Pad | O | O | O +PadV2 | O | O | O +PReLU | | O | O +Quantize | O | | +Rank | O | | +ReduceMax(Max) | | O | +ReduceMin(Min) | | O | +ReduceSum(Sum) | O | O | +ReLU | | O | O +ReLU6 | | O | O +Reshape | O | O | O +ResizeBilinear | O | O | O +ResizeNearestNeighbor | | O | O +Shape | O | | +Slice | O | O | O +Softmax | O | O | O +SpaceToBatchND | O | O | O +SpaceToDepth | O | O | O +Split | O | O | O +SplitV | O | O | +Squeeze | O | O | O +StatelessRandomUniform | O | | +StridedSlice | | O | O +Sub | O | O | O +Tanh | O | O | O +Tile | O | | +Transpose | O | O | O +TransposeConv | | O | O +Unpack(Unstack) | | O | O + +### Quantization format (int8) + +Operation | CPU | ACL-CL | ACL-NEON +-- | -- | -- | -- +Add | O | O | O +ArgMax | O | O | O +ArgMin | O | O | O +AvgPool2D | O | | +Concat | O | O | O +Conv2D | O | | +DepthToSpace | O | | +DepthwiseConv2D | O | | +Dequantize | O | O | O +ExpandDims | O | O | O +MaxPool2D | O | | +Mul | O | O | O +Pad | O | O | O +PadV2 | O | | +PReLU | | O | O +Quantize | O | | +Rank | O | | +Reshape | O | O | O +ResizeBilinear | O | O | O +ResizeNearestNeighbor | | O | O +Shape | O | | +Softmax | O | O | O +Squeeze | O | O | O +Sub | O | O | O |