Imported Upstream version 1.4.0upstream/1.4.0submit/tizen/20200423.054851

3 files changed, 336 insertions, 0 deletions

diff --git a/compiler/dredd-rule-lib/CMakeLists.txt b/compiler/dredd-rule-lib/CMakeLists.txt
new file mode 100644
index 000000000..b39d86272
--- /dev/null
+++ b/compiler/dredd-rule-lib/CMakeLists.txt
@@ -0,0 +1,21 @@
+#
+# copy rule-lib.sh (a library of shell script functions)
+#
+set(SOURCE_RULE_LIB "${CMAKE_CURRENT_SOURCE_DIR}/rule-lib.sh")
+set(TARGET_RULE_LIB "${CMAKE_CURRENT_BINARY_DIR}/rule-lib.sh")
+
+add_custom_command(
+  OUTPUT ${TARGET_RULE_LIB}
+  COMMAND ${CMAKE_COMMAND} -E copy "${SOURCE_RULE_LIB}" "${TARGET_RULE_LIB}"
+  DEPENDS ${SOURCE_RULE_LIB}
+  COMMENT "Generate rule lib"
+)
+
+# Generate dependencies
+add_custom_target(dredd_rule_lib ALL DEPENDS ${TARGET_RULE_LIB})
+
+# How to get the path of rule-lib.sh in other CMakeLists.txt
+#
+# get_target_property(DREDD_RULE_LIB_DIR
+#                     dredd_rule_lib BINARY_DIR)
+# set(RULE_LIB_PATH "${DREDD_RULE_LIB_DIR}/rule-lib.sh")
diff --git a/compiler/dredd-rule-lib/README.md b/compiler/dredd-rule-lib/README.md
new file mode 100644
index 000000000..348b0aefb
--- /dev/null
+++ b/compiler/dredd-rule-lib/README.md
@@ -0,0 +1,112 @@
+# dredd-rule-lib
+
+*dredd-rule-lib* is a library that defines functions to run *dredd* tests, which checks non-functional aspect of compiled files.
+
+## Terms
+
+Assume that we want to check the size of generated tflite file to be less than 1024 Bytes.
+In such case, we'd like to use the following terms:
+
+- "metric" : *file size*
+- "rule" : *file size < 1024*
+- "metric function": `file_size` that returns size of a compiled tflite file
+
+Models (input of test) exist in *model repo*, where
+
+- "model repo" : directory where models exist. For *tf2tflite-dredd-pbtxt-test*, model repo is
+  `res/TensorFlowTests`.
+
+## Metrics supported
+
+The following metric functions are provided:
+- `all_op_count` : the count of operations inside a compiled tflite file
+- `file_size` : the size of compiled tflite file
+- In addition, `op_count`, `conv2d_weight_not_constant`, etc.
+- Please , refer to [`rule-lib.sh`](rule-lib.sh) for metric functions
+
+## Related projects - *dredd* tests
+
+Four *dredd* test projects use *dredd-rule-lib*:
+
+- *tf2tflite-dredd-pbtxt-test*
+  - Models in `pbtxt`, text file, are compiled into `tflite` file.
+  - Then `rule` file that each model has is checked against the `tflite` file.
+- *tf2tflite-dredd-pb-test*
+  - Models in `pb`, binary file, are compiled into `tflite` file.
+  - Then `rule` file that each model has is checked against the `tflite` file.
+- *tf2circle-dredd-pbtxt-test*
+  - Models in `pbtxt`, text file, are compiled into `circle` file.
+  - Then `rule` file that each model has is checked against the `circle` file.
+- *tf2circle-dredd-pb-test*
+  - Models in `pb`, binary file, are compiled into `circle` file.
+  - Then `rule` file that each model has is checked against the `circle` file.
+
+## Rule file
+
+To be a target of *dredd*-tests, a `.rule` file **must** exist in a model directory.
+Please refer to `res/TensorFlowTests/NET_0025/tflite_1.0_rel_requirement.rule` for an example.
+
+### Naming convention of rule file
+
+Note that the file name `tflite_1.0_rel_requirement.rule` is our convention containing the
+information below:
+- Generated file type (`tflite`)
+- SDK version (`1.0_rel`)
+- Purpose (`requirement`)
+
+## How do all these work?
+
+For *tf2tflite-dredd-pbtxt-test*, (*tf2circle-dredd-pbtxt-test* works similarly)
+
+```
+model repo                                   tf2tflite-dredd-pbtxt-test
+-----------------------------------------------------------------------------------------------
+   NET_0025
+    ├── test.pbtxt  ---------------------->  converted to NET_0025.pb, and then NET_0025.tflite
+    |                                       /|\
+    ├── test.info ---------------------------+
+    |   (input/output info of model)
+    |
+    └── tflite_1.0_rel_requirement.rule -->  running rule file against tflite --> pass or fail
+                                                      /|\
+                          dredd-rule-lib               | (using)
+                      ----------------------           |
+                          rule-lib.sh                  |
+                            - defining rule function --+
+```
+
+For *tf2tflite-dredd-pb-test*, (*tf2circle-dredd-pb-test* works similarly)
+
+```
+model repo                                   tf2tflite-dredd-pb-test
+-----------------------------------------------------------------------------------------------
+   Inception_v3
+    ├── model.pb  ------------------------>  converted to Inception_v3.tflite
+    |                                       /|\
+    ├── model.info --------------------------+
+    |   (input/output info of model)
+    |
+    └── tflite_1.0_rel_requirement.rule -->  running rule file against tflite --> pass or fail
+                                                      /|\
+                          dredd-rule-lib               | (using)
+                      ----------------------           |
+                          rule-lib.sh                  |
+                            - defining rule function --+
+```
+
+## Model repo and How to add a model as a target of a *dredd*-test.
+
+For *tf2tflite-dredd-pbtxt-test* and *tf2circle-dredd-pbtxt-test*,
+model repo is `res/TensorFlowTests`.
+
+To add a model into these tests, the model directory name should be added into one of the following files:
+- `test.lst` : This file resides in git
+- `test.local.lst` : This file is ignored by git. Use this for personal purpose.
+
+For *tf2tflite-dredd-pb-test* and *tf2circle-dredd-pb-test*,
+model repo is `tf2tflite-dredd-pb-test/contrib` and .`tf2circle-dredd-pb-test/contrib` respectively.
+
+Use these tests for binary models in large size.
+
+To add a model into these tests, the model directory name should be added into the following file:
+- `contrib.lst` : This file is ignored by git.
diff --git a/compiler/dredd-rule-lib/rule-lib.sh b/compiler/dredd-rule-lib/rule-lib.sh
new file mode 100755
index 000000000..8ebe3d7af
--- /dev/null
+++ b/compiler/dredd-rule-lib/rule-lib.sh
@@ -0,0 +1,203 @@
+#!/bin/bash
+
+# the following env vars should be defined to call dredd function (except RULE):
+#   COMPILED_FILE
+#   INSPECT_PROG_PATH
+#   VERIFY_PROG_PATH
+#   ERROR_LOG
+
+# exit if unknown var is used
+set -u
+
+# ---------------
+# HELPER FUNCTION
+
+init_error_log()
+{
+  # create ${ERROR_LOG} that redirect stderr for pipe
+  exec 2>"${ERROR_LOG}"
+}
+
+argc_check()
+{
+  ACTUAL_ARGC=$1
+  EXPECTED_ARGC=$2
+
+  if [ "$#" -ne 2 ];then
+    echo "argc_check : param count must be 2" > ${ERROR_LOG}
+    echo "error"  # return value of sub-shell
+    exit 1
+  fi
+
+  if [ ${ACTUAL_ARGC} -ne ${EXPECTED_ARGC} ];then
+    echo "arg count mismatch: actual = ${ACTUAL_ARGC} vs expected = ${EXPECTED_ARGC}" > ${ERROR_LOG}
+    echo "error"  # return value of sub-shell
+    exit 1
+  fi
+}
+
+file_path_check()
+{
+  argc_check $# 1
+
+  if [ ! -f $1 ]; then
+    echo "$1 does not exist" > ${ERROR_LOG}
+    echo "error"  # return value of sub-shell
+    exit 1
+  fi
+}
+
+check_success_exit_code()
+{
+  ACTUAL_EXIT_CODE=$1
+  EXPECTED_SUCCESS_CODE=$2
+
+  if [ ${ACTUAL_EXIT_CODE} -ne ${EXPECTED_SUCCESS_CODE} ];then
+    echo "error"
+    exit 1
+  fi
+}
+
+check_error_exit_code()
+{
+  ACTUAL_EXIT_CODE=$1
+  EXPECTED_ERROR_CODE=$2
+
+  if [ ${ACTUAL_EXIT_CODE} -eq ${EXPECTED_ERROR_CODE} ];then
+    echo "error"
+    exit 1
+  fi
+}
+
+# END of HELPER FUNCTION
+# ----------------------
+
+#
+# Define rule
+#
+#   - Params: rule name (metric), actual value, condition, expected value
+#     - condition is '=', '!=', '<', '>', '<=', '>='. Refer to "man expr"
+#   - Return
+#     - 0 : success
+#     - 1 : fail (condition check fail)
+#
+
+RULE()
+{
+  argc_check $# 4
+
+  RULE_NAME=$1
+  ACTUAL=$2
+  COND=$3
+  EXPECTED=$4
+
+  # not to exit when expr result with 0
+  set +e
+
+  expr ${ACTUAL} ${COND} ${EXPECTED} > /dev/null
+  RESULT=$?
+
+  # roll-back
+  set -e
+
+  # Note: return value of 'expr'
+  # - 0 : result is true
+  # - 1 : result is false
+  # - 2 : error
+
+  if [ ${RESULT} -eq 0 ];then
+    echo -e "** [${RULE_NAME}] \t success \t ([actual: ${ACTUAL}] ${COND} [expected: ${EXPECTED}])"
+  elif [ ${RESULT} -eq 1 ];then
+    echo -e "** [${RULE_NAME}] \t ** fail \t ([actual: ${ACTUAL}] ${COND} [expected: ${EXPECTED}])"
+  else
+    echo -e "\t** Error in [expr ${ACTUAL} ${COND} ${EXPECTED}]"
+  fi
+
+  return ${RESULT}
+}
+
+#
+# Define each function to get quality value
+#
+
+# Note: These function is called by a sub-shell.
+# So return value should be passed through "echo return_value"
+# tip: for debugging, surround the code with "set -x" and "set +x"
+
+file_size()
+{
+  file_path_check ${COMPILED_FILE}
+
+  set -o pipefail
+
+  ACTUAL=`init_error_log ; cat ${COMPILED_FILE} | wc -c`
+
+  check_success_exit_code $? 0
+
+  echo ${ACTUAL}
+}
+
+all_op_count()
+{
+  file_path_check ${COMPILED_FILE}
+  file_path_check ${INSPECT_PROG_PATH}
+
+  set -o pipefail
+
+  ACTUAL=`init_error_log ; ${INSPECT_PROG_PATH} --operators ${COMPILED_FILE} | wc -l`
+
+  check_success_exit_code $? 0
+
+  echo ${ACTUAL}
+}
+
+op_count()
+{
+  argc_check $# 1
+  file_path_check ${COMPILED_FILE}
+  file_path_check ${INSPECT_PROG_PATH}
+
+  set -o pipefail
+
+  RESULT=`init_error_log ; ${INSPECT_PROG_PATH} --operators ${COMPILED_FILE}`
+  check_success_exit_code $? 0
+
+  # note : grep's exit code is 2 in case of error.
+  ACTUAL=`init_error_log ; echo "${RESULT}" | grep -wc "$1"`
+  check_error_exit_code $? 2
+
+  echo ${ACTUAL}
+}
+
+conv2d_weight_not_constant()
+{
+  file_path_check ${COMPILED_FILE}
+  file_path_check ${INSPECT_PROG_PATH}
+
+  set -o pipefail
+
+  ACTUAL=`init_error_log ; \
+          ${INSPECT_PROG_PATH} --conv2d_weight ${COMPILED_FILE} | \
+          awk -F, '{ if ($2 != "CONST") print $0}' | wc -l`
+
+  check_success_exit_code $? 0
+
+  echo ${ACTUAL}
+}
+
+verify_file_format()
+{
+  file_path_check ${COMPILED_FILE}
+  file_path_check ${VERIFY_PROG_PATH}
+
+  set -o pipefail
+
+  ACTUAL=`init_error_log ; ${VERIFY_PROG_PATH} ${COMPILED_FILE} | grep -c "PASS"`
+
+  # note grep can exit with 1 ("PASS" not found) and this is treated as an error
+  check_success_exit_code $? 0
+
+  echo ${ACTUAL}
+}
+
+# TODO define more qullity test function