1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
|
# dredd-rule-lib
*dredd-rule-lib* is a library that defines functions to run *dredd* tests, which checks non-functional aspect of compiled files.
## Terms
Assume that we want to check the size of generated tflite file to be less than 1024 Bytes.
In such case, we'd like to use the following terms:
- "metric" : *file size*
- "rule" : *file size < 1024*
- "metric function": `file_size` that returns size of a compiled tflite file
Models (input of test) exist in *model repo*, where
- "model repo" : directory where models exist. For *tf2tflite-dredd-pbtxt-test*, model repo is
`res/TensorFlowTests`.
## Metrics supported
The following metric functions are provided:
- `all_op_count` : the count of operations inside a compiled tflite file
- `file_size` : the size of compiled tflite file
- In addition, `op_count`, `conv2d_weight_not_constant`, etc.
- Please , refer to [`rule-lib.sh`](rule-lib.sh) for metric functions
## Related projects - *dredd* tests
Four *dredd* test projects use *dredd-rule-lib*:
- *tf2tflite-dredd-pbtxt-test*
- Models in `pbtxt`, text file, are compiled into `tflite` file.
- Then `rule` file that each model has is checked against the `tflite` file.
- *tf2tflite-dredd-pb-test*
- Models in `pb`, binary file, are compiled into `tflite` file.
- Then `rule` file that each model has is checked against the `tflite` file.
- *tf2circle-dredd-pbtxt-test*
- Models in `pbtxt`, text file, are compiled into `circle` file.
- Then `rule` file that each model has is checked against the `circle` file.
- *tf2circle-dredd-pb-test*
- Models in `pb`, binary file, are compiled into `circle` file.
- Then `rule` file that each model has is checked against the `circle` file.
## Rule file
To be a target of *dredd*-tests, a `.rule` file **must** exist in a model directory.
Please refer to `res/TensorFlowTests/NET_0025/tflite_1.0_rel_requirement.rule` for an example.
### Naming convention of rule file
Note that the file name `tflite_1.0_rel_requirement.rule` is our convention containing the
information below:
- Generated file type (`tflite`)
- SDK version (`1.0_rel`)
- Purpose (`requirement`)
## How do all these work?
For *tf2tflite-dredd-pbtxt-test*, (*tf2circle-dredd-pbtxt-test* works similarly)
```
model repo tf2tflite-dredd-pbtxt-test
-----------------------------------------------------------------------------------------------
NET_0025
├── test.pbtxt ----------------------> converted to NET_0025.pb, and then NET_0025.tflite
| /|\
├── test.info ---------------------------+
| (input/output info of model)
|
└── tflite_1.0_rel_requirement.rule --> running rule file against tflite --> pass or fail
/|\
dredd-rule-lib | (using)
---------------------- |
rule-lib.sh |
- defining rule function --+
```
For *tf2tflite-dredd-pb-test*, (*tf2circle-dredd-pb-test* works similarly)
```
model repo tf2tflite-dredd-pb-test
-----------------------------------------------------------------------------------------------
Inception_v3
├── model.pb ------------------------> converted to Inception_v3.tflite
| /|\
├── model.info --------------------------+
| (input/output info of model)
|
└── tflite_1.0_rel_requirement.rule --> running rule file against tflite --> pass or fail
/|\
dredd-rule-lib | (using)
---------------------- |
rule-lib.sh |
- defining rule function --+
```
## Model repo and How to add a model as a target of a *dredd*-test.
For *tf2tflite-dredd-pbtxt-test* and *tf2circle-dredd-pbtxt-test*,
model repo is `res/TensorFlowTests`.
To add a model into these tests, the model directory name should be added into one of the following files:
- `test.lst` : This file resides in git
- `test.local.lst` : This file is ignored by git. Use this for personal purpose.
For *tf2tflite-dredd-pb-test* and *tf2circle-dredd-pb-test*,
model repo is `tf2tflite-dredd-pb-test/contrib` and .`tf2circle-dredd-pb-test/contrib` respectively.
Use these tests for binary models in large size.
To add a model into these tests, the model directory name should be added into the following file:
- `contrib.lst` : This file is ignored by git.
|
