Smarter Than Monkey, Meet APE
2 likes1,342 views
The document describes KKBOX's efforts to develop a smarter monkey testing tool called APE using machine learning. APE uses the TensorFlow Object Detection API to train a model on screenshots labeled with UI elements. It then deploys the trained model on Android and iOS to automatically detect and interact with elements during testing. Some key findings included that training with GPU is faster, active learning helps labeling, F1 scores measure model performance, minimizing labels improves learning, and grayscale training works well. The vision is that APE can help with localization testing by detecting languages and aid design patterns like page objects through element detection. In summary, the document discusses how KKBOX is leveraging machine learning for more intelligent automated testing.
Smarter Than Monkey, Meet APE
- 1. Smarter Than Monkey, Meet APE [email protected]
- 2. Agenda ● What’s Monkey ? ● What’s APE ? ● What We Learned ? ○ Recommendation ○ Limitation ○ Vision
- 4. Yihsuan Chen KKBOX Software Engineer in Test
- 6. Monkey Test ● Fuzz testing ○ For stability ○ Crash, OOM, ANR ● Advantage ○ Fast ○ Low-cost
- 7. Monkey Test In KKBOX ● 2016 - 2019 ○ 291 / 310 of closed / open issues ● Daily Crash Free Rate ○ 96% in 2016 ○ 99.9% in 2019 ● Development ○ 5 workdays
- 8. [email protected]:dino-su/android-monkey-quick-start.git Dino Su @KKBOX Android Monkey Test @TestCorner#14
- 9. ?
- 10. Limitation ● Hard to reproduce ○ Random script ○ Random location ○ Random event All things are random! ● Minor issue A Little Stupid
- 13. What We Want 🔲 Continuous Integration 🔲 Incremental Datasets 🔲 Speed 🔲 Well Integrity Lifecycle 🔲 Cross-Platform
- 14. TensorFlow Object Detection API README.md
- 17. Setup ● Training ○ Local or Cloud ● TensorFlow ○ CPU or GPU
- 20. Training Inputs ● Screenshots ● Annotation tool ● Transfer your training dataset to TFRecord file format
- 23. Training Configs ● Pre-trained model Zoo ● Sample configs e.g. ● ssd_mobilenet_v2_quantized ● ssd_mobilenet_v2_quantized_*.config
- 25. Training ● Config, saving place, training steps $ python object_detection/model_main.py --pipeline_config_path= ${CONFIG} --model_dir=${MODEL_SAVING_PLACE} --num_train_steps= ${TRAINING_STEPS} --sample_1_of_n_eval_examples=1 --alsologtostderr
- 26. Tensorboard ● To inspect the progress of training $ tensorboard --logdir= ${MODEL_SAVING_PLACE}
- 29. TensorFlow Lite ● Convert checkpoint to TFLite model.ckpt-20000 $ python object_detection/export_tflite_ssd_graph.py ... tflite/ detect.tflite
- 34. TensorFlow Lite ● Deploy TFLite & labelmap.txt into your project Android app/src/main/assets/labelmap.txt app/src/main/assets/detect.tflite iOS ObjectDetection/Model/labelmap.txt ObjectDetection/Model/detect.tflite
- 35. Live Demo TFLite Object Detector
- 36. How We Test ● Loop, in a specific event count ○ Screenshot ○ Feed the screenshot to TFLite Object Detection API ○ Get labels’ information (label name, coordinate, confidence) ○ Do actions (click, scroll, keyevent)
- 38. A Quick Recap
- 40. What We Want ✅ Continuous Integration ✅ Incremental Datasets ✅ Speed 🔲 Well Integrity Lifecycle 🔲 Cross-Platform
- 41. What We Learned ?
- 42. About ENV
- 44. About ENV ● Between TensorFlow-CPU & TensorFlow-GPU tensorflow (Intel® Core™ i7-7700 Processor) 4h 11 min / 25000 steps tensorflow-gpu (GeForce® GTX 1060 WINDFORCE OC 3G) 1h 12 min / 25000 steps
- 45. Training with GPU is highly recommended.
- 48. Live Demo VoTT Active Learning
- 49. Active Learning is very helpful for labeling.
- 50. About Measure
- 53. Deploy To Freeze Model Convert checkpoint to freeze model (README.md)
- 55. Inference For Single Image Detection ● Get a list of labels ● Make an assertion of specific condition
- 56. Fail assertTrue('play button' in labels) Case 1: Necessary
- 57. Fail assertEqual(len(labels), 0) Case 2: Not-Allowed
- 58. Inference For Single Image ● Advantage ○ Fast ○ Effective It works, but not that smart.
- 60. F1 Measure ● With a general formula ○ Detection ○ Boxes information of ground truth ○ Confidence score ○ Precision / Recall ○ F1 Score ● F1 Score (Performance indicators)
- 62. Label F1 Criteria F1 Score F1 Diff back button 0 0.946 0.946 cancel button 0 0.969 0.969 check box 0 1.0 1.0 cover 0 0.955 0.955 pause button 0 0.882 0.882 play button 0 0.876 0.876 text button 0 0.949 0.949 overall 0 0.939 0.939 F1 Score - 1st Model
- 63. Label F1 Criteria F1 Score F1 Diff back button 0.946 0.948 0.02 cancel button 0.969 0.979 0.1 check box 1.0 1.0 0 cover 0.955 0.998 0.043 pause button 0.882 1.0 0.118 play button 0.876 1.0 0.124 text button 0.949 0.949 0 overall 0.939 0.982 0.043 F1 Score - 2nd Model
- 64. Label F1 Criteria F1 Score F1 Diff back button 0.948 0.939 -0.009 cancel button 0.979 0.981 0.002 check box 1.0 1.0 0 cover 0.998 0.98 -0.018 pause button 1.0 0.955 -0.045 play button 1.0 1.0 0 text button 0.949 0.956 0.007 overall 0.982 0.973 -0.009 F1 Score - 3rd Model
- 65. Do performance measurement ASAP.
- 66. About Label
- 67. Before ● Labeling guideline ○ Each elements -> Different labels
- 68. go off air buttoncancel button
- 69. Before After allow button cancel button confirm button delete button deny button download button go off air button ok button
- 71. Why I'm here ?
- 72. After ● Labeling guideline ○ Each functional groups -> Different labels
- 73. Before After allow button ok button cancel button cancel button confirm button ok button delete button ok button deny button cancel button download button ok button go off air button ok button ok button ok button
- 74. ok button
- 75. Well, not bad
- 76. Keep minimizing labels Model is more learnable than we thought.
- 77. We Lost All Annotations Of All Images
- 78. Before After allow button text button cancel button text button confirm button text button delete button text button deny button text button download button text button go off air button text button ok button text button
- 79. OMG, it’s really not that bad
- 80. Keep minimizing labels Model is “really” more learnable than we thought.
- 81. About Color
- 83. About Color ● Color is not that important to Testing ○ Button color ○ Text color ○ Background color ● Take a long shot, we converted the training into grayscale
- 84. About Color ● Set model config to read image by garyscale model { ... image_resizer { fixed_shape_resizer { ... convert_to_grayscale : true } } }
- 88. Grayscale training is a good choice.
- 89. A Quick Recap
- 91. What We Want ✅ Continuous Integration ✅ Incremental Datasets ✅ Speed ✅ Well Integrity Lifecycle 🔲 Cross-Platform
- 92. Limitation
- 93. About Resolution ● Training dataset ○ Android Phone (Portrait mode) ○ 1080p 16:9 ● POC ○ Android Tablet (Landscape mode) ○ iOS iPhone
- 95. 1136 x 6401920 x 1080 1536 x 2048 HTC Nexus 9iPhone SESamsung A7
- 96. 300 x 300300 x 300 300 x 300 HTC Nexus 9iPhone SESamsung A7
- 97. 200x200x 200x HTC Nexus 9iPhone SESamsung A7
- 98. We are working on it and will update you soon.
- 99. Vision
- 100. Vision ● Conclusion ML is not that far away from us. ● For Mobile ○ Libraries is easy to use ○ Detect speed of TensorFlow Lite is fast ○ As above, and also precise
- 103. Localization Testing ● Optical Character Recognition System ● Flow ○ Set language ○ Test & Detect ○ Assert specific language string 登入 KKBOX Login to KKBOX ログイン
- 104. About Page Object
- 106. [email protected]/iKhC2oCgF4M Vivian Liu @KKBOX Design Patterns in XCUITest @iPlayground 2018
- 107. Page Object class LoginPage { init() { # assertion of this page expectExist(element: LoginButton) } func login() { type(element: accountTextField) type(element: passwordTextField) click(element: loginButton) } }
- 108. Page Object ● Assertion with specific condition ○ isDisplay or isExist ■ On screen ? ■ On hierarchy ?
- 109. Page Object ● Assertion with object detection ● Truly exist and display on the screen PlayerPage assertTrue('play button' in labels) LoginPage assertTrue('login button' in labels)
- 110. Summary
- 111. APE ● TensorFlow Object Detection API ● Inference ● KKBOX APE with TensorFlow Lite
- 112. Recommended ● About Env ○ Ubuntu, with Nvidia GPU ● About Annotation ○ VoTT ● Optimization ○ Minimizing labels ○ Grayscale training
- 113. Vision ● What we thought ○ It’s helpful for Localization Testing ○ It’s helpful for Design Pattern ML is not that far away from us.
- 114. Welcome to the ML world ! Let's do some brainstorming !
- 117. Thanks