Abstract
Deep learning algorithms have achieved excellent performance lately in a wide range of fields (e.g., computer version). However, a severe challenge faced by deep learning is the high dependency on hyper-parameters. The algorithm results may fluctuate dramatically under the different configuration of hyper-parameters. Addressing the above issue, this paper presents an efficient Orthogonal Array Tuning Method (OATM) for deep learning hyper-parameter tuning. We describe the OATM approach in five detailed steps and elaborate on it using two widely used deep neural network structures (Recurrent Neural Networks and Convolutional Neural Networks). The proposed method is compared to the state-of-the-art hyper-parameter tuning methods including manually (e.g., grid search and random search) and automatically (e.g., Bayesian Optimization) ones. The experiment results state that OATM can significantly save the tuning time compared to the state-of-the-art methods while preserving the satisfying performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
For the sake of simplicity, we consider all the factors with the same number of levels. More advanced knowledge can be found in [8] for more complex situations.
- 3.
- 4.
- 5.
Assume all the hidden layers have the same fixed number of nodes.
- 6.
We consider each convolutional layer and the following pooling layer as whole.
- 7.
References
Andradóttir, S.: A review of random search methods. In: Fu, M. (ed.) Handbook of Simulation Optimization, pp. 277–292. Springer, New York (2015). https://doi.org/10.1007/978-1-4939-1384-8_10
Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: NeurIPS 24, pp. 2546–2554 (2011)
Calandra, R., Gopalan, N., Seyfarth, A., Peters, J., Deisenroth, M.P.: Bayesian gait optimization for bipedal locomotion. In: Pardalos, P.M., Resende, M.G.C., Vogiatzis, C., Walteros, J.L. (eds.) LION 2014. LNCS, vol. 8426, pp. 274–290. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09584-4_25
Fida, B., Bibbo, D., Bernabucci, I., et al.: Real time event-based segmentation to classify locomotion activities through a single inertial sensor. In: MobiHealth, pp. 104–107 (2015)
Mahapatra, S., Patnaik, A.: Optimization of wire electrical discharge machining (wedm) process parameters using taguchi method. IJAMT 34(9), 911–925 (2007)
Nalbant, M., Gökkaya, H., Sur, G.: Application of taguchi method in the optimization of cutting parameters for surface roughness in turning. Mater. Des. 28(4), 1379–1385 (2007)
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: NeurIPS 25, pp. 2951–2959. Curran Associates, Inc. (2012)
Taguchi, G., Taguchi, G.: System of experimental design; engineering methods to optimize quality and minimize costs. Technical report (1987)
Yao, L., et al.: Compressive representation for device-free activity recognition with passive RFID signal strength. IEEE Trans. Mob. Comput. 17(2), 293–306 (2017)
Zhang, X., Yao, L., Huang, C., Sheng, Q.Z., Wang, X.: Intent recognition in smart living through deep recurrent neural networks. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.S. (eds.) ICONIP 2017. LNCS, vol. 10635, pp. 748–758. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70096-0_76
Zhang, X., Yao, L., Sheng, Q.Z., Kanhere, S.S., Gu, T., Zhang, D.: Converting your thoughts to texts: enabling brain typing via deep feature learning of EEG signals. In: PerCom 2018. IEEE (2018)
Zhang, X., Yao, L., Wang, X., Monaghan, J., Mcalpine, D., Zhang, Y.: A survey on deep learning based brain computer interface: Recent advances and new frontiers. arXiv preprint arXiv:1905.04149 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, X., Chen, X., Yao, L., Ge, C., Dong, M. (2019). Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Communications in Computer and Information Science, vol 1142. Springer, Cham. https://doi.org/10.1007/978-3-030-36808-1_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-36808-1_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36807-4
Online ISBN: 978-3-030-36808-1
eBook Packages: Computer ScienceComputer Science (R0)