Skip to main content
SpringerLink
Log in

Generating Stochastic Processes Through Convolutional Neural Networks

  • Published:
Journal of Control, Automation and Electrical Systems Aims and scope Submit manuscript

Abstract

The present work establishes the use of convolutional neural networks as a generative model for stochastic processes that are widely present in industrial automation and system modelling such as fault detection, computer vision and sensor data analysis. This enables researchers from a broad range of fields—as in medical imaging, robotics and control engineering—to develop a general tool for artificial data generation and simulation without the need to identify or assume a specific system structure or estimate its parameters. We demonstrate the approach as a generative model on top of data retrieved from a wide set of classic, simplest to the most complex, deterministic and stochastic data generation processes of technological importance—from damped oscillators to autoregressive conditional heteroskedastic and jump-diffusion models. Also, a nonparametric estimation and forecast was carried out for the traditional benchmark “Fisher River” time-series dataset, yielding the superior mean absolute prediction error results compared to a standard ARIMA model. This approach can have potential applications as an alternative to simulation tools such as Gibbs sampling and Monte Carlo-based methods, in the enhancement of the understanding of generative adversarial networks (GANs) and in data simulation for training Reinforcement Learning algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Source: van den Oord et al. (2016b)

Fig. 3

Source: van den Oord et al. (2016b)

Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Aibinu, A. M., Salami, M. J. E., & Shafie, A. A. (2012). Artificial neural network based autoregressive modeling technique with application in voice activity detection. Engineering Applications of Artificial Intelligence, 25(6), 1265–1276.

    Article  Google Scholar 

  • Avishek, P., & Prakash, P. (2017). Practical time-series analysis: Master time series data processing, visualization, and modeling using python (1st ed.). Birmingham: Packt Publishing Ltd.

    Google Scholar 

  • Bahrpeyma, F., Roantree, M., McCarren, A. (2018). Multistep-ahead prediction: A comparison of analytical and algorithmic approaches. In Big data analytics and knowledge discovery: 20th International conference, DaWaK 2018.

    Chapter  Google Scholar 

  • Bishop, C. M. (2006). Pattern recognition and machine learning. Cambridge: Springer.

    MATH  Google Scholar 

  • Borovykh, A., Bohte, S., & Oosterlee, C. W. (2017). Conditional time series forecasting with convolutional neural networks. arXiv:1703.04691v3.

  • Bresler, Y., & Macovski, A. (1986). Exact maximum likelihood parameter estimation of superimposed exponential signals in noise. IEEE Transactions on Acoustics, Speech, and Signal Processing, 34, 1081–1089.

    Article  Google Scholar 

  • Briandet, R., Kemsley, E. K., & Wilson, R. H. (1996). Discrimination of Arabica and Robusta in instant coffee by Fourier transform infrared spectroscopy and chemometrics. Journal of Agricultural and Food Chemistry, 44, 170–174.

    Article  Google Scholar 

  • Bruni, V., Vitulano, D., & Kokaram, A. (2003). Line scratches detection and restoration via light diffraction. In 3rd International symposium on image and signal processing and analysis, Rome, Italy (Vol. 1, pp. 5–10).

  • Cheng, X., Chen, X., & Mallat, S. (2016). Deep haar scattering networks. Information and Inference: A Journal of IMA, 5, 105–133.

    Article  MathSciNet  Google Scholar 

  • Chinchali, S., Hu, P., Chu, T., Sharma, M., Bansal, M., Misra, R., Pavone, M., & Katti, S. (2018). Cellular network traffic scheduling with deep reinforcement learning. AAAI.

  • Crum, W. R., Hartkens, T., & Hill, D. L. G. (2004). Non-rigid image registration: Theory and practice. The British Journal of Radiology, 77(2), 140–153.

    Article  Google Scholar 

  • Davis, L. M. (2013). Predictive modelling of bone ageing. Unpublished doctoral dissertation, University of East Anglia.

  • de Marcelo, S. P., et al. (2017). Fault identification in doubly fed induction generator using FFT and neural networks. Journal of Control, Automation and Electrical Systems, 28, 228–237.

    Article  Google Scholar 

  • Dosiek, L., Pierre, J. W., & Follum, J. (2013). A recursive maximum likelihood estimator for the online estimation of electromechanical modes with error bounds. IEEE Transactions on Power Systems, 28(1), 441–451.

    Article  Google Scholar 

  • Dubois, S. R., & Glanz, F. H. (1986). An autoregressive model approach to two-dimensional shape classification. IEEE Transactions PAMI, 8(1), 55–66.

    Article  Google Scholar 

  • Dulakshi, S. K., & Liong, S. Y. (2006). Chaotic time series prediction with a global model: Artificial neural network. Journal of Hydrology, 323(1–4), 92–105.

    Google Scholar 

  • Giordano, F., Rocca, M., & Perna, C. (2008). Neural network sieve bootstrap prediction intervals: Some real data evidence. In New directions in neural networks: 18th Italian workshop on neural networks: WIRN 2008.

  • Glaessgen, E. H., & Stargel, D. S. (2012) The digital twin paradigm for future NASA and U.S. air force vehicles. In 53rd Structures, structural dynamics, and materials conference.

  • Gregor, K., Danihelka, I., Mnih, A., Blundell, C., & Wierstra, D. (2014). Deep autoregressive networks. PMLR, 32(2), 1242–1250.

    Google Scholar 

  • Grenander, U., & Miller, M. I. (1994). Representations of knowledge in complex systems. Journal of the Royal Statistical Society B, 56(4), 549–603.

    MathSciNet  MATH  Google Scholar 

  • Hayashi, F. (2000). Econometrics. Princeton, NJ: Princeton University Press.

    MATH  Google Scholar 

  • He, K., Zhang, X., Shaoqing, R., Sun, J., & Kavukcuoglu, K. (2015). Deep residual learning for image recognition. arXiv:1512.03385v1.

  • Jalba, A. C., Wilkinson, M. H. F., & Roerdink, J. B. T. M. (2004). Automatic segmentation of diatom images for classification. Microscopy Research and Technique, 65, 72–85.

    Article  Google Scholar 

  • Jzefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016). Exploring the limits of language modeling. arXiv:1602.02410v2.

  • Keogh, E., Wei, L., Xi, X., Vlachos, M., Lee, S., & Protopapas, P. (2006). Supports exact indexing of shapes under rotation invariance with arbitrary representations and distance measures. In VLDB ’06: Proceedings of the 32nd International Conference on Very Large Data Bases (pp.882–893).

  • Klein, A., et al. (2009). Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage, 46(3), 786–802.

    Article  Google Scholar 

  • Kolman, E., & Margaliot, M. (2007). Knowledge extraction from neural networks using all-permutation fuzzy rule base. IEEE Transactions on Neural Networks, 18, 925–931.

    Article  Google Scholar 

  • Kou, S. G. (2002). A jump-diffusion model for option pricing. Management Science, 48(8), 1086–1101.

    Article  Google Scholar 

  • Lee, J., Bagheri, B., & Kao, H.-A. (2015). A cyber-physical systems architecture for industry 4.0-based manufacturing systems. Manufacturing Letters, 3, 18–23. https://doi.org/10.1016/j.mfglet.2014.12.001.

    Article  Google Scholar 

  • Li, L., McCann, J., Pollard, N., & Faloutsos, C. (2009). DynaMMo: mining and summarization of coevolving sequences with missing values. KDD’09. Paris, France.

  • Luong, Q.-T., Weber, J., Koller, D., & Malik, J. (1995). An integrated stereo-based approach to automatic vehicle guidance. In Proceedings of IEEE International Conference on Computer Vision.

  • Matsuda, K. (2004). Introduction to Merton jump diffusion model. Working Paper.

  • May, R. M. (1976). Simple mathematical models with very complicated dynamics. Nature, 261, 459–467.

    Article  Google Scholar 

  • Meer, P., Mintz, D., Rosenfeld, A., & Kim, D. Y. (1991). Robust regression methods for computer vision: A review. Int J Comput Vision, 6, 59.

    Article  Google Scholar 

  • Monteiro, N. A. B., Silva, J. J., & Neto, J. S. R. (2019). Soft sensors to monitoring a multivariate nonlinear process using neural networks. Journal of Control, Automation and Electrical Systems, 30, 54–62.

    Article  Google Scholar 

  • Neto, F. F. (2017). Building function approximators on top of Haar scattering networks. Working Paper on Research Net.

  • Nowicka-Zagrajek, J., & Weron, R. (2002). Modeling electricity loads in California: ARMA models with hyperbolic noise. Signal Processing, 82, 1903–1915.

    Article  Google Scholar 

  • Okuma, K., Taleghani, A., de Freitas, N., Little, J. J., & Lowe, D. G. (2004). A boosted particle filter: Multitarget detection and tracking. In Computer vision—ECCV (pp. 28–39).

  • Pareek, N. K., Patidar, V., & Sud, K. K. (2006). Image encryption using chaotic logistic map. Image and Vision Computing, 24, 926–934.

    Article  Google Scholar 

  • Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017). Mastering the game of Go without human knowledge. Nature, 550(19), 354–372.

    Article  Google Scholar 

  • Soofi, A. S., & Cao, L. (2002). Modelling and forecasting financial data: techniques of nonlinear dynamics. New York: Springer.

    Book  Google Scholar 

  • Specht, J. I., Dimieri, L., Urdapilleta, E., & Gasaneo, G. (2017). Minimal dynamical description of eye movements. The European Physical Journal B, 90, 25.

    Article  MathSciNet  Google Scholar 

  • Tak, Y. S., & Hwang, E. (2007). A leaf image retrieval scheme based on partial dynamic time warping and two-level filtering. In Seventh international conference on computer and information technology.

  • Tang, W. K., & Wong, Y. K. (2000). Simulation of manufacturing processes using ARMA and ARMAX models. Advances in Modelling and Analysis B, 43, 1–2.

    Google Scholar 

  • Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., & Sui, F. (2017). Digital twin-driven product design, manufacturing and service with big data. The International Journal of Advanced Manufacturing Technology, 94(9–12), 3563–3576. https://doi.org/10.1007/s00170-017-0233-1.

    Article  Google Scholar 

  • Thirion, J.-P. (1998). Image matching as a diffusion process: An analogy with Maxwell’s demons. Medical Image Analysis, 2(3), 243–260.

    Article  Google Scholar 

  • Vakulenko, S., Radulescu, O., Morozov, I., & Weber, A. (2017). Centralized networks to generate human body motions. Sensors (Basel), 17(12), E2907.

    Article  Google Scholar 

  • van den Oord, A., Dieleman, S., Espeholt, L., Vinyals, O., Graves, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016a). Conditional image generation with PixelCNN decoders. arXiv:1606.05328v2.

  • van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016b). WaveNet: a generative model for raw audio. arXiv:1609.03499v2.

  • Wang, L., Hu, W., & Tan, T. (2003). Recent developments in human motion analysis. Pattern Recognition, 36(3), 585–601.

    Article  Google Scholar 

  • Zhang, G. P. (2003). Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing, 50, 159–175.

    Article  Google Scholar 

  • Zitov, B., & Flusser, J. (2003). Image registration methods: A survey. Image and Vision Computing, 21(11), 977–1000.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universidade de São Paulo (USP), Avenida Prof. Luciano Gualberto, 908; Cidade Universitária, São Paulo, Brazil

    Fernando Fernandes Neto & Rodrigo de Losso da Silveira Bueno

  2. Department of Physics and Astronomy, Rio de Janeiro State University, R. Sao Francisco Xavier - 524, Rio de Janeiro, 20559-900, Brazil

    Pedro Delano Cavalcanti

  3. Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, 08854, USA

    Alemayehu Solomon Admasu

Authors
  1. Fernando Fernandes Neto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rodrigo de Losso da Silveira Bueno
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pedro Delano Cavalcanti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alemayehu Solomon Admasu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando Fernandes Neto.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work at Rutgers University was supported in part by the Van Dyck fund under the school of Graduate Studies and the Department of Physics and Astronomy.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fernandes, F., Bueno, R.d.L.d.S., Cavalcanti, P.D. et al. Generating Stochastic Processes Through Convolutional Neural Networks. J Control Autom Electr Syst 31, 294–303 (2020). https://doi.org/10.1007/s40313-020-00567-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40313-020-00567-y

Keywords

Search

Navigation