Abstract
Multivariate temporal data generally exists in the whole manufacturing process and forecasting lays the foundation for many intelligent services in industry. In this paper, we propose an end-to-end deep learning framework named dual-dimensional attention-based network (DANet) to solve the multivariate time series forecasting problem in industry. It leverages the strengths of recurrent neural network (RNN) structures to discover the underlying temporal patterns of the multi-dimensional input. A recurrent module is used for capturing sequential relationships between adjacent timesteps and embedding the original observations. Then, we apply a novel dual-dimensional attention mechanism to cope with the intrinsic characteristics of industrial big data. Feature-wise self-attention enables the network to adaptively learn the correlations between features, while time-wise attention captures complex long and short-term temporal dependencies. Our model shows its advantages over the baseline methods and a more stable and robust performance in the experiments on several real-world manufacturing datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: International Conference on Learning Representations (ICLR) (2015)
Cho, K., et al.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734 (2014)
Dogan, A., Birant, D.: Machine learning and data mining in manufacturing. Expert Syst. Appl. 166, 114060 (2021)
Dudek, G.: Pattern-based local linear regression models for short-term load forecasting. Electric Power Syst. Res. 130, 139–147 (2016)
Ghahramani, M., Qiao, Y., Zhou, M., O’Hagan, A., Sweeney, J.: AI-based modeling and data-driven evaluation for smart manufacturing processes. IEEE/CAA J. Automatica Sinica 7(4), 1026–1037 (2020)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hsieh, T.Y., Sun, Y., Tang, X., Wang, S., Honavar, V.G.: SrVARM: state regularized vector autoregressive model for joint learning of hidden state transitions and state-dependent inter-variable dependencies from multi-variate time series. In: Proceedings of the Web Conference (WWW), pp. 2270–2280 (2021)
Hsu, C., Liu, W.: Multiple time-series convolutional neural network for fault detection and diagnosis and empirical study in semiconductor manufacturing. J. Intell. Manuf. 32(3), 823–836 (2021)
Huang, S., Wang, D., Wu, X., Tang, A.: DSANet: dual self-attention network for multivariate time series forecasting. In: ACM International Conference on Information and Knowledge Management (CIKM), pp. 2129–2132 (2019)
Kaggle: multi-stage continuous flow manufacturing process. Website (2020). https://www.kaggle.com/supergus/multistage-continuousflow-manufacturing-process
Lai, G., Chang, W., Yang, Y., Liu, H.: Modeling long- and short-term temporal patterns with deep neural networks. In: ACM International Conference on Research and Development in Information Retrieval (SIGIR), pp. 95–104 (2018)
Lepenioti, K., et al.: Machine learning for predictive and prescriptive analytics of operational data in smart manufacturing. In: International Conference on Advanced Information Systems Engineering Workshops (CAiSE), vol. 382, pp. 5–16 (2020)
Li, J., Yang, B., Li, H., Wang, Y., Qi, C., Liu, Y.: DTDR-ALSTM: extracting dynamic time-delays to reconstruct multivariate data for improving attention-based LSTM industrial time series prediction models. Knowl.-Based Syst. 211, 106508 (2021)
Li, M., Hua, Z., Zhao, J., Zou, Y., Xie, B.: ARIMA model-based web services trustworthiness evaluation and prediction. In: International Conference on Service-Oriented Computing (ICSOC), pp. 648–655 (2012)
Liu, C., Mao, W., Gao, Y., Gao, X., Li, S., Chen, G.: Adaptive recollected RNN for workload forecasting in database-as-a-service. In: International Conference on Service-Oriented Computing (ICSOC), pp. 431–438 (2020)
Sen, R., Yu, H., Dhillon, I.S.: Think globally, act locally: a deep neural network approach to high-dimensional time series forecasting. In: International Conference on Neural Information Processing Systems (NeurIPS), pp. 4838–4847 (2019)
Shih, S., Sun, F., Lee, H.: Temporal pattern attention for multivariate time series forecasting. Mach. Learn. 108(8–9), 1421–1441 (2019)
Vaswani, A., et al.: Attention is all you need. In: International Conference on Neural Information Processing Systems (NeurIPS), pp. 6000–6010 (2017)
Wang, J., Wang, K., Wang, Y., Huang, Z., Xue, R.: Deep Boltzmann machine based condition prediction for smart manufacturing. J. Ambient. Intell. Humaniz. Comput. 10(3), 851–861 (2019)
Wang, J., Yang, J., Zhang, J., Wang, X., Zhang, W.C.: Big Data Driven Cycle Time Parallel Prediction for Production Planning in Wafer Manufacturing. Enterprise Inf. Syst. 12(6), 714–732 (2018)
Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. Computing Research Repository (CoRR) abs/2012.07436 (2020)
Acknowledgements
This work was supported by the National Key R&D Program of China [2020YFB1707903]; the National Natural Science Foundation of China [61872238, 61972254], Shanghai Municipal Science and Technology Major Project [2021SHZDZX0102], the Tencent Marketing Solution Rhino-Bird Focused Research Program [FR202001], and the CCF-Tencent Open Fund [RAGR20200105].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, X., Gao, X. (2021). An Attention-Based Forecasting Network for Intelligent Services in Manufacturing. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_67
Download citation
DOI: https://doi.org/10.1007/978-3-030-91431-8_67
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91430-1
Online ISBN: 978-3-030-91431-8
eBook Packages: Computer ScienceComputer Science (R0)