Journal of Intelligent Systems and Internet of Things JISIoT 2690-6791 2769-786X 10.54216/JISIoT https://www.americaspg.com/journals/show/3076 2019 2019 Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Andhra Pradesh, India; Department of ECE, Sir C R Reddy College of Engineering, Eluru, India Nallamothu Nallamothu Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Andhra Pradesh, India Penke Satyanarayana Channel estimation poses critical challenges in millimeter-wave (mmWave) massive Multiple Input, Multiple Output (MIMO) communication models, particularly when dealing with a substantial number of antennas. Deep learning techniques have shown remarkable advancements in improving channel estimation accuracy and minimizing computational difficulty in 5G as well as the future generation of communications. The main intention of the suggested method is to use an optimal hybrid deep learning strategy to create a better channel estimation model. The proposed method, referred to as optimized D-LSTM, combines the power of a deep neural network (DNN) and long short-term memory (LSTM), and the optimization process involves the integration of the Reptile Search Algorithm (RSA) to enhance the performance of  deep learning model. The suggested hybrid deep learning method considers the correlation between the measurement matrix and the signal vectors that were received as input to predict the amplitude of the beam space channel. The newly proposed estimation model demonstrates remarkable superiority over traditional models in both Normalized Mean-Squared Error (NMSE) reduction and enhanced spectral efficiency. The spectral efficiency of the designed RSA-D-LSTM is 68.62%, 62.26%, 30.3%, and 19.77% higher than DOA, DHOA, HHO, and RSA. Therefore, the suggested system provides better channel estimation to improve its efficiency. 2024 2024 347 360 10.54216/JISIoT.130227 https://www.americaspg.com/articleinfo/18/show/3076