Volume 16 • Issue 1 • PP: 152-165 • 2025
Human to Chatbot Text Classification Using Multi-Source AI Chatbots and Machine Learning Models
Open Access & Copyright
© 2025 The Author(s). Published by ASPG. This article is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
View PDF Abstract
The fast growth of artificial intelligence technologies, especially language processing technology has obscured the lines in between human-generated text comparing to chatbot-generated message. Recognizing which generated such, a text is essential for applications like information generating and manipulated text in order to guarantee authenticity between communicated parties. This research applies to a set of machine learning models to identify text as either human-written or chatbot-generated. The methodology of this research starts with a dataset including text generated from different Large Language Models (LLMs) along with a text generated by a human. After that, Tf-Idf ranking vectorization was used to define word embedding has and represent the text numerically. Then, different Machine Learning (ML) models leveraged recognize whether a human or a chatbot generated a text. The ML models applied include Logistic Regression, Random Forest, Decision Tree, Gradient Boosting, Naïve Bayes, and XGBoost. For this study accuracy, precision, recall, F1-score were used to evaluate the system. The dataset first was split into 80% for training and 20% for testing. Out of all implemented models, the Random Forest model reported the best with accuracy of 88%. Logistic Regression reported a close accuracy of 85%. The Random Forest model showed an 8% improvement compared to previous studies that reported an accuracy of 80%. Confusion matrices revealed that the Random Forest model provided high precision and recall, minimizing classification misleading of human or chatbot text. The research focused on studying the ability of ML models in identifying human vs. chatbot-generated text. The results showed the RF model was the best among other models with 88% accuracy. This accuracy shows a possible usage of such models in real-world applications that requires the confidentiality of human writing.
Keywords
References
[1] E. Lozić and B. Štular, “Fluent but Not Factual: A Comparative Analysis of ChatGPT and Other AI Chatbots’ Proficiency and Originality in Scientific Writing for Humanities,” Future Internet, vol. 15, no. 10, Art. no. 10, Oct. 2023, doi: 10.3390/fi15100336.
[2] K. I. Roumeliotis and N. D. Tselikas, “ChatGPT and Open-AI Models: A Preliminary Review,” Future Internet, vol. 15, no. 6, Art. no. 6, Jun. 2023, doi: 10.3390/fi15060192.
[3] A. Vaswani et al., “Attention Is All You Need,” Jul. 23, 2023, arXiv: 1706.03762. [Online]. Available: https://arxiv.org/abs/1706.03762. doi: 10.48550/arXiv.1706.03762.
[4] OpenAI et al., “GPT-4 Technical Report,” Mar. 04, 2024, arXiv: 2303.08774. [Online]. Available: https://arxiv.org/abs/2303.08774. doi: 10.48550/arXiv.2303.08774.
[5] E. Adamopoulou and L. Moussiades, “An Overview of Chatbot Technology,” in Advances in Intelligent Systems and Computing, vol. 1056, Cham, Switzerland: Springer, 2020, pp. 383–396, doi: 10.1007/978-3-030-49186-4_31.
[6] B. Galitsky, “Adjusting Chatbot Conversation to User Personality and Mood,” in Artificial Intelligence for Customer Relationship Management, Human–Computer Interaction Series, Cham, Switzerland: Springer, 2021, pp. 93–127, doi: 10.1007/978-3-030-61641-0_3.
[7] Z. Peng and X. Ma, “A survey on construction and enhancement methods in service chatbots design,” Springer, vol. 1, no. 3, pp. 204–223, 2019. [Online]. Available: https://link.springer.com/article/10.1007/s42486-019-00012-3.
[8] S. Izadi and M. Forouzanfar, “Error Correction and Adaptation in Conversational AI: A Review of Techniques and Applications in Chatbots,” AI, vol. 5, pp. 803–841, Jun. 2024, doi: 10.3390/ai5020041.
[9] T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing, “Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity,” May 29, 2023, arXiv: 2301.12867. [Online]. Available: https://arxiv.org/abs/2301.12867. doi: 10.48550/arXiv.2301.12867.
[10] Y. Huang et al., “TrustLLM: Trustworthiness in Large Language Models,” Sep. 30, 2024, arXiv: 2401.05561. [Online]. Available: https://arxiv.org/abs/2401.05561. doi: 10.48550/arXiv.2401.05561.
[11] T. R. Hannigan, I. P. McCarthy, and A. Spicer, “Beware of botshit: How to manage the epistemic risks of generative chatbots,” Elsevier, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0007681324000272.
[12] M. U. Hadi et al., “Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects,” Nov. 16, 2023, [Online]. Available: https://techrxiv.23589741.v4. doi: 10.36227/techrxiv.23589741.v4.
[13] S. Wyer and S. Black, “Algorithmic bias: sexualized violence against women in GPT-3 models,” AI Ethics, Jan. 2025, doi: 10.1007/s43681-024-00641-0.
[14] T. Choudhary, “Political Bias in Large Language Models: A Comparative Analysis of ChatGPT-4, Perplexity, Google Gemini, and Claude,” IEEE, 2024. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10817610/.
[15] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event Canada, ACM, Mar. 2021, pp. 610–623, doi: 10.1145/3442188.3445922.
[16] G. A. Godghase, R. Agrawal, T. Obili, and M. Stamp, “Distinguishing Chatbot from Human,” arXiv: 2408.04647v1, Jan. 19, 2025. [Online]. Available: https://arxiv.org/abs/2408.04647v1.
[17] I. Katib, F. Y. Assiri, H. A. Abdushkour, D. Hamed, and M. Ragab, “Differentiating Chat Generative Pretrained Transformer from Humans: Detecting ChatGPT-Generated Text and Human Text Using Machine Learning,” Mathematics, vol. 11, no. 15, Art. no. 15, Jan. 2023, doi: 10.3390/math11153400.
[18] X. Luo, S. Tong, Z. Fang, and Z. Qu, “Machines vs. Humans: The Impact of Artificial Intelligence Chatbot Disclosure on Customer Purchases,” Marketing Science, vol. 38, no. 6, pp. 937–947, Nov. 2019, doi: 10.1287/mksc.2019.1192.
[19] S. Mitrović, D. Andreoletti, and O. Ayoub, “ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text,” arXiv.org. [Online]. Available: https://arxiv.org/abs/2301.13852v1.
[20] I. J. Akpan, Y. M. Kobara, J. Owolabi, A. A. Akpan, and O. F. Offodile, “Conversational and Generative Artificial Intelligence and Human–Chatbot Interaction in Education and Research,” Int. Trans. Oper. Res., vol. 32, no. 3, pp. 1251–1281, 2025, doi: 10.1111/itor.13522.
[21] N. Prova, “Detecting AI Generated Text Based on NLP and Machine Learning Approaches,” Apr. 15, 2024, arXiv: 2404.10032. [Online]. Available: https://arxiv.org/abs/2404.10032. doi: 10.48550/arXiv.2404.10032.
[22] K. T. Repaka, M. A. Bondugula, and S. S. Adibhatla, “Benchmarking Distributed Machine Learning Systems with Large Language Models on Human vs. LLM Text Corpus,” Jan. 19, 2025. [Online]. Available: https://disml2024.github.io/disml-workshop-2024/assets/8_945276_86359389_Group8_DISML_Project_Report.pdf.
[23] Z. Grinberg, “Human vs. LLM Text Corpus,” Kaggle, Jan. 30, 2025. [Online]. Available: https://www.kaggle.com/dsv/7378735.
[24] F. Habibzadeh, “GPTZero Performance in Identifying Artificial Intelligence-Generated Medical Texts: A Preliminary Study,” J. Korean Med. Sci., vol. 38, no. 38, p. e319, Sep. 2023, doi: 10.3346/jkms.2023.38.e319.
Cite This Article
Choose your preferred format
Publisher's Note
The statements, opinions, and data presented in this article are solely those of the author(s) and do not necessarily represent those of ASPG, the journal, or its editors. ASPG and the editors disclaim responsibility for any harm arising from the use of any ideas, methods, instructions, or products described in this article, to the fullest extent permitted by applicable law.