Lamaakal Ismail, Maleh Yassine, El Makkaoui Khalid, Ouahbi Ibrahim, Pławiak Paweł, Alfarraj Osama, Almousa May, Abd El-Latif Ahmed A
Multidisciplinary Faculty of Nador, Mohammed Premier University, Oujda 60000, Morocco.
National School of Applied Sciences, Sultan Moulay Slimane University, Beni Mellal 23000, Morocco.
Sensors (Basel). 2025 Feb 21;25(5):1318. doi: 10.3390/s25051318.
Large Language Models (LLMs), like GPT and BERT, have significantly advanced Natural Language Processing (NLP), enabling high performance on complex tasks. However, their size and computational needs make LLMs unsuitable for deployment on resource-constrained devices, where efficiency, speed, and low power consumption are critical. Tiny Language Models (TLMs), also known as BabyLMs, offer compact alternatives by using advanced compression and optimization techniques to function effectively on devices such as smartphones, Internet of Things (IoT) systems, and embedded platforms. This paper provides a comprehensive survey of TLM architectures and methodologies, including key techniques such as knowledge distillation, quantization, and pruning. Additionally, it explores potential and emerging applications of TLMs in automation and control, covering areas such as edge computing, IoT, industrial automation, and healthcare. The survey discusses challenges unique to TLMs, such as trade-offs between model size and accuracy, limited generalization, and ethical considerations in deployment. Future research directions are also proposed, focusing on hybrid compression techniques, application-specific adaptations, and context-aware TLMs optimized for hardware-specific constraints. This paper aims to serve as a foundational resource for advancing TLMs capabilities across diverse real-world applications.
像GPT和BERT这样的大语言模型（LLMs）极大地推动了自然语言处理（NLP）的发展，使其在复杂任务上具备高性能。然而，它们的规模和计算需求使得大语言模型不适用于在资源受限的设备上部署，而在这些设备上，效率、速度和低功耗至关重要。小语言模型（TLMs），也被称为微型语言模型，通过使用先进的压缩和优化技术，提供了紧凑的替代方案，以便在智能手机、物联网（IoT）系统和嵌入式平台等设备上有效运行。本文对小语言模型的架构和方法进行了全面综述，包括知识蒸馏、量化和剪枝等关键技术。此外，还探讨了小语言模型在自动化和控制领域的潜在及新兴应用，涵盖边缘计算、物联网、工业自动化和医疗保健等领域。该综述讨论了小语言模型所特有的挑战，例如模型规模与准确性之间的权衡、有限的泛化能力以及部署中的伦理考量。还提出了未来的研究方向，重点关注混合压缩技术、针对特定应用的适配以及针对硬件特定约束进行优化的上下文感知小语言模型。本文旨在成为推动小语言模型在各种实际应用中能力提升的基础资源。
