Joint multilingual adaptive attention fusion based multi-teacher KD with contrastive learning for Indic LoRes cross-domain, multi-intent NLU

With the rapid advancements in conversational AI, ensuring efficiency, scalability and accuracy in NLU remains a challenge, particularly for cross-domain user utterances expressing multiple complex intents in low-resource multilingual settings.LLMs require extensive computational resources, leading to high costs, slow inference, and scalability challenges in resource-constrained environments. To address these issues, we propose a multilingual, multitask Knowledge Distillation (KD) framework for Intent Detection (ID), Domain Classification (DC), and Slot Filling (SF) across six low-resource Indic languages: Tamil, Telugu, Kannada, and Malayalam from the Dravidian family, and Hindi and Bengali from the Indo-Aryan family. Our approach explores two KD architectures for low-resource NLU: a joint multitask, single-teacher model distilling knowledge into a unified student model, and a multitask student framework leveraging an adaptive attention-based fusion mechanism along with temperature scaling to learn from three multitask, multi-teacher models (ID-DC, ID-SF, DC-SF). The student model, optimized with KD Loss, Cross-Entropy Loss, Mean Squared Error Loss, and Contrastive Learning, achieves superior accuracy and efficiency while reducing computational overhead. Extensive evaluations on MASSIVE and a custom multilingual dataset demonstrate that multitask-based multi-teacher KD enhances task performance and accuracy, improving inference time and making it a viable solution for real-world multilingual NLU applications in resource-constrained environments.