Case Study

Banking Intent Classification

Project Overview

Reducing the operational risk of misrouted customer queries — an intent classification pipeline that automates triage and compares transformer and traditional models for reliable, auditable routing. This project addresses the operational risk of manually triaging high-volume customer support queries in the banking sector. Using the Banking77 dataset, which contains over 13,000 queries across 77 unique intent categories, we implemented and compared two distinct approaches to text classification to reduce misrouting, delay, and inconsistent handling.

Python PyTorch Transformers (RoBERTa) PEFT (LoRA) Scikit-Learn

Business Problem

Manually triaging thousands of customer queries introduces routing errors, compliance exposure, and inconsistent service — key operational risks. An accurate intent classification system reduces these risks through consistent routing, automated self-service, and auditable decision trails that improve both customer satisfaction and regulatory consistency.

Project Goals

  • Develop a baseline classification model using traditional machine learning techniques.
  • Implement a state-of-the-art transformer-based model using Parameter-Efficient Fine-Tuning (PEFT).
  • Compare performance trade-offs between speed, complexity, and accuracy.
  • Demonstrate the efficacy of Low-Rank Adaptation (LoRA) for natural language tasks.
  • Reduce misclassification risk for high-stakes financial queries through human-in-the-loop escalation of low-confidence predictions.

Dataset Overview

The Banking77 dataset is a fine-grained intent detection dataset specifically for the banking domain. It contains 13,083 customer service queries labeled with one of 77 intents, such as "activate_card", "balance_not_updated_after_bank_transfer", or "wrong_amount_of_cash_received".

Distribution of Intent Labels

Figure 1: Distribution of intent labels in the Banking77 dataset, showing the frequency of queries across all 77 categories.

Methodology

We adopted a comparative research design to evaluate the performance gain provided by modern LLM fine-tuning techniques over traditional architectures.

Data Preprocessing

For the baseline model, we used TF-IDF (Term Frequency-Inverse Document Frequency) vectorization to transform text into numerical features. For the RoBERTa model, we utilized the RoBERTa tokenizer to convert raw text into input IDs and attention masks compatible with the transformer architecture.

Model Architectures

Baseline: Multi-Layer Perceptron (MLP)

  • Input: TF-IDF vectors.
  • Architecture: Fully connected dense layers with ReLU activation.
  • Training: Fast, CPU-efficient, but lacks semantic understanding of word order.

Advanced: RoBERTa + LoRA

  • Foundation Model: roberta-base from Hugging Face.
  • PEFT Technique: Low-Rank Adaptation (LoRA) was applied to the query and value projections in the attention layers.
  • Efficiency: Reduced the number of trainable parameters to less than 1% of the total model size, enabling faster training and lower memory usage while maintaining performance.
Text Length Distribution

Figure 2: Text length distribution by intent category, highlighting the variability in customer query lengths across different intents.

Results

The comparison showed that the transformer approach materially reduces misclassification risk in complex intent categories, where routing errors carry the greatest operational impact. The traditional MLP approach and the fine-tuned RoBERTa model differed significantly in how reliably they handled nuanced financial queries.

Key Findings

  • Accuracy: RoBERTa-LoRA significantly outperformed the MLP baseline, achieving 93.67% accuracy compared to 87.79% for the baseline.
  • Macro F1: The transformer model reached 93.67% Macro F1, a 5.85% improvement over the MLP's 87.82%.
  • Semantic Nuance: The model excelled in complex categories like contactless_not_working (96.10% F1 vs 72.72% for MLP).
Word Clouds for Top Intents

Figure 3: Word clouds showing the most frequent terms across key banking intent categories, illustrating the semantic themes the models must distinguish.

Model Accuracy Macro F1 Weighted F1
MLP (Baseline) 0.8779 0.8782 0.8782
LoRA-RoBERTa 0.9367 0.9367 0.9367

Recommendations

Based on the findings of this project, we provide the following recommendations for implementing automated intent classification in a production environment.

Model Selection

For production banking applications where accuracy is critical, RoBERTa-LoRA should be favored over simpler MLP architectures. The semantic understanding provided by transformers is essential for handling the complexity of financial queries.

Implementation Strategy

  • Utilize PEFT: Implement LoRA or similar techniques to keep infrastructure costs low and deployment speeds high.
  • Continuous Learning: Regularly update the model with new, anonymized customer queries to adapt to changing banking trends and terminology.
  • Human-in-the-Loop: Route low-confidence predictions to human agents — a key control that contains model risk and ensures quality for high-stakes financial queries — while providing training data for future model versions.

Jupyter Notebook

The complete implementation, including data loading, model training, and performance evaluation, is available in the embedded notebook below. This workflow demonstrates the practical use of the Hugging Face ecosystem for advanced NLP tasks.

Loading notebook... This may take a moment.

← Back to Projects