Utterance-Level Dialogue Understanding: Task Definition and Approach
This repository provides PyTorch implementations for baseline models introduced in the paper Utterance-level Dialogue Understanding: An Empirical Study, published in 2020. The task operates on structured dialogue transcripts, where each utterance is paired with speaker information. Formally, the input consists of a sequence of N utterances, represented as [(u1, p1), (u2, p2), ..., (uN, pN)], where each utterance ui contains T words and is spoken by party pi. The goal is to predict a label ei for each utterance ui, leveraging conversational context where applicable. Some dialogues may lack labels for all utterances, requiring selective classification.
The task distinguishes between two primary modes of context utilization: dialogue-level and utterance-level minibatch processing. This distinction allows for experiments assessing the impact of context granularity on classification performance.
The datasets used in this study include IEMOCAP and DailyDialog for emotion recognition, MultiWOZ for intent recognition, and Persuasion for Good for act classification. These datasets are provided in both original and context-perturbed formats, such as style transfer, paraphrasing, label augmentation, and spelling attacks.
The paper also explores the role of context in utterance-level classification through controlled experiments. For instance, models are trained with shuffled contexts during training, validation, or testing to evaluate their dependency on utterance order and sequential dependencies. This approach helps isolate the influence of contextual cues on classification accuracy.
Data Format and Preprocessing
The repository organizes data into tab-separated text files, each adhering to a standardized format. Four types of files are required for training: utterances, labels, loss masks, and speaker information. Utterance files contain dialogue IDs followed by the text of each utterance in sequence. Label files encode the corresponding emotion, act, intent, or strategy labels for each utterance, while loss masks determine which utterances contribute to loss calculations. For example, in MultiWOZ intent classification, system-generated utterances may be excluded from classification despite being included in the input context.
Speaker files associate each utterance with its speaker ID, enabling models to incorporate speaker-specific features. The datasets are split into original and perturbed versions, with perturbations applied to assess model sensitivity to stylistic, lexical, and structural variations in dialogue.
Dialogue-level minibatches process entire dialogues, classifying all labeled utterances within a single forward pass. Utterance-level minibatches, conversely, focus on one utterance per minibatch, using the surrounding context for classification. This distinction is critical for evaluating how models utilize local versus global conversational context.
Model Architectures and Implementation Details
The bcLSTM and DialogueRNN architectures support optional residual connections, which can improve gradient flow and training stability.
Two feature extraction backbones are supported: GloVe embeddings and RoBERTa-based transformers. GloVe-based models are implemented in the glove-end-to-end directory, while RoBERTa-based models reside in roberta-end-to-end. The latter can be extended to use BERT or Sentence Transformers as feature extractors. Scripts for running analysis experiments, as reported in the paper, are provided alongside the implementations.
Training commands vary depending on the model, dataset, and context strategy. For dialogue-level minibatch processing, the command python train.py is used with arguments specifying the dataset, classification task (emotion, act, intent), and model type (logreg, lstm, or dialogrnn). Residual connections can be enabled with the --residual flag. Utterance-level minibatch training uses python trainuttlevel.py, while speaker-level models (intra-speaker or inter-speaker context) are trained via python trainintraspeaker.py or python traininterspeaker.py, respectively.
Additional experiments involve shuffled context training, implemented via python trainshuffledcontext.py.
Key Features and Capabilities
- Multi-Task Classification: Supports emotion recognition, dialogue act classification, intent detection, and persuasion strategy analysis across datasets like IEMOCAP, DailyDialog, MultiWOZ, and Persuasion for Good.
- Contextual Flexibility: Models can be trained with or without conversational context, enabling comparisons between context-aware and context-agnostic approaches.
- Perturbation Analysis: Includes datasets with stylistic, lexical, and structural variations to evaluate model robustness under adversarial or noisy conditions.
- Residual Connections: Optional residual connections in bcLSTM and DialogueRNN architectures to mitigate vanishing gradients and improve training efficiency.
- Feature Extraction Variants: Compatible with both GloVe embeddings and transformer-based models (RoBERTa, BERT, Sentence Transformers), allowing for comparisons between traditional and modern embedding techniques.
- Minibatch Strategies: Provides dialogue-level and utterance-level minibatch formation to study the impact of context granularity on classification performance.
- Reproducibility: Scripts and datasets are structured to facilitate replication of experiments reported in the associated papers, with baseline results documented for reference.