$\textbf{\emph{CLMSM}}$: A Multi-Task Learning Framework for Pre-training on Procedural Text

Abhilash Nandy; Manav Nitin Kapadnis; Pawan Goyal; Niloy Ganguly

EMNLP 2023long findings0 citations

$\textbf{\emph{CLMSM}}$: A Multi-Task Learning Framework for Pre-training on Procedural Text

Abhilash Nandy, Manav Nitin Kapadnis, Pawan Goyal, Niloy Ganguly

Abstract

In this paper, we propose ***CLMSM***, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. ***CLMSM*** uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of ***CLMSM*** on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that ***CLMSM*** not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.

pre-trainingprocedural reasoningcontrastive learningmasked language modelingmulti-task learningnlp

BibTeX

@inproceedings{
nandy2023textbfemphclmsm,
title={\${\textbackslash}textbf\{{\textbackslash}emph\{{CLMSM}\}\}\$: A Multi-Task Learning Framework for Pre-training on Procedural Text},
author={Abhilash Nandy and Manav Nitin Kapadnis and Pawan Goyal and Niloy Ganguly},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=SP8zIwanHD}
}