Cross-Domain Detection of GPT-2-Generated Technical Text
Juan Diego Rodriguez, Todd Hay, David Gros, Zain Shamsi, Ravi Srinivasan
Abstract
Machine-generated text presents a potential threat not only to the public sphere, but also to the scientific enterprise, whereby genuine research is undermined by convincing, synthetic text. In this paper we examine the problem of detecting GPT-2-generated technical research text. We first consider the realistic scenario where the defender does not have full information about the adversary’s text generation pipeline, but is able to label small amounts of in-domain genuine and synthetic text in order to adapt to the target distribution. Even in the extreme scenario of adapting a physics-domain detector to a biomedical detector, we find that only a few hundred labels are sufficient for good performance. Finally, we show that paragraph-level detectors can be used to detect the tampering of full-length documents under a variety of threat models.
BibTeX
@inproceedings{rodriguez-etal-2022-cross,
title = "Cross-Domain Detection of {GPT}-2-Generated Technical Text",
author = "Rodriguez, Juan Diego and
Hay, Todd and
Gros, David and
Shamsi, Zain and
Srinivasan, Ravi",
editor = "Carpuat, Marine and
de Marneffe, Marie-Catherine and
Meza Ruiz, Ivan Vladimir",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.naacl-main.88/",
doi = "10.18653/v1/2022.naacl-main.88",
pages = "1213--1233"
}