TLDR: This survey paper systematically categorizes authorship attribution in the era of LLMs into four problems: attributing unknown texts to human authors, detecting LLM-generated texts, identifying specific LLMs or human authors, and classifying texts as human-authored, machine-generated, or co-authored by both, while also highlighting key challenges and open problems.
Accurate attribution of authorship is crucial for maintaining the integrity of digital content, improving forensic investigations, and mitigating the risks of misinformation and plagiarism. Addressing the imperative need for proper authorship attribution is essential to uphold the credibility and accountability of authentic authorship. The rapid advancements of Large Language Models (LLMs) have blurred the lines between human and machine authorship, posing significant challenges for traditional methods. We present a comprehensive literature review that examines the latest research on authorship attribution in the era of LLMs. This survey systematically explores the landscape of this field by categorizing four representative problems: (1) Human-written Text Attribution; (2) LLM-generated Text Detection; (3) LLM-generated Text Attribution; and (4) Human-LLM Co-authored Text Attribution. We also discuss the challenges related to ensuring the generalization and explainability of authorship attribution methods. Generalization requires the ability to generalize across various domains, while explainability emphasizes providing transparent and understandable insights into the decisions made by these models. By evaluating the strengths and limitations of existing methods and benchmarks, we identify key open problems and future research directions in this field. This literature review serves a roadmap for researchers and practitioners interested in understanding the state of the art in this rapidly evolving field.
The table below is a summary of Authorship Attribution Datasets and Benchmarks with LLM-Generated Text. Size is shown as the sum of LLM-generated and human-written texts (with the percentage of human-written texts in parentheses). Language is displayed using the two-letter ISO 639 abbreviation. Columns P2, P3, and P4 indicate whether the dataset supports problems described in Problem 2, 3, and 4, respectively.
Name | Domain | Size | Length | Language | Model | P2 | P3 | P4 |
---|---|---|---|---|---|---|---|---|
TuringBench | News | 168,612 (5.2%) | 100 to 400 words | en | GPT-1,2,3, GROVER, CTRL, XLM, XLNET, FAIR, TRANSFORMER-XL, PPLM | ✓ | ✓ | |
TweepFake | Social media | 25,572 (50.0%) | less than 280 characters | en | GPT-2, RNN, Markov, LSTM, CharRNN | ✓ | ||
ArguGPT | Academic essays | 8,153 (49.5%) | 300 words on average | en | GPT2-Xl, text-babbage-001, text-curie-001, davinci-001,002,003, GPT-3.5-Turbo | ✓ | ||
AuTexTification | Tweets, reviews, news, legal, and how-to articles | 163,306 (42.5%) | 20 to 100 tokens | en, es | BLOOM, GPT-3 | ✓ | ✓ | |
CHEAT | Academic paper abstracts | 50,699 (30.4%) | 163.9 words on average | en | ChatGPT | ✓ | ||
GPABench2 | Academic paper abstracts | 2.385M (6.3%) | 70 to 350 words | en | ChatGPT | ✓ | ✓ | |
Ghostbuster | News, student essays, creative writing | 23,091 (87.0%) | 77 to 559 (median words per document) | en | ChatGPT, Claude | ✓ | ||
HC3 | Reddit, Wikipedia, medicine, finance | 125,230 (64.5%) | 25 to 254 words | en, zh | ChatGPT | ✓ | ||
HC3 Plus | News, social media | 214,498 | N/A | en, zh | ChatGPT | ✓ | ||
HC-Var | News, reviews, essays, QA | 144k (68.8%) | 50 to 200 words | en | ChatGPT | ✓ | ||
HANSEN | Transcripts of speech (spoken text), statements (written text) | 535k (96.1%) | less than 1k tokens | en | ChatGPT, PaLM2, Vicuna-13B | ✓ | ✓ | |
M4 | Wikipedia, WikiHow, Reddit, QA, news, paper abstracts, peer reviews | 147,895 (24.2%) | more than 1k characters | ar, bg, en, id, ru, ur, zh | davinci-003, ChatGPT, GPT-4, Cohere, Dolly2, BLOOMz | ✓ | ||
MGTBench | News, student essays, creative writing | 21k (14.3%) | 1 to 500 words | en | ChatGPT, ChatGLM, Dolly, GPT4All, StableLM, Claude | ✓ | ✓ | |
MULTITuDE | News | 74,081 (10.8%) | 200 to 512 tokens | ar, ca, cs, de, en, es, nl, pt, ru, uk, zh | GPT-3,4, ChatGPT, Llama-65B, Alpaca-LoRa-30B, Vicuna-13B, OPT-66B, OPT-IML-Max-1.3B | ✓ | ||
OpenGPTText | OpenWebText | 58,790 (50.0%) | less than 2k words | en | ChatGPT | ✓ | ||
OpenLLMText | OpenWebText | 344,530 (20%) | 512 tokens | en | ChatGPT, PaLM, Llama, GPT2-XL | ✓ | ✓ | |
Scientic Paper | Scientific papers | 29k (55.2%) | 900 tokens on average | en | SCIgen, GPT-2,3, ChatGPT, Galactica | ✓ | ||
RAID | News, Wikipedia, paper abstracts, recipes, Reddit, poems, book summaries, movie reviews | 523,985 (2.9%) | 323 tokens on average | cs, de, en | GPT-2,3,4, ChatGPT, Mistral-7B, MPT-30B, Llama2-70B, Cohere command and chat | ✓ | ||
M4GT-Bench | Wikipedia, Wikihow, Reddit, arXiv abstracts, academic paper reviews, student essays | 5,368,998 (96.6%) | more than 50 characters | ar, bg, de, en, id, it, ru, ur, zh | ChatGPT, davinci-003, GPT-4, Cohere, Dolly-v2, BLOOMz | ✓ | ✓ | ✓ |
MAGE | Reddit, reviews, news, QA, story writing, Wikipedia, academic paper abstracts | 448,459 (34.4%) | 263 words on average | en | GPT, Llama, GLM-130B, FLAN-T5 OPT, T0, BLOOM-7B1, GPT-J-6B, GPT-NeoX-2 | ✓ | ||
MIXSET | Email, news, game reviews, academic paper abstracts, speeches, blogs | 3.6k (16.7%) | 50 to 250 words | en | GPT-4, Llama2 | ✓ | ✓ |
The Table below presents an overview of LLM-Generated Text Detectors.
Detector | Price | API | Website |
---|---|---|---|
GPTZero | 150k words at $10/month, 10k words for free per month | Yes | https://gptzero.me/ |
ZeroGPT | 100k characters for $9.99, 15k characters for free | Yes | https://www.zerogpt.com/ |
Sapling | 50k characters for $25, 2k characters for free | Yes | https://sapling.ai/ai-content-detector |
Originality.AI | 200k words at $14.95/month | Yes | https://originality.ai/ |
CopyLeaks | 300k words at $7.99/month | Yes | https://copyleaks.com/ai-content-detector |
Winston | 80k words at $12/month | Yes | https://gowinston.ai/ |
GPT Radar | $0.02/100 tokens | N/A | https://gptradar.com/ |
Turnitin’s AI detector | License required | N/A | https://www.turnitin.com/solutions/topics/ai-writing/ai-detector/ |
GPT-2 Output Detector | Free | N/A | https://github.com/openai/gpt-2-output-dataset/tree/master/detector |
Crossplag | Free | N/A | https://crossplag.com/ai-content-detector/ |
CatchGPT | Free | N/A | https://www.catchgpt.ai/ |
Quil.org | Free | N/A | https://aiwritingcheck.org/ |
Scribbr | Free | N/A | https://www.scribbr.com/ai-detector/ |
Draft Goal | Free | N/A | https://detector.dng.ai/ |
Writefull | Free | Yes | https://x.writefull.com/gpt-detector |
Phrasly | Free | Yes | https://phrasly.ai/ai-detector |
Writer | Free | Yes | https://writer.com/ai-content-detector |
@article{huang2024aa_llm,
title = {Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges},
author = {Baixiang Huang and Canyu Chen and Kai Shu},
year = {2024},
journal = {arXiv preprint arXiv: 2408.08946},
url = {https://arxiv.org/abs/2408.08946},
}
The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis remains under-explored. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) How can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our assessment demonstrates LLMs' proficiency in both tasks without the need for domain-specific fine-tuning, providing insights into their decision-making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis.
@artile{huang2024authorship,
title = {Can Large Language Models Identify Authorship?},
author = {Baixiang Huang and Canyu Chen and Kai Shu},
year = {2024},
journal = {arXiv preprint},
volume = {abs/2403.08213},
url = {https://arxiv.org/abs/2403.08213},
}