logo Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges

TLDR: This survey paper systematically categorizes authorship attribution in the era of LLMs into four problems: attributing unknown texts to human authors, detecting LLM-generated texts, identifying specific LLMs or human authors, and classifying texts as human-authored, machine-generated, or co-authored by both, while also highlighting key challenges and open problems.


Emory University
ACM SIGKDD Explorations 2024
framework
Representative Problems in Authorship Attribution:
  1. Human-written Text Attribution (attributing unknown texts to human authors)
  2. LLM-generated Text Detection (detecting if texts are generated by LLMs)
  3. LLM-generated Text Attribution (identifying the specific LLM or human responsible for a text)
  4. Human-LLM Co-authored Text Attribution (classifying texts as human-written, machine-generated, or a combination of both)


Abstract

Accurate attribution of authorship is crucial for maintaining the integrity of digital content, improving forensic investigations, and mitigating the risks of misinformation and plagiarism. Addressing the imperative need for proper authorship attribution is essential to uphold the credibility and accountability of authentic authorship. The rapid advancements of Large Language Models (LLMs) have blurred the lines between human and machine authorship, posing significant challenges for traditional methods. We present a comprehensive literature review that examines the latest research on authorship attribution in the era of LLMs. This survey systematically explores the landscape of this field by categorizing four representative problems: (1) Human-written Text Attribution; (2) LLM-generated Text Detection; (3) LLM-generated Text Attribution; and (4) Human-LLM Co-authored Text Attribution. We also discuss the challenges related to ensuring the generalization and explainability of authorship attribution methods. Generalization requires the ability to generalize across various domains, while explainability emphasizes providing transparent and understandable insights into the decisions made by these models. By evaluating the strengths and limitations of existing methods and benchmarks, we identify key open problems and future research directions in this field. This literature review serves a roadmap for researchers and practitioners interested in understanding the state of the art in this rapidly evolving field.

Contributions


  • We provide a timely overview to discuss the challenges and opportunities presented by LLMs in the field of authorship attribution. By systematically categorizing authorship attribution into four problems and balancing problem complexity with practicality, we reveal insights into the evolving filed of authorship attribution in the era of LLMs.
  • We offer a comprehensive comparison of state-of-the-art methodologies, datasets, benchmarks, and commercial tools used in authorship attribution. This analysis not only improves the understanding of authorship attribution but also provides a valuable resource for researchers and practitioners to use as guidelines for approaching this direction.
  • We discuss open issues and provide future directions by considering crucial aspects such as generalization, explainability, and interdisciplinary perspectives. We also discuss the broader implications of authorship attribution in real-world applications. This holistic approach ensures that authorship attribution not only yields accurate results but also provides insights that are explainable and socially relevant.

  • Benchmarks

    The table below is a summary of Authorship Attribution Datasets and Benchmarks with LLM-Generated Text. Size is shown as the sum of LLM-generated and human-written texts (with the percentage of human-written texts in parentheses). Language is displayed using the two-letter ISO 639 abbreviation. Columns P2, P3, and P4 indicate whether the dataset supports problems described in Problem 2, 3, and 4, respectively.

    Name Domain Size Length Language Model P2 P3 P4
    TuringBench News 168,612 (5.2%) 100 to 400 words en GPT-1,2,3, GROVER, CTRL, XLM, XLNET, FAIR, TRANSFORMER-XL, PPLM
    TweepFake Social media 25,572 (50.0%) less than 280 characters en GPT-2, RNN, Markov, LSTM, CharRNN
    ArguGPT Academic essays 8,153 (49.5%) 300 words on average en GPT2-Xl, text-babbage-001, text-curie-001, davinci-001,002,003, GPT-3.5-Turbo
    AuTexTification Tweets, reviews, news, legal, and how-to articles 163,306 (42.5%) 20 to 100 tokens en, es BLOOM, GPT-3
    CHEAT Academic paper abstracts 50,699 (30.4%) 163.9 words on average en ChatGPT
    GPABench2 Academic paper abstracts 2.385M (6.3%) 70 to 350 words en ChatGPT
    Ghostbuster News, student essays, creative writing 23,091 (87.0%) 77 to 559 (median words per document) en ChatGPT, Claude
    HC3 Reddit, Wikipedia, medicine, finance 125,230 (64.5%) 25 to 254 words en, zh ChatGPT
    HC3 Plus News, social media 214,498 N/A en, zh ChatGPT
    HC-Var News, reviews, essays, QA 144k (68.8%) 50 to 200 words en ChatGPT
    HANSEN Transcripts of speech (spoken text), statements (written text) 535k (96.1%) less than 1k tokens en ChatGPT, PaLM2, Vicuna-13B
    M4 Wikipedia, WikiHow, Reddit, QA, news, paper abstracts, peer reviews 147,895 (24.2%) more than 1k characters ar, bg, en, id, ru, ur, zh davinci-003, ChatGPT, GPT-4, Cohere, Dolly2, BLOOMz
    MGTBench News, student essays, creative writing 21k (14.3%) 1 to 500 words en ChatGPT, ChatGLM, Dolly, GPT4All, StableLM, Claude
    MULTITuDE News 74,081 (10.8%) 200 to 512 tokens ar, ca, cs, de, en, es, nl, pt, ru, uk, zh GPT-3,4, ChatGPT, Llama-65B, Alpaca-LoRa-30B, Vicuna-13B, OPT-66B, OPT-IML-Max-1.3B
    OpenGPTText OpenWebText 58,790 (50.0%) less than 2k words en ChatGPT
    OpenLLMText OpenWebText 344,530 (20%) 512 tokens en ChatGPT, PaLM, Llama, GPT2-XL
    Scientic Paper Scientific papers 29k (55.2%) 900 tokens on average en SCIgen, GPT-2,3, ChatGPT, Galactica
    RAID News, Wikipedia, paper abstracts, recipes, Reddit, poems, book summaries, movie reviews 523,985 (2.9%) 323 tokens on average cs, de, en GPT-2,3,4, ChatGPT, Mistral-7B, MPT-30B, Llama2-70B, Cohere command and chat
    M4GT-Bench Wikipedia, Wikihow, Reddit, arXiv abstracts, academic paper reviews, student essays 5,368,998 (96.6%) more than 50 characters ar, bg, de, en, id, it, ru, ur, zh ChatGPT, davinci-003, GPT-4, Cohere, Dolly-v2, BLOOMz
    MAGE Reddit, reviews, news, QA, story writing, Wikipedia, academic paper abstracts 448,459 (34.4%) 263 words on average en GPT, Llama, GLM-130B, FLAN-T5 OPT, T0, BLOOM-7B1, GPT-J-6B, GPT-NeoX-2
    MIXSET Email, news, game reviews, academic paper abstracts, speeches, blogs 3.6k (16.7%) 50 to 250 words en GPT-4, Llama2

    Detectors

    The Table below presents an overview of LLM-Generated Text Detectors.

    Detector Price API Website
    GPTZero 150k words at $10/month, 10k words for free per month Yes https://gptzero.me/
    ZeroGPT 100k characters for $9.99, 15k characters for free Yes https://www.zerogpt.com/
    Sapling 50k characters for $25, 2k characters for free Yes https://sapling.ai/ai-content-detector
    Originality.AI 200k words at $14.95/month Yes https://originality.ai/
    CopyLeaks 300k words at $7.99/month Yes https://copyleaks.com/ai-content-detector
    Winston 80k words at $12/month Yes https://gowinston.ai/
    GPT Radar $0.02/100 tokens N/A https://gptradar.com/
    Turnitin’s AI detector License required N/A https://www.turnitin.com/solutions/topics/ai-writing/ai-detector/
    GPT-2 Output Detector Free N/A https://github.com/openai/gpt-2-output-dataset/tree/master/detector
    Crossplag Free N/A https://crossplag.com/ai-content-detector/
    CatchGPT Free N/A https://www.catchgpt.ai/
    Quil.org Free N/A https://aiwritingcheck.org/
    Scribbr Free N/A https://www.scribbr.com/ai-detector/
    Draft Goal Free N/A https://detector.dng.ai/
    Writefull Free Yes https://x.writefull.com/gpt-detector
    Phrasly Free Yes https://phrasly.ai/ai-detector
    Writer Free Yes https://writer.com/ai-content-detector

    BibTeX

    @article{huang2024aa_llm,
        title   = {Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges},
        author  = {Baixiang Huang and Canyu Chen and Kai Shu},
        year    = {2024},
        journal = {arXiv preprint arXiv: 2408.08946},
        url     = {https://arxiv.org/abs/2408.08946}, 
    }

    Can Large Language Models Identify Authorship?

    TLDR: We discover LLMs' strong capacities of performing authorship verification and attribution in a zero-shot way, which have surpassed the state-of-the-art supervised models, and providing explanations through the role of linguistic features.


    Emory University
    EMNLP 2024 Findings
    framework
    A Comparison Between Linguistically Informed Prompting (LIP) and other Prompting Strategies for Authorship Verification. "Analysis" and "Answer" are the output of prompting GPT-4. Only LIP strategy correctly identifies that the given two texts belong to the same author. Text colored in orange highlights the differences compared to vanilla prompting with no guidance. Text colored in Blue indicates the linguistically informed reasoning process. Blue text represents the text referenced from the original documents.


    Abstract

    The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis remains under-explored. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) How can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our assessment demonstrates LLMs' proficiency in both tasks without the need for domain-specific fine-tuning, providing insights into their decision-making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis.

    Contributions


  • We conduct a comprehensive evaluation of LLMs in authorship attribution and verification tasks. Our results demonstrate that LLMs outperform existing BERT-based models in a zero-shot setting, showcasing their inherent stylometric knowledge essential for distinguishing authorship. This enables them to excel in authorship attribution and verification across low-resource domains without the need for domain-specific fine-tuning.
  • We develop a pipeline for authorship analysis with LLMs, encompassing dataset preparation, baseline implementation, and evaluation. Our novel Linguistically Informed Prompting (LIP) technique guides LLMs to leverage linguistic features for accurate authorship analysis, enhancing their reasoning capabilities.
  • Our end-to-end approach improves the explainability of authorship analysis. It elucidates the reasoning and evidence behind authorship predictions, shedding light on how various linguistic features influence these predictions. This contributes to a deeper understanding of the mechanisms behind LLM-based authorship identification.

  • BibTeX

    @artile{huang2024authorship,
        title   = {Can Large Language Models Identify Authorship?}, 
        author  = {Baixiang Huang and Canyu Chen and Kai Shu},
        year    = {2024},
        journal = {arXiv preprint},
        volume  = {abs/2403.08213},
        url     = {https://arxiv.org/abs/2403.08213}, 
    }