Ghostbuster: Detecting Text Ghostwritten by Large Language Models
Large language models like ChatGPT have gained notoriety for their impressive text generation capabilities, leading to concerns about their misuse in ghostwriting assignments and spreading misinformation. To address these issues, we have developed Ghostbuster, a cutting-edge method for detecting AI-generated text.
Traditional tools for identifying AI-generated text often struggle with new data types and risk misclassifying human-generated content. Ghostbuster takes a unique approach by analyzing the probability of generating each token in a document under multiple weak language models and combining these probabilities into a final classifier. This method does not require knowledge of the specific model used to generate the text, making it highly effective for detecting text generated by commercial models like ChatGPT and Claude.
Why this Approach?
Existing AI-generated text detection systems face challenges in classifying diverse text types and struggle with generalization. Ghostbuster overcomes these limitations by integrating probability-based features from multiple language models and utilizing a structured search procedure to select the most relevant features for classification.
How Ghostbuster Works
Ghostbuster employs a three-stage training process that involves computing probabilities, selecting features, and training a linear classifier. By incorporating probabilities from various language models and carefully selecting features, Ghostbuster achieves exceptional performance in detecting AI-generated text across different domains and models.
Results
Ghostbuster outperforms existing detection methods, achieving high F1 scores in both in-domain and out-of-domain evaluations. Its robustness to different prompts and models further highlights its effectiveness in detecting AI-generated text accurately.
Conclusion
Ghostbuster represents a significant advancement in AI-generated text detection, offering a versatile and reliable solution for identifying text produced by large language models. With its strong generalization capabilities and robust performance, Ghostbuster is poised to make a positive impact in combating misinformation and ensuring the integrity of written content.
Explore Ghostbuster and learn more about our research:
Test your skills in identifying AI-generated text:
