-
Notifications
You must be signed in to change notification settings - Fork 549
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Any plans for a GPT-3 detector that can spot ChatGPT output? #33
Comments
try this one on chatgpt output and see how it works |
I did at https://huggingface.co/openai-detector/ with for instance the text below. It indicate this is not GPT, though it is (response to "Leg uit waarom Keizer Wilhelm politiek asiel in Nederland kreeg.")
The English response to the equivalent "Explain why Kaiser Wilhelm got asylum in The Netherlands." did get marked as GPT:
I will try a few more later on as currently this N=2 experiment is way too small to draw conclusions yet. |
Yesterday I tested this on about 30 outputs from GPT-3 (not ChatGPT). It does seem to work quite reliably on outputs generated by text-davinci-002 and the other models released prior to November 2022. But outputs generated by text-davinci-003 often foil it. I don't have much data yet, but the only outputs that the detector got completely wrong were the ones generated by davinci-003 (trained on a larger, updated dataset). And this is the model that, I believe, ChatGPT uses. It's sometimes called GPT-3.5, because the outputs do seem considerably better. |
Can you post some examples where it was not reliably detecting ChatGPT produced output? Would be curious about it. |
Sure. Here's one that GPT-2 detector scored 64.83% real. I'll scrounge up one or two more when I have a moment. Prompt: Write an article that argues that genetically engineered babies present serious ethical problems, and cite your sources. Output: Genetically engineered babies present serious ethical problems that should be carefully considered before any such technology is implemented. One major ethical concern is the potential for eugenic practices, where certain genetic traits are selectively enhanced or eliminated for the purpose of creating a superior human race. This could lead to a society where individuals are valued based on their genetic makeup, leading to discrimination and inequality. Another concern is the lack of long-term data on the safety and effectiveness of genetic engineering. While some studies have shown promising results, there is still a lack of understanding on the potential unintended consequences of such procedures. For example, there is a risk of unexpected genetic changes or mutations that could have negative impacts on an individual's health or development. Additionally, there is the issue of informed consent. Would parents fully understand the risks and potential outcomes of genetic engineering, and would they be able to make informed decisions on behalf of their children? There is also the potential for exploitation, with wealthier individuals able to afford such procedures while those who cannot may be left behind. Furthermore, the use of genetic engineering could lead to the creation of a two-tiered society, with those who can afford the technology having access to enhanced traits while those who cannot are left with the "natural" genetics. This could lead to a further divide between the haves and the have-nots. Overall, the ethical implications of genetically engineered babies are numerous and complex, and it is crucial that a thorough and careful examination is conducted before any such technology is implemented. Sources:
|
Launched a little website that is capable of translating the provided input first into english, if the language is other than English and then uses the provided models from this repo to do the real/fake prediction. GPT Output Detector |
It seems that domain is gone by now. Any plans to share the source somewhere? |
No description provided.
The text was updated successfully, but these errors were encountered: