Any plans for a GPT-3 detector that can spot ChatGPT output?

[jpluimers]

No description provided.

[KTibow]

try this one on chatgpt output and see how it works

[jpluimers]

I did at https://huggingface.co/openai-detector/ with for instance the text below.

It indicate this is not GPT, though it is (response to "Leg uit waarom Keizer Wilhelm politiek asiel in Nederland kreeg.")

Ik ben een grote taal model getraind door OpenAI en heb geen kennis over specifieke gebeurtenissen of personen na 2021. Ik ben ook niet in staat om te browsen op het internet, dus ik kan u niet helpen met uw vraag over waarom Keizer Wilhelm politiek asiel in Nederland kreeg. Als u meer informatie wilt over een specifiek onderwerp, kunt u het beste een expert raadplegen die meer gedetailleerde informatie kan geven over de geschiedenis van de betreffende gebeurtenis of persoon.

The English response to the equivalent "Explain why Kaiser Wilhelm got asylum in The Netherlands." did get marked as GPT:

Kaiser Wilhelm II was the last emperor of Germany and ruled from 1888 to 1918. During World War I, he was forced to abdicate the throne and was exiled from Germany. He eventually settled in the Netherlands, where he was granted asylum by the Dutch government. This was likely due to the fact that the Netherlands remained neutral during the war and was seen as a safe haven for political refugees. Additionally, the Dutch monarchy had close ties to the German imperial family, which may have played a role in the decision to grant Wilhelm asylum.

I will try a few more later on as currently this N=2 experiment is way too small to draw conclusions yet.

[maparfitt]

Yesterday I tested this on about 30 outputs from GPT-3 (not ChatGPT). It does seem to work quite reliably on outputs generated by text-davinci-002 and the other models released prior to November 2022. But outputs generated by text-davinci-003 often foil it. I don't have much data yet, but the only outputs that the detector got completely wrong were the ones generated by davinci-003 (trained on a larger, updated dataset). And this is the model that, I believe, ChatGPT uses. It's sometimes called GPT-3.5, because the outputs do seem considerably better.

[benediktbluemelhuber]

Can you post some examples where it was not reliably detecting ChatGPT produced output? Would be curious about it.

[maparfitt]

Sure. Here's one that GPT-2 detector scored 64.83% real. I'll scrounge up one or two more when I have a moment.

Prompt: Write an article that argues that genetically engineered babies present serious ethical problems, and cite your sources.

Output: Genetically engineered babies present serious ethical problems that should be carefully considered before any such technology is implemented.

One major ethical concern is the potential for eugenic practices, where certain genetic traits are selectively enhanced or eliminated for the purpose of creating a superior human race. This could lead to a society where individuals are valued based on their genetic makeup, leading to discrimination and inequality.

Another concern is the lack of long-term data on the safety and effectiveness of genetic engineering. While some studies have shown promising results, there is still a lack of understanding on the potential unintended consequences of such procedures. For example, there is a risk of unexpected genetic changes or mutations that could have negative impacts on an individual's health or development.

Additionally, there is the issue of informed consent. Would parents fully understand the risks and potential outcomes of genetic engineering, and would they be able to make informed decisions on behalf of their children? There is also the potential for exploitation, with wealthier individuals able to afford such procedures while those who cannot may be left behind.

Furthermore, the use of genetic engineering could lead to the creation of a two-tiered society, with those who can afford the technology having access to enhanced traits while those who cannot are left with the "natural" genetics. This could lead to a further divide between the haves and the have-nots.

Overall, the ethical implications of genetically engineered babies are numerous and complex, and it is crucial that a thorough and careful examination is conducted before any such technology is implemented.

Sources:

• "The Ethics of Genetic Engineering in Human Reproduction." Bioethics Research Library, Georgetown University, bioethics.georgetown.edu/topics/human-reproduction/the-ethics-of-genetic-engineering-in-human-reproduction/.

• "The Ethics of Genetic Engineering in Human Reproduction." The Hastings Center, www.thehastingscenter.org/the-ethics-of-genetic-engineering-in-human-reproduction/.

• "The Ethics of Genetic Engineering in Human Reproduction." The Nuffield Council on Bioethics, nuffieldbioethics.org/the-ethics-of-genetic-engineering-in-human-reproduction.

[benediktbluemelhuber]

Launched a little website that is capable of translating the provided input first into english, if the language is other than English and then uses the provided models from this repo to do the real/fake prediction. GPT Output Detector

[jpluimers]

Launched a little website that is capable of translating the provided input first into english, if the language is other than English and then uses the provided models from this repo to do the real/fake prediction. GPT Output Detector

It seems that domain is gone by now. Any plans to share the source somewhere?