X discussion: open-source AI builder Jun Song says he turned down an anonymous black hat group offering huge money to post-train open-weight models on exploit data

Background and the problem

The safety debate around open-weight models has mostly stayed abstract: once weights are public anyone can keep training them on their own data, and the refusal safeguards a vendor adds can be trained away. On Nov 1, open-source AI builder Jun Song (@jun_song, whose X bio says he is building Supergemma and runs a datacenter hardware export business in Korea) turned that worry into a concrete story: an anonymous black hat group approached him wanting to pay a lot to have models trained on attack data.

How it would work

By his account, the group wanted reinforcement learning (RL) post-training of open-weight models on the exploit data it holds, in a volume he said he would have believed was an undercover FBI dataset. He compared it with a white-hat project of his built on Zhipu's GLM-5.3, which used under 1% of that volume and already beat frontier models by a huge margin. The mechanism is not new: real attack traces used as a reward signal can quickly specialize a general model into an attack agent. What is new is someone offering serious money for it.

Benchmarks and numbers

None published and nothing verifiable: the offer, the data volume and the group's identity rest on the author's word alone, and "outperformed frontier models" comes with no public benchmark. Only the post's reach can be checked: about 2,029 likes, 222.2K views and 771 bookmarks as of 17:10 SGT. Several accounts also copied its opening lines as a joke, so part of the audience reads it as an exaggerated story.

What developers can take from it

For teams using open models there are three practical points. Do not treat a model's built-in refusals as a security boundary; enforce permissions, sandboxing and auditing on the deployment side. When fine-tuning for security work, keep records of where the data came from and what it is for, so you are not pulled into gray-market deals. And watch vendors' open-weight release policies, since the next round of regulation talk will likely cite cases like this. This is an X discussion, not a confirmed fact.