AI · Retrospective
Where a language model could help a live game, and where it should not be trusted

In the past six months language models became something anyone can open in a browser. ChatGPT was released at the end of November, and GPT-4 followed on 14 March. For a small studio running a live game, the interesting question is practical. Which of the jobs we do every week could a model take on, and which should it never touch?
This is my own line, written as CEO of Playermon. It is a view, not a description of a system we run.
Where I am coming from
My own machine learning work sits in a different corner. In a Master by Research at Multimedia University, which I left before the thesis, I was a TM R&D-sponsored scholar working on deep learning for predicting network faults. That research was published in IJET in 2018. When I introduced myself in an AMA in November 2021, I listed AI experiments and data pipelines among my past work.
That background makes me ask two questions before any other. What was the model trained on, and how are its errors found? A language model produces fluent text whether or not the text is correct. Fluency is not evidence. Any use I would accept has to have a way to catch the wrong answer before a player sees it.
The weekly work of a live game
A game like ours produces a lot of words. In the first four months of this year we released three patches: 2.6.0 in February, 2.7.0 in March and 2.8.0 in April. Each one needs patch notes. Around them come game tips, skill card reveals, rule changes such as the SpaceDen decoration update at the start of this month, and questions in the community. Since January all our language groups share one Telegram forum, so a single announcement now reaches readers of several languages.
Most of that work is drafting and answering. That is where a model could help.
Where a model could help
- First drafts of patch notes and tips. The source is our own changelog, and a person checks the draft against it before anything is published. Every number in the notes, such as the 23 skill cards rebalanced in 2.8.0, has to come from the changelog, not from the draft.
- First drafts of translations. A model could give the forum a first version of an announcement in another language, for someone who reads that language to correct.
- Answering questions from published pages. Our tutorials already explain battles, PvE, SpaceDen, treasure hunting and buying energy. A model restricted to those pages, that links the page it drew from and says when the pages do not cover a question, could shorten the wait for simple answers. Without the link, nobody can check the answer.
- Flagging, not judging, in moderation. A model could flag messages that look like spam or impersonation for a moderator to review. The decision to remove someone stays with a person.
The common thread is that a person or a published source sits between the model's output and the player.
Where it should not be trusted
- Wallets and keys. Seed phrases, signing, recovering a wallet: none of it. Scammers already impersonate support staff in crypto communities, and a model that writes friendly wallet-recovery instructions is producing exactly the message a scammer would send. The answer to a wallet question should be a fixed one, written by people: we will never ask for your seed phrase.
- Rewards. Our leaderboard rewards are sent on-chain, and Season 3's PYM went out that way in March. A transfer cannot be recalled. A model should not decide who is eligible, calculate a payout or promise a reward in chat. A generated "you will receive" is a commitment players will screenshot.
- Game balance. A draft that invents a healing value or an energy rule misleads players about the game itself. Balance numbers come from the build.
- Confidential material. At the start of this month Samsung restricted staff use of ChatGPT after internal code was uploaded to it. The same caution applies to a game studio: unreleased patches, contract administration details and player data do not go into a hosted model unless we have read and accepted how that data is kept.
The line
The rule I would hold a game like ours to is short. A model may draft words that a person checks before they reach players. It may not take any action that moves value, and it may not give any answer that could cost a player something they cannot get back.
That leaves a lot of useful work on the table, and it keeps the irreversible parts of a blockchain game where they belong: with code we have tested and people who are accountable for it.