[{"data":1,"prerenderedAt":157},["ShallowReactive",2],{"\u002Fblog\u002F2021-07-15-code-writing-models-and-audited-contract-libraries":3},{"id":4,"title":5,"archiveType":6,"body":7,"category":141,"contextDate":142,"cover":143,"coverAlt":144,"date":145,"description":146,"extension":147,"featured":148,"historicalDate":145,"historicalLabel":142,"meta":149,"navigation":150,"path":151,"publishedDate":142,"seo":152,"stem":153,"thumbnail":142,"workPeriod":154,"writtenDate":155,"__hash__":156},"blogs\u002Fblog\u002F2021-07-15-code-writing-models-and-audited-contract-libraries.md","Code-writing models and why audited contract libraries matter more","retrospective",{"type":8,"value":9,"toc":132},"minimark",[10,28,33,36,39,42,46,49,52,56,59,74,77,81,84,87,122,126,129],[11,12,13,14,21,22,27],"p",{},"On 29 June GitHub opened a ",[15,16,20],"a",{"href":17,"rel":18},"https:\u002F\u002Fgithub.blog\u002F2021-06-29-introducing-github-copilot-ai-pair-programmer\u002F",[19],"nofollow","technical preview of Copilot",", a tool that suggests whole lines and functions as you type. On 7 July OpenAI published the paper behind it, ",[15,23,26],{"href":24,"rel":25},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2107.03374",[19],"Evaluating Large Language Models Trained on Code",". A month earlier I had created a token for the Developer Kaki group's live session, and the contract I typed was ten lines long. I read the paper with that contract in mind.",[29,30,32],"h2",{"id":31},"what-the-paper-measured","What the paper measured",[11,34,35],{},"Codex is a GPT language model fine-tuned on public code from GitHub, and a production version of it powers Copilot. To measure it, the authors wrote 164 programming problems by hand and called the set HumanEval. Each problem is a Python function signature and docstring, with an average of 7.7 unit tests behind it. A problem counts as solved if the generated function passes its tests.",[11,37,38],{},"With one attempt per problem, the 12-billion-parameter Codex solved 28.8 per cent. GPT-3 solved none. Allowed 100 attempts per problem, and counted as solved if any attempt passed, it reached 70.2 per cent.",[11,40,41],{},"Two details stood out to me. First, \"correct\" here means \"passes the tests the authors wrote\". The tests are the definition of correct. Second, the language. The paper measured Python, and GitHub's announcement names Python, JavaScript, TypeScript, Ruby and Go as the languages where the preview works especially well. Solidity is on neither list.",[29,43,45],{"id":44},"the-sections-i-reread","The sections I reread",[11,47,48],{},"The paper is direct about risk. Its section on over-reliance warns that Codex may suggest solutions that \"superficially appear correct but do not actually perform the task\" the user intended. It reports that when a prompt contains subtle bugs, the model tends to produce worse code than it is capable of, and that the gap grows with model size. Its section on security says the model can produce vulnerable code, and that qualified people should review its output before running or trusting it.",[11,50,51],{},"None of that surprised me once I thought about where the model learned. It learned from public code, and public code includes bugs.",[29,53,55],{"id":54},"why-contracts-are-the-hard-case","Why contracts are the hard case",[11,57,58],{},"A Python function that fails a unit test fails on a developer's laptop. A token contract that fails does so on a public chain, often holding other people's money, with its source open to anyone looking for the flaw and usually no way to patch it in place.",[11,60,61,62,67,68,73],{},"In April, Uranium Finance, a fork of Uniswap V2 on Binance Smart Chain, ",[15,63,66],{"href":64,"rel":65},"https:\u002F\u002Fwww.theblock.co\u002Fpost\u002F103076\u002Fbinance-smart-chain-defi-uranium-finance-exploited-lost-50-million",[19],"lost about $50 million",". Its pair contract ",[15,69,72],{"href":70,"rel":71},"https:\u002F\u002Fcryptoslate.com\u002Fthe-most-recently-hacked-defi-project-couldnt-even-copy-and-paste-uniswap-and-sushiswaps-code\u002F",[19],"changed a fee constant"," from 1,000 to 10,000 in the balance calculation, while the check of the constant-product rule kept the original scale. The original Uniswap code had been audited. The edited copy carried its reputation without its review.",[11,75,76],{},"Nobody has suggested a model wrote that change. But it is the kind of change a suggestion tool makes cheap: plausible, local, and wrong in a way the surrounding code does not reveal.",[29,78,80],{"id":79},"what-i-would-tell-developers-now","What I would tell developers now",[11,82,83],{},"The DEVK contract was ten lines because everything that moves balances came from OpenZeppelin's ERC20. The README I wrote for it calls those contracts \"battle tested\". Use and review by many people looking for exactly the failures the paper describes is the property a code model cannot supply.",[11,85,86],{},"So the advice gets stricter, not looser:",[88,89,90,98,104,110,116],"ol",{},[91,92,93,97],"li",{},[94,95,96],"strong",{},"Import audited libraries instead of accepting a generated version of them."," A suggested ERC20 that looks like OpenZeppelin's is not OpenZeppelin's.",[91,99,100,103],{},[94,101,102],{},"Do not edit library code in place."," If a behaviour has to change, extend the contract and test the extension.",[91,105,106,109],{},[94,107,108],{},"Write the tests first, and make them state properties."," Total supply, who can mint, what happens at zero. If a model writes the code, your tests are the only definition of correct you control.",[91,111,112,115],{},[94,113,114],{},"Read every suggestion as a reviewer."," Treat it as code from an unknown author, because that is what it is.",[91,117,118,121],{},[94,119,120],{},"Keep the hand-written surface small."," Ten lines are easy to review. Three hundred suggested lines are not.",[29,123,125],{"id":124},"what-it-changes-for-teaching","What it changes for teaching",[11,127,128],{},"For anyone teaching smart contracts, as I do through TriveAcademy, the order of lessons matters more than it did a month ago. If writing code becomes cheaper, reading it becomes the scarcer skill. A student who can explain why a contract imports a library, and what each test proves, is better placed than one who can produce a contract quickly.",[11,130,131],{},"I do not think these models are a passing curiosity. The paper's numbers on Python are already high enough to be useful, and its authors write about what future models might do. That is the reason to settle the habits now. Better suggestions will not shrink the review that a contract holding value needs. They will make it easier to skip.",{"title":133,"searchDepth":134,"depth":134,"links":135},"",2,[136,137,138,139,140],{"id":31,"depth":134,"text":32},{"id":44,"depth":134,"text":45},{"id":54,"depth":134,"text":55},{"id":79,"depth":134,"text":80},{"id":124,"depth":134,"text":125},"AI",null,"\u002Fimg\u002Fblog-covers\u002F2021-07-15-code-writing-models-and-audited-contract-libraries.webp","A small robotic arm placing a block beside a shelf of white volumes with blue spines.","2021-07-15","Reading the Codex paper as someone who teaches smart contracts: suggested code makes audited libraries and tests more important, not less.","md",false,{},true,"\u002Fblog\u002F2021-07-15-code-writing-models-and-audited-contract-libraries",{"title":5,"description":146},"blog\u002F2021-07-15-code-writing-models-and-audited-contract-libraries","2021","2026-11-07","Nx5vmrQr02Lx0zZkUorgn278EzxGQTGep1JgEUI6eps",1794838285797]