Can ChatGPT Really Replace Crypto Audits? Not Yet, Say Researchers

Everyone’s experimenting with ChatGPT, even crypto exchanges.

Coinbase recently turned to artificial intelligence to experiment with how accurately ChatGPT could carry out a token security review—a requirement for all tokens listed on the exchange.

After reviewing 20 different smart contracts, the mega-popular AI tool produced the same results as the manual review 12 times.

However, of the eight misses, five were cases where ChatGPT incorrectly labeled a high-risk asset as low-risk, which is the worst-case failure.

The experiment also revealed that the AI sometimes produced inconsistent results, with the same prompt generating different outcomes, especially when moving from one iteration of ChatGPT to the next.

Still, the Coinbase team is optimistic that—with further prompt engineering—they can increase the accuracy of ChatGPT to a point where it could be used as a secondary quality assurance check.

“We are not surprised because such smart contracts can also be automatically audited by [other traditional programming] tools,” a spokesperson from BlockSec, a blockchain security infrastructure firm, told Decrypt. “However, it cannot work for complicated business logic, which are the main attack surfaces and the main loopholes that smart contract audits should focus on.

Coinbase’s optimism about using the tool for additional assurances was nonetheless echoed by other security experts in the crypto-security space.

“At this stage, it [AI] can not replace a person, but it is an indispensable aid, including for tired or inattentive auditors,” independent security researcher Officer’s Notes told Decrypt via Twitter. “I think that Q/A [quality assurance] and fuzzing will definitely not be able to do without AI tools in the future.”

ChatGPT replacing engineers

Though tentative, the blockchain security sector seems to be accepting of the possible implementation of AI tools.

But could AI replace manual security auditors in the future?

“Perhaps one day we’ll get to that point, but we’re still a long way off. What’s more likely is a complementary approach. There are some things humans do better than machines and vice versa,” Certik’s head of solutions architecture Connie Lam told Decrypt via email, “Tools help us build new things, but they don’t replace us. The invention of the calculator didn’t make accountants obsolete, it made them better at their job.”

For now, though, non-AI security tools are still far more useful than anything just entering the market to locate vulnerabilities.

“Current security audit tools are still superior to OpenAI,” a spokesperson from OpenZeppelin told Decrypt. “They [Coinbase] are testing it for listing ERC-20 tokens, which is a well-known pattern. That makes it more suitable for automation.”

That may change though.

The rapid advances seen even between ChatGPT 3.5 and ChatGPT 4 are palpable, suggesting that further upgrades will continue to impress.

And as they do, integrating these tools should even be “encouraged.”

“The use of ChatGPT should be encouraged during the development phase. It’s a powerful tool, and refusing to work with it and learn what it can do would be a setback,” said Lam. “ChatGPT is also a very powerful tool for searching for information and building a knowledge hub. It can help users quickly grasp complicated topics and keep up to date on the latest security information.”