Exploring AI’s Role in Smart Contract Security | Alice & Daky Wang, OneSavie Lab | ETHTaipei 2025

[Music] Hello everyone. Uh we are delighted to have you join us to explore how AI can uh enhance smart conscious security. So English sharing uh we will examine the optimization that can be made during uh development phase and specifying the specific improvement AI can bring. So also we will talk about the current potential limitation in this field. Uh finally we will show the contribution we hope to re our research can bring to the security industry.

Okay. So next slide. Okay. Before we begin our uh presentation, let's have a quick uh brief self introduction. I'm Deki.

Currently, I'm in charge of uh web3 product and development team at 1 sav lab and I'm also the member of diva head labs. So if you have any question about today's sharing uh or any s about the open source project we will mention later uh feel free to direct message me on Twitter. Hi, I'm Alis currently a security research engineer at one lab and also a member of the ops team and whitehead at define labs. My primary research focuses on web3 security. Here is the outline of today's sharing.

In the background section, we will introduce the current state of the commonly used tools and the stage at which AI can assist us in improving our work. Next, we will introduce our opensource project, Bastet. In the introduction section, we will briefly explain the project structure, how we select targets. And in the misology section we will introduce the components of the workflow in basted. And finally in the conclusion we will share what we found during the experimental process.

Okay let's get started. Define applications are becoming increasingly diverse and with the growing awareness of security, the violent of security audits before contract deployment is also increasingly uh growing along with a growing variety of audit platforms and related services. Against this backdrop, we hope to work with the power of the community to drive the ecosystem security of the web3 industry. We want to more reliable in identify viabilities that are not easily detected by common tools but frequently occur. Reduce the back and forth cost of audit report writing and developer fix and ensure least common vulnerabilities can be addressed during the development phase.

As a white hat involved in compared a platforms when seeing a new code base of a certain type of define many common buck list for that type immediately come to my mind. However, many common bugs continue to repeatedly appear in codebase even up until now. When a protocol integrates with unis swap, most people will first look for issues related to improper slipage settings. When the protocol is a vote, we will first check for the possibility of the first deposit inflation. If the remy mentions that it needs to compile with a certain EIP, we will review the EIP documentation and conduct a detailed review of the codebase.

These issues are almost like old friends to us. When we conduct core reviews, our goal is to focus on enhancing security coverage, identifying all problematic code snippets and writing detail reports for less. Even if these issues are just like low hanging fruits, they are just like your bodies. When you audit a specific type of protocol, they always show up and knock at your door partying in your audit report. However, whether it's a private audit or a competition, it is always limited by allocation of resources.

The most valuable part is of course the cost of time. As the execution of the project is time limited, we must strive to uncover as many issues as possible within the limited time and resources. More precisely, we need to focus on identifying higher quality problems. Assuming a more complete development life cycle is divided as follows. Development, testing, auditing, continual monitoring, and incident response.

The most suitable stage for improvement would be during the testing to auditing phase. Identify le common issues and fix len. This can reduce researchers from rewriting reports on repetitive issues and help the projecting avoid pitfalls during the development phase. In current smart contract audits, the most common tools are static analyzers such as laser which can be integrated into CI/CD environment. Phonalizer is used in the well-known competition audit platform like C4.

Le static analyzers mostly use a or reget methods to detect common vulnerabilities that perform static code scanning and cannot fully understand the execution of logic of the code when the contract is written in an in another programming language. They are also unable to identify issues with the same rule codes. They are not effective at detecting some common vulnerabilities that involve logic. Based on the background mentioned above and we think kids are adorable, our open source project Betat was born. We will use the slipage vulnerability as a case study and introduce the methodology of our project bust including the warflow design and the results of our preliminary experiments.

Our open source project busted mainly consists of two parts. First is vulnerability data set plus AI audit benchmark and second is AI vulnerability identification process. Let's have a sneak peek of our user scenarios. The users can use CLI to scan the smart contract codebase or evaluate the accuracy of an AI assessment workflow. If you choose to scan, you can place the contract in the specified folder.

There are many optimized rules available in the workflow to use and the output will be a report. If you choose to evaluate, you can select a workflow or import a customized workflow. The corresponding data set for the vulnerability will be used and the output will be the occurrency results. How do we select our targets? Our vulnerability selection focuses on common viabilities that are difficult to maintain with static analyzers.

These vulnerabilities are often classified as median to high risk in audit competitions or platforms and can lead to losses. By designing corresponding vulnerability detection and enhancing AI's accuracy in vulnerability identification, we aim to effectively optimize the security life cycle management from development to auditing and subsequent monitoring. This growth floor contains some essential elements. First is knowledges. AI is highly skilled in understanding code and identifying subto syntax errors.

Before using AI to help us identify a specific type of vulnerability, the knowledge of the vulnerability needs to be prepared. Once the knowledge is prepared, we need to create clear prompts to guide AI in detecting vulnerabilities. The design of these prompts will directly impact the occurrency of vulnerability detection. And finally, in the validation part, the purpose of validation is to ensure the accuracy and effectiveness of the detection results. We use different taste data to calculate and evaluate the detection results.

Okay, let's now introduce the vulnerability case for today's sharing. Slipage. Slippage refers to the differences between the expected price of a transaction and a actual price at which the transaction is executed. Using unis swap as an example, the amount of assets you can receive from a swap is calculated based on the reserve of the two assets and a formula. However, the amount of assets you expect to receive, the time of the swap may differ from the actual amount you get.

This is because the transaction may not be executed immediately after submission. If the market change during the transaction execution, it can lead to a different result than expected. If there is no slippage check, an EV can monitor transactions and perform a sandwich attack before and after the transaction, prohibiting from price fluctuations. the user will receive fewer assets than expected. Before we dive into introducing prompt engineering technique, let's assume we want to use a static analyzer to search for this type of vulnerability.

How can we go about it? In the case of unis swap v2, we would first use find all to search of all functions where slage could potentially occur such as swap exact tokens for tokens, swap exact eth for tokens, add liquidity, remove liquidity and so on. In the case of unis swap B3 there are functions such as decrease decrease liquidity in health increase liquidity current range swap exact output single swap exact input multih hope and so on. And next we will identify where parameters like mini among are located within the functions and at which position they appear in the parameters. Then for each type of improper slipage setting scenario, we need to write a condition statement to perform the check.

What are the challenges of doing this? There are many protocols that perform swaps such as balancer, curve and other. The function names for implementing swaps vary across protocols and the parameters and their position for setting slippage are also completely different. Moreover, these condition statements in conquer the same issues but they do not work with co snapets re written in other syntax. This make it difficult for us to maintain and update these rules.

Okay. So now we move to the methology part and I forget. Yeah. So uh if we want to use AI to assist us to identify vulnerability. So what method are we using?

Are we using? So let's give an example about how a human brain learned to recognize a cat. So you can see on the screen we have the knowledge base. Uh suppose today we use this passage to uh describe the appearance of a cat. So here's the definition right.

Uh we can take a look at it quickly since later we will have a section for everyone to join. Yeah. So you have two image. You'll never see a cat or a dog before. And now follow the guideline to recognize cat image.

Are ready? So let's go to the next. So AI might think so image like three uh four and six are cats because the uh in the knowledge base we mentioned that cats have upright ears and sharp calls. Yep. So everybody want to try?

So if you think image number one is a cat, you raise your hand. Okay. Three, two, one. Number one is a cat. Okay.

Number two, you think it's a cat. Okay. Three and four, five, six. Okay. So let's go to the let's go to the next slide and let's look at the uh answer this one.

Okay. So the real answer is image one, two and three is a cat. Yeah. So this example shows that a simple simply having a paragraph of test to describe a cat is not enough should be very limited to its effectiveness for use AI to recognize a cat. So if you want to teach me how to recognize a new species and give me some example plus the description will be very helpful.

Yeah. So the learning reser uh is very similar to how AI learn new knowledge. Therefore we adopt the uh coot approach. The box you can see here uh contain contain the whole prompts and the red the red part is the question to be answered by AI and the blue part step here uh represent the coot steps by using the coot uh the rep the reasoning ability of large language model can be improved. So if we directly feed knowledge to AI um the recognition result will be poor and by breaking down the reasoning uh process into multi-steps we help the model to think step by step to solve the problem.

So this reasoning process are close related to the root cause of the vulnerabilities. Yeah. Okay. So here is the prompt structure we designed uh which includes sross for identify vulnerabilities. We have AI to analyze each function according to the specific step and detail the s process in both contrast contract contains uh vulnerabilities and incorrect uh implement examples.

Finally uh the report is generated based on the common report format presenting the detection rules. Okay. So next slide. So here you can see the uh the output of bist using one of our contract uh in the data set with improper sleepage parameter settings. The journal report is show in the image.

This report includes one sentence summary of the vulnerability and severity followed by vulnerability details which contains the code snip you can see here and describe land in the issue. The report point out the function where the issue occur and explain why and also the last the recommendation part provides suggestion to uh for fixing the issue. Okay. And also when we talk about uh accuracy right let's get start also with the example to show how we calculate accuracy of AI recognition. So in the diagram you can see uh we use a model to identify which uh image contain a dog is a dog right.

So the the left part of the square represent uh a contra with small ability which we symbolize ear as a dog. Yeah. And the right the right part the right side represent contra without abilities. So we symbolize ear by cat. And the s the the circle in the middle is the result that AI recognized.

So with this formula uh we can calculate like uh the correct classification is the here we have five dogs plus the the here we have seven cats. So uh is 12 divided by the total number of images 22 is about 54% accuracy. So our goal of this workflow is to increase the accuracy rate but also minimize the force positive because the force positive might significantly increase manual validation cost. Okay. Okay.

So here is the result um that the sleepage minimum hour flow provide in this case study. We use our detection tool to evaluate then we can have the result true positive 12 true negative 27 false part two and force negative five. So the accuracy is about 85% of our workflow for this showcase. And also the result is what we want to see. We want to maximize the accuracy of our detection.

And our focus is on reducing the force positive. We refer to normal contrast being incorrect flag as probable contrast. This is important because during the development and testing phase as a larger number of false positive can lead to significant human cost increase. I would like to pose some questions and share the points we have discovered so far from our experiments. First, can bus state one day replace human researchers?

This question is very sharp, but I believe this applies to other fields as well. The purpose of tool is to assist us in completing task more easily. The purpose of fast test existent is to allow researchers to focus more on identifying systematic risk, logic errors, integration issues and other type of vulnerabilities. AI technology is moving fast and constantly being updated. We believe that as an assistant it will keep getting better.

Bastad is an public nonprofit project. How to make it more powerful? First is data. We need more vulnerable smart contracts and attack vector ideas. And second is knowledge base needs more quality content as sources for the knowledge base.

We have also benefited from the transmission of these knowledges. and last methodology. We need more AI engineers to propose better AI identification methods. We believe that all of these are open source. Everyone can benefit from LIN.

And the last question is busted limited by the knowledge base. What if there are unknown vulnerabilities in the codebase? Currently, Bastad is also experimenting with identifying vulnerabilities that are not in the knowledge base. We have partnered with Defa Labs and the group and have made some preliminary progress during the auditing process. Bastad has already identified valid findings in every audit including high and median severity issues.

It is particularly skilled at identifying subtoc details and implementation errors in code which allowed us to detect potential vulnerabilities that might otherwise be overlooked. We believe that they have potential for development in helping researchers initially address and eliminate such issues. So let's uh summarize today's sharing. So we have two key takeaway for you about bastards. The first is data set and evaluation tool.

By combining this together, you can easily learn from the vulnerability database and test accuracy of the tool. The second is a AI workflow. We you can easily modify prompts or add any custom feature you want you need. The GUI tool we used also provide flexible togo and you can swap the model as you want. The final part is 3 minutes defa give a sneak peek of the platform that defa will launch next.

Recently, Defa He Labs has teamed up with slow mist and scans sniffer to create a public good web three anti-ishing training platform supported by an Isorian funding grant. As a member of Defa Labs, I have learned a lot and benefit greatly from this community. I am deeply grateful to all the researchers who selflessly contribute to the advancement of the web free security field. Each person's focus on security is a driving force for the progress of the industry. We also aspire to uphold Lee spirit gathering everyone's intelligence to make world three more secure together strong.

You can follow our X. We will continue to update the latest web three knowledges and activities on it. Thank you all for your time today. Start our project now. Okay, great talk.

Yeah, less has hallucination and more protection. It's great way to use AI. So, we have couple minutes for questions. Anyone has questions? Hey, uh do you have microphone?

Yeah, he goes first. Um, so you mentioned the code is open source. I'm wondering if the knowledge base is also open source and uh second question, sorry. Um, have you guys tried uh tuning fine-tuning the model or are you using a generalized model? Okay.

So the first question we open source the all the code and uh also the data set and also the workflow. So everything is open source. You can get clone it and do whatever you want. Yeah. So the second question is do we uh finetune the model?

No we uh now we use GPT4 or mini to do the showcase for sleepage vulnerabilities. So you can swap the model for needing and for sure in the future we might think about if the maybe the finetuning model uh can improve the accuracy but at least now we have the evaluation tool so everyone can know if you change the palm the result is uh you got is better or is even worse. Yeah. Cool. Another question from this gentleman.

Oh, okay. Okay. Same question. Okay. Cool.

Cool. Another question from this this one. Uh there. Yeah. Hi.

I like the unis swap examples you use because when we made the unis swap v4 cantina competition, there were hundreds of false positives that we got and that was a massive massive manual cost to go through all of them. I was wondering if you use this fast stat for any recent competitions or bug bounties and what your accuracy rate looks like on them. Uh we are now focused on knowledge based identification. Uh if the bug is not in the our not in our white list, we still experiment on it. Yeah.

Cool. We have one minute left. So if you have a short question still can accommodate. Any questions? Okay.

No. Then special thanks to Allison Day again. Thank you. Heat. Heat.

[Music]