AMA with Paritosh | EPF5 Office Hours

okay awesome yeah so let's K it off uh welcome Perry again thank you so much for joining yeah and um for having me yeah it's it's an honor to have you uh yeah uh because of course Pary is uh all over the core protocol development and research uh you can bump into him in if you're following the ACD if you're following the current devet if you've been looking into how uh the Cod protocol is being developed and tested over the past maybe more than two years at this point since before way before merch so um so yeah uh it's it's great to have you here and uh of course the thing is that many people already know you because we have had you in uh Brussels for the discussion with Terren but uh most of the people couldn't make it uh make it to Brussels so I was hoping they'll be here to to have a chance to get to know you better um and I already asked you about your background there maybe you can keep it shorter but uh let's start with with this like just how you get into ethereum where you're coming from yeah a a little bit about your story yeah um really high level I got interested in ethereum but I initially got interested in Bitcoin when I was in my bachelor's and during my masters a close friend of mine told me about ethereum and the introduction was more like imagine Bitcoin but you can do computation on top so you can build anything you like with money programmable money which blew my mind at the time and um I tried doing a small research project doing my Master's Theses and I liked it quite a bit so I spent a few years afterwards um it's locket working on iot and blockchain together and that's where I was introduced to the world of devops by a close colleague who was basically sitting next to me and we got along so um that was my introduction to the devop space and 20 20 there was a tweet by um the infamous Justin Drake about um ethereum starting a security team so I applied for the security team I did not get in but I had a bunch of ideas on how to test ethereum and how to like automate some things so uh they decided to hire me the devop team instead and since then I've been here and I mainly help with protocol upgrades so whatever needs to be done for that process uh sometimes it's more research Focus so trying to understand how many blobs can meet handle right like we arrived at the number 36 through some methodology so we had to spend some time thinking about it someone had to run the monitoring someone had to help figure out the analysis so that was me as well as a lot of people on my team and sometimes it's something like setting up a Deb net or there's a main net issue and you need to just dig into figuring out what happened so kind of jump wherever it's needed uh but whatever is related to the protocol thank you so much Perry um yeah awesome and um of course now uh there's been a lot of things happening well since then and and these days with backra testing uh can you maybe give us an overview of what all kinds of things have are you working on now uh not even in the past just like maybe D apart from like dropping the jitsy maintenance uh uh yeah what's what cool stuff is happening what you're working on yeah um so I I'll maybe phrase it a bit as uh not just the stuff I'm working on but also other people on the team because the e panda offs team tends to have a very high overlap uh because of the way we're structured uh the way we're structured is that we have a subset of people working on data a subset working on tooling and a subset working on integration and then each of us has our own vertical so we spent a decent chunk of time optimizing ourselves as much as we can as a result you can never say that one person worked on this but a group worked on a lot of things um so on the Petra front uh I've been spending a lot of time on Deet 3 um so DEET 3 is live as of last week and this includes 7702 as the EIP so in case someone's interested in trying out account abstraction and how this is going to look please feel free to experiment with that um and before we launch the devet we we went through a couple of cycles of local testing so we did that with curosis and we didn't have to change anything in curosis itself which was great but there is a testing tool we use called aera and the issue was that every time there was a new feature the testing team so Mario as well not this Mario but Mario Vega um as well as his team would write a th000 tests and then we'd write like I know 50 tests for the exact same Behavior but on different layers of the stack instead of both of us doing repetitive work we wanted to figure out how we can better harmonize it so that one person wrs the test and other people can use it um so we have a new module in aerta that can Now consume the execution spec tests so the execution spec tests become the source of Truth for the tests and then the test Nets are just one way to consume it and doing this consumption led to a certain number of bugs especially ones that happen bya the engine API that aren't triggered locally but are triggered on De Nets and local Deets and public de Nets so we figured a few bug we found a few bugs we already cleaned cleaned them up and then we had the public pet launch um now the focus is a bit more to get people to use 7702 and try and Trigger some issues there and start doing some Edge case testing with Max um so that's the Petra box that's happening at the same time pias is a active research topic as well um and in pias it's been um really interesting because there's a issue where the devet essentially works perfectly fine for about 10 epochs or so and then breaks we still don't know why so the reason the the reason of why breaks is to me not that important because I'm sure we have so many smart TS we'll figure it out given the time but um the approach that me and the team usually think about it is how could we shortened that dep time like how could we have optimized or helped people figure out tools to make this easier and one approach is the stuff that Kya is working on which is metrics so having metrics help us find visibility and whatever um blind spots are available and then if you were listening to the P call breakout room call today um Jimmy and I SP a bunch of time last week um trying to understand how to better visualize what's happening on the network right so one one thing that was missing was um which pier is supposed to have which column it's trivial if you have the pr ID but if I have 30 nodes doing this manually for 30 times on that website is really combersome so uh someone on the team just added this as a feature toor so it just this a view um I don't know if it's deployed yet will be deployed soon so you can just get a global view of who's supposed to have what information and you just load a page and you don't have to think about it again that's an example of an optimization and another one that we're spending Cycles on is when there's invalid things happening on the P2P layer how do we get visibility over it to help on the debugging side so load star might be saying that it has C it's supposed to have custody of column three and five but if I ask it for five it's not giving it to me then that's something that has to be surfaced somehow um and that that is ideally not the answer should not be the locks because that's just not a scalable solution for finding issues um so we spend was spending some time and at least the answer that I like the most is having potentially a debug name space for Network that is like an like the beacon API event stream you can have a network API event stream um and then you could use the you could like whoever is interested can subscribe to this and you could also build really cool one layer top visualiz ation so you essentially push the problem to someone else but it's in a nice abstract abstracted way that the problem is maintainable in the future and this is something that would be useful for pasas but also potentially future upgrades D in general networking changes in general there's people who are working on formal verification of the PTP layer or formal modeling of the PTP layer this would be useful for them as well so it it's an answer that is H happy for me um so yeah these are the two large buckets um there's a third large bucket eof unfortunately my knowledge of the evm is horrible chances are it's significantly worse than almost everyone on this call uh for me it's a nice black box where I send things in and things come out um Mario V is the person to talk about that box or philli from the team is the person to talk about that box um but yeah that's that's the third box that I don't have that much in yet um but yeah Philip and others on the team are looking into it I think that's most of my time spend on yeah these yeah thank you so much Pari um yeah it's a it's a wide range of things that you and the develops are working on Panda Ops um yeah and there is like yeah there is just so much great tooling and initiatives com in from you all over the place and I saw all the fellows using it to for their work or just learning about a protocol I think I think it's amazing um so I hope fellows have any some questions regarding all these initiatives I have a first one uh but folks if you have anything feel free to raise your hands I wanted to just check before we dive into all those questions about the state of the torify 44s and the era format uh whether there is I still have torrent I'm seing it so like whether I should dump it or should I get a new one or yeah the state so uh just some uh I guess context for people people who might not have heard about this um EIP 44s is an attempt at dropping history um this is super interesting because nodes right now need to store all the data right from Genesis up until now and we're starting to hit limits such that it doesn't fit in a two terabyte disc anymore um two terabyte SSD and people have to upgrade to four terabyte drives but um there's no real reason to store for example the last from the years 2015 to 2020 unless you're requesting it as long as you can ensure that the network has this information somewhere right and you can fetch it if you want to like if you're an RPC provider it's interesting if you're a home Staker not really so e44 is the first attempt at dropping for example pre-merge history and now the question is okay if I drop it from every node where a history how do I get it that answer is unfortunately not specified in E4 course so we have to get creative with it and one very famous answer is to use the portal Network for this and a lot of client teams are looking to integrate portal Network in some way or form to enable this feature set but another approach that I was suggesting is um to use a torrent so a client could just have the torrent um file built in torrent come with the benefit of its check sound everything is it's it's a proven technology it's existed for 20 something years at this point um scalable and ericon actually already uses Torrance in their client for syncing so it's also an attempt at a technology that a client team has used for the last few years with a lot of success so rather than Reinventing the wheel my Approach was like why not just take this this approach that works and apply it to something else and maybe some clients do torrent syn um or torrent EIP 44s some clients do portal Network and that's also the future that I Envision that clients start to get specialized for different use cases or allow the user to decide that would also be fine but there's no real update I think clients are still looking into it but uh Petra is so big that no one has the extra Cycles to work on this I guess yeah of course it's uh it's also like sort of a soft Fork right like I mean we don't need to includ anywhere just activate one point um yeah so so uh just to just to WR through it a bit uh I should keep the torent there is no new version because you mentioned there will be some some update to it or last time we talk there's a new version of the torrent because uh there was the proofs file missing in the torrent and now there's a version of the torrent with the proofs file inside I still don't necessarily agree with the change so the my argument was that if you're getting the thing you're proving in the torrent that doesn't really make sense to me it's like I give you a false proof and then I give you false data and you're using the proof to verify that the data is valid so it made sense for me to bake in the proofs in the client and to get the data from somewhere else you get it from two sources and then you compare that wasn't the client team's view on it so they requested the proofs to be in there so I did the same um and that's why we have mov and a way Torrance work is the moment you change one file the whole torrent file changes yeah of course so yeah I have the old one uh which is better basically I got the roots file from uh from GitHub right so from The exal Source yeah that's pretty cool yeah thanks thanks for diving into that because yeah I think it's also very interesting and we don't talk about the uh historic here much and but we have Cher waiting for a while uh do you have a question for us um so I had a question but I forgot while I was thinking about EIP 44s so I have a question on 44s instead um so uh would it make uh sense because since it's like a one out of end trash model would it make sense to like just generalize an API of sorts or an interface of sorts uh so that people can choose any kind of medium like basically I can also put it on my Google Drive Right the entire history if obviously the Google Drive does not support that bigger uh storage but I I could do that right uh does that make sense as a step in protocol yeah in general I think it does um one approach is rather than doing it bya apis you can do this bya file system in itself so you can just say this folder is where the for Force data is it's a defined format the era file format and if you have your data in Google Drive you can mount it um as like SLG Google Drive sl44 and then when you run the client you just pass it for fors data is over here so the file system already abstracts away everything you needed to okay makes sense also I uh sorry can I ask second question or does Kya want to ask yeah go on go on okay uh so regarding um like I I uh I had a look through like in the start of the cohort uh there were some tooling for uh P2P project ideas uh but in general like um since I'm dealing with P2P right now uh there's a lot of difficulty in defining test cases themselves for like a decentralized network like are there any tips or tricks that you guys employ to Define test cases what test cases are you imagining uh anything that like you know tests uh decentralized Behavior basically like in an abstract sense stimes so there's there's kind of two testing approaches right and that's also indicative of the two testing teams we have so the EF testing team the approach is very much here is a pre-state here is something that happens and here is a post state so it's very defined so in the scenario that you're talking about you would probably use Hive and you would have a pre-state and then a block comes in bya the network interface and then you have a post statee to check for something um this tends to work really well for edge cases that you know you want to implement like this is the pre-state when this block happens I want the node to fail or I want the node to succeed or I want it to import in a certain way um the second approach is I call it a bit more like Behavior be behavioral integration testing and that's the part that the e panda Ops Team tends to do where um I know that the network has to fork and it has to fork for 10 EPO I don't care what state it was in before I just want to make sure it's healthy and after 10 EPO I join the network again and then I want to see how it happens so this is like the you want to assert a certain behavior that you want the network to have um and for that uh kosis plus there's a tool called attack net um and a modified version of that should be released relatively soon or an updated version relatively soon um and that allows you to define the these behaviors and then assert that the behavior is indeed true so you can assert that the node is fine after a re York that it's fine after a network split that it's fine if ntp was changed um so it uses the chaos testing framework of kubernetes to make all of these changes um but that is a world that is still extremely new um I think we have like a lot of space for improving our Network testing stack so might not even be the answer you're looking for um but the ethereum motto still stands true if you're asking if someone is working on something the answer is you yeah makes sense yeah it answers my question pretty much yeah okay uh we had Kya before I'm not sure if you jumped into it yeah uh so you can go ahead yeah yeah I have a question uh for Perry U I was always wondering how hard for goes on the backstage is it like quite intense week or maybe it depends could you please tell more about your experience um I think there's a lot of phases so right now is kind of a ramp up phase again because we might be splitting the fork into two so suddenly it's if we Sprint a bit we can actually ship the for quickly so there's a known definition the notes are starting to get stable so for example if you came a month ago the way to find a bug is to just around a client and you find a bug so it's very trivial it doesn't take that much mental effort it's easy um so at that down time we try to focus on updating our stack making our tooling better and stuff like that today the networks are mostly stable and they're also mostly stable on the happy path so now is when it gets really hard that the payoff for the tools we've built will kind of be felt now so we have to start pushing the nodes into edge cases we have to start doing sync testing we have to start doing reog we have to start doing all of these Edge case tests um and hopefully find a book and there's no so the downside is there's no time where we can say we've done everything this will be a Flawless for will always be a guess um I think the easiest approach I have is the harder it finds to get bugs probably the more secure the FOC is so we try to wait until there's like a few Cycles where um there's nothing found um yeah and then I feel like we're kind of ready for the fork but usually just before the fork is very relaxing because once we've agreed to it and client releases are out there's not that much we can do anymore um I think we lost Kad while you were answering so I'm not sure if you got it the recording yeah I almost everything okay okay bad conne sorry yeah I get most of it anyway uh yeah thank you so much par and Daniel do have a question as well yeah a bit of an open question um in your opinion what is like the biggest disadvantage or problem with cortosis base testing and what is the biggest problem or disadvantage of definite based testing um yeah I think anything Network related is scary for me in general because so um for ebm related changes right curtosis doesn't play a big deal it's honestly most of the bugs are found on high most of them are found in execution spec tests um curtosis is more of a sanity check at that point um the devet is where it'll be public so there'll be real world latency involved I don't know the random person on Twitter might submit an interesting transaction and break stuff so that's why de Nets are really interesting from the evm layer so kosis for evm Focus upgrades for me is just sanity checking it's nothing else it's an easy way to parallelize development you don't need to wait for a lighthouse Dev to update something you can try 10 different images locally without having to disturb anyone for peer-to-peer testing like peer do that's where it gets really interesting um because you can run experiments for a longer time on different Hardware um and try and see if you can trigger a bug and an example of that is the one we're currently facing with the pias stets we still don't know it but at the same time four people parall are trying different things and maybe one person finds out what the issue is um but additionally curosis adds the ability to do these uh chaos tests which we really haven't explored much yet but I think the payoff is really high for um for example for um pias uh you can do a lot of reor you can have memory issues and things like or dropped packets or corrupted packets and stuff like this which would be really nice for a full stack of the client um but I would say 99% of the unlock comes from just these sanity checks kosis luns in the CI the paralel parallelizability ETC um so it's more like regression testing and integration testing more than anything else thank you yeah and actually one more thing I do want to add is um the initial idea when we set it up was that it's used for prototyping um which I'm hoping I do see enough people using so uh for for example if you're uh building a new Mev stack or a new me boost replacement it's it's perfect for that you already know that what you're building works over all the clients Etc there's a person who released a um it's not DVT but it's kind of like distributed um CER uh tech tech it's like a layer that's in between the validator as well as the web3 sign up I forget the exact tool um they use kosis extensively to prototype their whole thing and they knew that when they released it'll work with all the clients all the Els all the CLS um and I think part of the reason why it feels like it finds less today is that so A year ago when kosis was like relatively feature new um all we had to do was run prism with a nimus node and we found like four inconsistencies so like over the years we've cleaned these things up but uh we just didn't have such a tool to check all of these things back then so I think it fills that role really well um but yeah it has a niche it's not meant for everything one random thought I have about curtois now is that I'm actually happy that it's like third party software that it's not all pands or Hive and it's actually some somebody else doing this useful tooling for the core protocol not just not just you it's a bit more distributed yeah and yeah Cher has a question so go ahead uh yeah uh so my question was much more on uh like administrative side of things a little bit on that side maybe uh but um what is the incident response uh procedure for ethereum you know like in zero day or something like that what what do you guys do uh the first thing yeah um so there there's two things right one is there's a zero day that has not been actively exploited on Main net and two there's a zero day that has been actively exploited on Main net or main net has broken for whatever reason for stuff that has not been exp exploited that's clearly the domain of the security team uh so we have security at ethereum.org there's a bug Bounty uh there's a vitalic on vacation so frally vitalic never on the incident response calls or the even the groups to be honest I don't think he is that person um yeah so the security team is mostly responsible for that um they are the ones that will validate the zero day figure out with Clan teams how to Pat stuff all of that is their domain chances are I won't even hear about a lot of zero days there was a nice paper that someone shared today I didn't even know about the attack um which was perfect when stuff happens on Main net um there's typically two to three ways we get notified so we run a lot of nodes we have alerting in inhouse uh so when something happens on me that chances are we've been alerted ASAP and based on what the severity is or what's going on we also have some fine grain alerting so for example if mainnet participation was to drop even by 10% we get an alert but clearly that is critical enough to wake someone up at 3: a.m. it's probably a client issue we can deal with it in due time but if May net stops finalizing that's basically all hands on deck we need to start dealing with this ASAP um rule of thumb uh the every main net issue has happened either when I'm on vacation with my partner or on a Saturday or a Sunday because the last time it happened he was like is this a rule like when we're spending time in breaks but um yeah uh I think the only big incident we've had was the um issue where me stopped finalizing Last year me um it was an issue with prism and how it tends to work is uh alerts go off we tend to create a small group to begin with to start looking at whatever is going on um if uh we identify that it's we try to identify if it's a CL issue or an El issue because the competencies to debug both of them are very different and that tends to be quite apparent so just look looking at which clients are broken or what the log say will already tell you that um then based on if it's a CL or El issue we uh add more people to the group more CL team or more El team there's of course enough people from both sides but just the mass of the group will change based on which side it has the issue um and then start to narrow down what's going on um is there some common Trend are all prism nodes offline are all X nodes offline Etc and the way we tend to do that is we have a lot of um data analysis tools like zatu is an example of one that we tend to use or we tend to use Dora or we have like an internal version of Dora or uh just public explorers or Cent teams running their own nodes um so then we look at what's going on there and then dig more and more to see what the common factor is and find stuff um I think the e panda Ops Team or client devs in general are only useful for the first like 15 20 minutes at some point you identify which client is broken and then it's honestly completely up to them and we're just playing support so if they say hey can I have a trace from that node we fetch it and we give it to them or um hey can you run this patch and see if it fits that's that's all we can do um most of my work is spent in like the first 15 20 minutes organizing things um after that chances of me finding out what exactly went wrong are very low cool cool makes sense and also the other thing is if all of our tooling fails and all of our alerts fail usually Twitter is quite responsive so yeah and um yeah the I always had the respect like when there was even some minor incident teams are able to put out hot fixes in like so it's always always a great work earlier this year was actually very interesting that there was five weeks in a row where every week some other client had a mainnet incident um and this was before we did the den fork or we were agreeing on doing the Denon fork and literally it was even on the weekend and at some point we were making fun about how these things always happen on the weekend there was a bzo issue there was a nimbus uh yeah Nimbus issue there was nethermind and then there was two other client teams that had issues on Main net releases but everyone hot fixed like with hours um I think the bezu issue we even yeah we had like a really nice record for that within 30 minutes we had identified what the problem was and within an hour we had a release out something like that it was very impressive yeah um awesome uh thank you so much Perry we have question waiting from Dan hey bu um are we going to have Petra split in scheduled into Forks probably and if that's the case uh what's the what's the procedure will defet tree be taken to uh you know delivery point and in the meantime work on four or with the rest or what's the yeah um I've been a fan of splitting the fork for a while um I think there was a position paper also from the team came like 6 months ago where I was a fan of two small FKS instead of one big one um so I'm very much in support of two small FKS but the final decision will still lie on allord devs this week um I'm trying to get a temperature check from different breakout rooms Etc it seems like most client devs are also in agreement that two fo is a good idea um that being said uh Petra devet 3 that has all the eips that we want to ship but there are still changes and there are updates to the spec that need to be made PR that have been merged so we will probably do devet 4 just with updated specs um and that would be considered feature freeze and the only things that will be put forward are Builder API changes remote sign API changes and bugs that we find on the template so we would ideally not want to keep working on the spec beyond what stage DEET 4 would be we don't want to stop feas so pias will get its own devet cycle and that's based off of the last release of the Petra cycle and we're still working the organization out for this but essentially right now Petra is on CL specs 1.5 so pias will say okay we're were working off of Cl 1.

5 Das Petra right oh dpas same for the eof guys so they will say execution spec test has a release eof version 1.6 and um that's the version that we pin for eof Deets ideally sooner rather than later eof and pias start becoming one DEET but for now it still makes sense to keep us true so we would have a grand total of four de Nets pectra eof pias work right right and and between three and four do you have like a rough estimation will it be some months to I think it should be on the weeks timeline uh that's what we were discussing how uh today as to how much work some of the eips are um I think it would be weeks but one of the things I want to make sure is in devet 4 is the Builder spec changes that hasn't been done at all yet and in my opinion is a blocker um so it would make sense for someone to First work on them to already have the implementation in devet 4 because then we can also have the builders and relays start updating and getting ready for the fork and they'll have a testing Target okay thank you yeah awesome thank you um yeah before we get to the comment questions mad do you have question um yes I have a question I want to know um out of here is it though how do you manage um log aggregation in terms of troubleshooting multiple clients at the same time yeah so uh we use this tool called Loki um and promp tail is the so we have this tiny agent that runs on all of the our machines that we run it takes the logs and then it passes it forward to a central server this Central server is running a software called Loki and what Loki does is it puts it in a nice database along with all the information it's a Time series database and if you're in the devops World Time series is a really nice thing to have because it's compatible with Prometheus and Prometheus is what we use for metrics so we have grafana which is a dashboard visualization interface and it can visualize any sort of Time series which means you can put in logs you can put in metrics you can do whatever so if for example you are reacting to an incident you basically go to grafana do observability Pand ops. you can expect that metrics from all of the instances are there you can expect that logs from all of the instances are there we run this as a service for all client teams so everyone from different client teams get access um some dashboards are public but not all um if you're interested in digging in nothing here nothing in this um stack is secret so it's it's like very loose privilege um so if an are fellows like herey I want to take through stuff we give them access um there's nothing that you can really break there's nothing sensitive there zero Val uh yeah yeah go ahead if you want to follow up okay um what about approaches in handling um some kind of sign like in terms of recovery of contious clients like how does that happen I mean what do you mean by the C um sing in terms of syncing or yes like how how how do how do you guys handle it yes um when there's a main net issue or when could you maybe expand on the question in general if it does happen in general if there's a main net issue and we have to recover the consensus side yeah yeah so uh um if there's a mainnet issue there's one of two things one is um the client can have a patch and the patch will allow you to continue going on same F right the second one is if a node is stuck on an imported block and no patch can revert this block in that scenario you need to do a resync but if it's a resync then the clients would have to make a release and specify in the release that everyone affected has to resync this um and since we mainly do checkpoint sync that shouldn't be too much of a strain on the network in itself thank you um K and we lost her again okay um so yeah to the common questions there was a first one from Rahul about uh whether you use AI for some data analysis or yeah what sort of tooling maybe um there's people who are who who've kind of talked about in the past but I don't think anyone's um actively done it yet so if you're interested please feel free uh there's a ton of data out there we also publish all of the data so in general check out the website um e ops. and if you go to data we have data sets um if you have a very specific test or very specific analysis or something you're looking into we can also provide you direct access to the click house cluster so we're using click house to store the metrics um but if otherwise you can just download the park files and run the analysis locally um we are not using any AI tools at the moment but yeah maybe in the future we would use some sort of analysis but in the end uh you would need it mostly for anomaly detection but P2P networks or ethereum in general tends to be so noisy that anomal detection can be too fine grained and we want to be careful about like false negatives um and we just stick to like basic alerts like participation drop is a good proxy for something that's wrong but yeah there's probably a lot of low hanging fruit that we could get with better analysis tools yeah if you want to right yeah you mentioned you work with um paret file I believe um you are logging a lot of archival data uh and the size of archive matters to you um have you done um like experiments with what kind of um data Works um um like is it Arrow or pet and what was what are the pros and cons between these files formats yeah um we have done analysis I only know the outcome of the analysis um reach out to either Sam or Andrew on the team who are in charge of that whole world um one of the other reasons for us choosing par was it's also just a well understood standard format uh flashbox is releasing some stuff as SP format we are releasing some stuff um there's like Paradigm also has some data sets as par so it was also part of the approach of like let's not build something new because if I have an analysis pipeline two things are parade and one is something else that doesn't really work out let's just standardize and use the same thing um we also do some cross importing so we import The Flash pods data set into our data set um most of it is requirement driven um there's the arc team uh so applied res research group at EF um and Tony is like a super user of our data set um so if he's we try to cater to the people who are doing the analysis so if um yeah if they require this data set in a certain format then we try to achieve that and yeah in general we haven't noticed much issues with par or with thck house so we've stuck with both of the stacks on that note I think um one issue I fa with Pak is that it doesn't support appending new data um have you have you faced the same issue is there a work around no because we particularly use par as our export format so we've never had the issue where we had to append stuff to it we just export it again and make it as a new data set um all the appending and all of that operations have been one layer higher in uh click house where that's our real data source so if you're someone who's looking a lot into Data analysis then we can give you direct access to click house if you want and then you can use your jupyter notebooks against the cloud that has everything already set up for you cool and lastly do you prefer working from an office or you like working from home I I think I like um a mix so office days are my social days I try to have like a long lunch meet with people stuff like that so it tends to be one day in the week and then the other four days are like at home so mi nice thank you guys um yeah we have many questions come in in both chat let's let's prioritize the guys on the call Kya you've been waiting for a while yeah thank you uh what happens if uh the validator has been slashed and it's not his fault uh if there were any issues while updating consensus clients is it a use case or or not do you investigate every slash accident or we try to keep an eye on why slashing happened especially if it's something unexpected um at least in the last few years it was never anything unexpected the we don't necessarily do the entire analysis ourself um the each St repairs have been extremely helpful because every time there's an issue they especially with home stakers they tend to spend a lot of time talking to them and understanding why the problem happened so far it has it has never been a result of a client making a mistake purely because uh clients have this file called a slashing DV so every time you sign something you append an entry in the slashing database so the next time you update it first reads in the slashing database if you're going to sign something that would slash you and if it's true then you won't do it um usually the issue comes when people try to get really creative or set this up wrong or they have multiple slashing databases or multiple keys in multiple locations um but if you have a really like dumb setup with one node one slashing database chances of something going on are quite low it's very easy primitive so no accidents with the updates clients updates right no hasn't happened so far that's good that's good thank you yeah it has happened with updates on kubernetes but it's because uh they had set up the slashing database protection wrong so it's like a super user making a mistake thank you um BMA you also question in the comments but do you want to you want to read that or yeah okay this is a different question I wanted to know um is there like um will I say a guideline that you guys have in place for network based attack on the consens layer is there something you guys do to in terms of securing it or if something happens this is what we're going to do terms of network based attacks I don't think there's a very strong guideline it's more or less just the steps that I detailed so if there's an issue create a group try and find the minimum set of people who need to be informed and then um start digging into what the problem is and I guess try and increase the variables in the set until you find what has happened um so there's no real like guide book I think but that's a very similar process for any debugging you try to minimize the number of variables as much as possible add variables one by one until you find what the issue is I think in more like corporate environment or like centralized structures you have sort of a checklist like you know file you have to like okay we have this error I go through this checklist right but in ethereum like that's just not possible there's no one person to fix anything needs to be a coordination every time so you need to start just talking to people yeah um last time there was actually even this issue um that Tony and I triggered that ended up being a main net update but we triggered it on sepolia so Tony had this really nice transaction um he wanted to just see how this transaction would work on a de net and the issue is that get has a limit it won't accept this transaction so he tried submitting it it didn't work and he's like hey P can you figure out a way to submit this transaction I knew that bezo has an overridable limit so I tried overriding it on abzu nodes and it wasn't possible but I spoke to the bezu team who also run nodes on I think it was sepolia and they managed to um update it and get it to run on their system um so transaction went through you were like yeah cool easy um left left to the evening and at some point someone on Twitter noticed that a bunch of nodes were offline on zolia Chase he works at quick node and their monitoring went off before our monitoring did so he reached out and um I was like yeah what happened and then we um we got we got the basic group up and running and we started debugging okay this is the transaction and immediately it hit a flag yeah this is the transaction Tony wanted submitted what did you do Tony and then added him in the group um then the next thing we figured out was that it was an engine API limitation because we bypassed the Json RPC limitation on one node but the engine API had a different limitation and then it also hits you that this is exploitable on mainnet so this technically could have taken down Main net very easily um and because it's live on sepolia someone who spends a lot of time analyzing it could have figured it out and exploited this on me so we had to do this very quickly um then the question comes in yeah ex AI alerting is some on Twitter um then the question comes in how do you do a roll out right um if get updates saying fixes engine API limit this basically means someone's going to like roll this uh someone's going to exploit this in no time so you have to hide it amongst other changes and thankfully we had uh the dun upgrade relatively soon so we shipped the change under the GU of the dun upgrade so the fix was in the dankun upgrade technically and then the question is also how do you fix it do all clients need a patch do one client need a patch it was an interesting evening and not what one would have expected but yeah you have to be creative it's always hard discussion with these like zero days or or this this sort of fixes in clients where you you fix it in a client and you tell people like couple of weeks later or so when there's been like three years ago when something was three there was this discussion in in get how to even handle these because because you want to be honest but then somebody triggers to an accident and so on um yeah yeah but it's really cool because par like literally two weeks ago we had Tony here uh who at one point yeah I had this transaction and you know there par he's the greatest guy in the T I love par so much I just send him this transaction and he makes it work for me yeah it was so we just learned the the other side of the the story it's it's great thanks B yeah especially that transaction cuz I didn't think about it being an exploit no one thought about it being an exp Lo um but it also introduced a new class of attacks cuz now we know that if you put a limit on the engineer pii you also have to put it on the jonc or vice versa which yeah I guess you have to break some eggs to learn how to cook um so we have last few minutes uh Rahul do you want to do you want to finish maybe last question yeah sure um yeah so you mentioned um like exploiting on on online letor would uh probably expose the bug to everyone else have you seen a real world use case of that like a bug that was out in the open and someone figured it out and tried to exploit it um I think the incentive for breaking protocol itself is not too clear but examples of someone testing an exploit to I don't know Unis swap or like some d app definitely people have done that before and they've like if you do some chain analysis you'll see that someone has tried a similar thing on a test net before doing the on meain net um so 100% it the process exists um but breaking something on the protocol level I haven't seen before I think it happened three years ago um after I think it was after uh London or so remember there was up triggering something uh which was fixed uh but then there were like two of these incidents uh one one nethermind for oh man it's been a while but I will look I think somebody found just looked at a diff and like oh what is this doing here and maybe he was m i I agree what I wanted to emphasize yeah there is no direct incentive like you if you if you manage to find vulnerability in some def application you can directly make money but just like preventing few valid to lose money it's not going make you anything yeah I also think there was something with nether mind with the diff but I also wasn't sure what it was there was actually one uh Nimbus buug that was very interesting the um um I don't know if it was a mainnet issue or something the client have tried a bunch to figure out what the issue was we tried a bunch we couldn't find it so we just did like a binary search we kept rolling back which version we used until we found the early EST version that didn't have the issue and then B researched our way to the commit that was broken it took about 4 hours also really usedful DSis for this because we could just keep doing the exploit and keep doing it um and then we found the exact commit and then we um the person F was able to figure out what the bug was and nimus patched it that was I think the funniest like I don't know after a few hours of debugging it I'm also feeling useless I'm like cool I'm just going to buy and research where the issue was very well uh we are at full hour we got a w it up here I'm sure people have more more questions in follow-ups but uh it's been yeah got to switch another call as well uh so thank you so much yeah it was great uh um it's it's fun to sh with you as always and um U I hope folks if you have any more questions there are some that we didn't manage to answer from the chat but par is available he's he's here he's uh happy to chat so in Discord uh we can open it thread we can follow up there but yeah uh thank you so much for the for the hour AMA I really appreciate it uh yeah and for hope it was enjoyable yeah I I I I believe we learn a lot and also get inspired uh yeah guys go ahead and break things uh have a great week we'll see you next week break things not on Main net break things on yay yeah go ahead break the break the dev NS um yeah awesome and uh you you hopefully you most of you folks will have a chance to meet party in Devcon so in like only less than two months I guess yeah uh we'll see around yeah thanks everyone uh thanks everyone for the questions and per thanks again for and again if someone wants access to the data to do some analysis please reach out we're happy to help ourf Fells with that awesome thanks a lot uh yeah thanks everyone have a great day see you byebye thanks