EPF 6 Office Hours | Eitan

Awesome. Very well. Welcome everyone again. Uh thank you so much for joining us for the office hours number 11 uh with our guest Eton today. Uh so yeah, thank you Eton for being here.

It's an honor. Um yeah.

Yeah.

So Eton is um uh is uh a former EPF fellow from the cohort 4 um not that long ago actually and now working in Sigma Prime working on the lighthouse consensus plant. So uh yeah, thank you so much again for joining. It's uh uh it's really great to have a successful follow here, somebody to uh share also their journey, their perspective, their experience and uh I think something that people can relate uh more to here. Um so um let me start with that maybe like uh uh if you if you recall what was it like uh when you were starting uh what brought you to the EPF like what was uh what was your uh uh motivation to start? How was the how was the program for you?

Um yeah,

your journey.

Yeah. So, man, this was probably like three years ago. Um I was at the time working for some uh web 3 company. Um doing some software development and uh one of my co-workers told me about the EPF and this was like mid through midway through EPF3. Um, so I had just kind of joined a couple calls just listening in.

I didn't I didn't really participate in anything. Um, I was just kind of listening in and learning about it more. Um, and when a little bit before EPF4 came around, um, I started getting interested again and wanted to actually participate this time. Um, and so I, uh, opened a couple PRs on Lighthouse. really really small PRs like I don't know I think the first PR was like fixing some blogs um and then a couple other you know smaller ones and applied to EP4 and um I got in as a permission fellow um and yeah and just went from there.

Yeah, very well. Thank you. Thank you. Um yeah, and I think people can relate to that. Like I um I see this quite often that people hear about EPF and it takes them a while to be able to join and actually dive into it.

So um yeah, that's interesting. Thank you so much. And now um uh yeah, how about how about after like your uh your EPF experience or how was what was your experience like getting started in Lighthouse after? Um

yeah. Yeah. So like you know I had contributed a little bit to Lighthouse before the EPF. Um and uh one of the you know things that kind of um I learned because of like listening in on the EPF like the year before was that there was like actually like multiple client implementations. Um I hadn't known that before.

Um you know I thought there was like GU and you know whatever consensus client and that's it. Um I think this was like kind of around the merge when I like learned about that. Um so that like really piqued my interest. Um, and it was also at the same time where I started learning about um, like Rust, the programming language, a little bit. And so I think I was I was looking at Reth and Lighthouse trying to pick uh, which client to um, contribute to.

And uh, Reth at the time, I don't know if it's still today, like it was very competitive like trying to get issues. There was just so many external contributors. Like an issue would pop up and within like five minutes someone would take it. Um, it was kind of insane. Um, and so, uh, and then on Lighthouse, there was a lot of like good first issues, um, and less competition over those issues.

Um, so I picked up a few and, um, when I remember I opened like my first couple PRs and I was actually getting feedback from the Lighthouse team. Um, which was kind of like crazy to me. like I'm just like this random stranger on the internet and these you know pretty good software engineers are like giving me feedback and telling me how to you know oh you need to fix this or this is you know better and so it was also like a really good way for me to learn about Rust um and yeah I was just I never contributed to open source software before either um so I was kind of new to all of that um but yeah just like I felt like I was getting you know a little bit of attention on my PRs and like feedback back and it was like I felt like I should be like honestly paying for that kind of u feedback like it was like they were like teaching me in a way you know I mean obviously I was putting in effort um um but yeah after those first couple PRs after like getting feedback and um I just yeah decided to go work on Lighthouse.

Yeah. Awesome. Thank you so much. Um yeah that's also I think very good advice very interesting way how uh yeah you chose the client and the right places to contribute to because sometimes like people are drawn to the projects that are the most overhyped but it's also it means it's much more competitive to actually contribute there uh making harder that's that's interesting thank you um and uh yeah and what about right now so what are you working on these days what is your role within the team what part of the client what are you excited about

yeah um So, I mean, the last month or so, the whole team's been pretty focused on uh on Fulu, so getting purees um ready for the test nets. Um we're really trying hard to hit a couple deadlines. Um so that you know, Fulu is you know, forked on time and all of that. Um so, uh Jimmy, which was a EPF fellow in cohort 3, he's kind of leading he's been leading the the Pure Dash efforts. Um, so me and a couple other guys on the team have just been, you know, assisting him.

Um, trying to push optimizations out the door. We've been working a lot on like, uh, there's been a lot of sync issues that we've had, uh, because of purees. Um, so trying to fix those. Um, beforehand, I was doing a lot of work on Fossil. Um, so we, me and a couple of, um, other client teams were able to like set up a little like local test net and interact with each other.

um with like the fossil changes um which is cool. And I think mostly what I do is like a lot of a lot of different random things sometimes. Um so it could be like database related stuff. Um maybe um you know actual EIPs. Um small optimizations, lots of bug fixes.

Um yeah. So I'm kind of all over the place a lot of the times and um just try to like help where I can.

Ah, you're a soldier. I see. Awesome. Uh uh it's it's alo interesting that I mean you were working with Jimmy. I mean uh that's also cool and there is Daniel as well.

Uh I think we have three people in Lighthouse coming from different EPF cohorts. So yeah in the future the goal is to have like more people from EPF than non EPF in client teams. That's awesome. Cool. Um and maybe one more question before I open it up for the audience like uh if you are looking back at this experience of the transitioning and just like working with the client team like uh what what was surprising or maybe what is the difference between like doing the EPF and um like contributing as a part of the EPF and actually being part of the client team like what uh yeah like how do how does it feel different?

What does surprising or something? Um

yeah to be aware of

so you know during the EPF um I had worked on a on a on a couple different things like for my project plus like a lot of like really small PRs. Um, so I'd already like not become like super familiar with the codebase, but I had like a good understanding of like where things were. And um, I remember my first day of work at Sigma Prime, like when I had been onboarded to previous like um, companies, like you know, that process took a while. Like it was the first time looking at the codebase. So, it's kind of interesting like showing up to a company and I've already like been working on this codebase for like two or three months.

So, I felt like I was kind of ahead that way. Um, and I guess like the biggest difference is um, you know, because I'm working for Sigma Prime, you know, I can schedule calls with my co-workers whenever I want. Uh, we have meetings and so there's much more like uh, direct line of communication with the rest of the rest of the team. Whereas, you know, when you're an external contributor, you're kind of like outside looking in. um might not be able to to meet as much with some of the external contributors or get as much like attention I guess because you know they're they're off you know working their their day job or whatever.

Um but as a co-orker you know like um you know I have weekly meetings with Michael and Jimmy and weekly meetings with the rest of the team. Um and so yeah, I'm just their their co-orker. So I guess there's there's that dynamic that's just a little different.

Yeah, I got it. Yeah, being even closer to them of course. Yeah, makes sense. Awesome. Awesome.

Yeah, thanks. And um yeah, let me open this for the audience like we have some questions in the issue as well uh from folks who couldn't make it. But uh first I'm wondering if there is someone eager to ask a question feel free to raise your hand and um feel free to ask or write in the chat as well. if you cannot use the mic and if nobody's uh ready for questions yet. Okay, we got some already.

Okay, let's get started. I think Shane was first. Yeah, I'm Shane. I'm working with uh Mark over at Lighthouse implementing ETBS. And um yeah, I also read in your like cohort for I think review like for your final after you completed it that um you had done a good bit of like contributions to Lighthouse while also doing uh your main project.

I'm curious like you think it was very beneficial with doing those other like small uh PRs to maybe get more familiar with other folks from the team as opposed to like your core mentor. Uh I'm just curious like how much of a differentiator you think like doing just a variety of tasks at the same time was.

Yeah. Yeah. I guess um one thing I noticed when I was an external contributor is like how like the Lighthouse team was working on like a bunch of different things at the same time. Um and that's that kind of happens you know at a lot of different companies as well. And so um I kind of just wanted to show them that I could do that too that I could you know multitask and and work on a lot of different things.

Um, my personal situation was I had a lot of free time to do this. So, like I don't think I don't think anyone expects you to, you know, be opening a bunch of PRs and and all of that, but I did have extra time and I was, you know, working weekends sometimes. Um, and yeah, my my whole kind of mentality was if I just like keep opening PRs, like, you know, maybe maybe they'll hire me eventually. But, um, I don't know if like that's the the best like way to go about it. And it really depends on your personal situation and I don't think anyone expects you to do that either.

Um this was just me maybe kind of taking it um a little too far maybe. I don't know.

Yeah.

No, you got more experience also with the codebase like you were saying it was easier to start uh once you've already touched a few different things and also you build confidence as well from having done a variety of things as opposed to just like DB work for example the the entirety of the cohort.

For sure. And I I was I guess I was trying to tap into like things that I'm good at. Like I'm good at just like you know maybe like working long hours like grinding things out and like you know I'm not I'm not like the the smartest person in the world but I have my own strengths. Um and I was just trying to like lean into those. Um

um and so I think that that was my strategy. Yeah.

Get done. Nice.

Yeah.

Awesome. Uh yeah. Um okay before we jump to the GitHub questions Lee has another one. Uh yeah, thanks uh thanks also for your previous answer. That's also why I'm question I was c uh curious about and my question is more about the pure dust because I'm working on D and I also asked this question before.

So did you identify any like challenges or any issues you want to uh like you think like pure DOS has or you want something new for the for us in the future.

Yeah.

Um so like challenges we've recently identified with pure I think I think pure is a is a very like big change because it touches a lot of different things. Um so um you know uh block production um a little bit of fork choice um sync sync's a big one um you know with uh in DAB when uh blobs were introduced um there was like a pretty big change to sync but now that col uh validators are are only custoing like a subset of columns it makes sync even more complicated because the amount of peers that can like um there's only certain peers that can serve you those columns during sync. Um and so for example during DevNet 3 uh we um there was a bug in one of the clients I don't remember the exact details um but we got into like non-finality and it became like a very like forky chain there was just like a ton of forks everywhere and Lighthouse was really struggling syncing uh because we were having not only trouble finding peers on the right chain but finding peers that are custodying the columns that we need to custody on the right chain. Um and so over the last like three weeks we've actually like overhauled sync pretty dramatically. Um one of the big changes is we were using um uh data columns by range RPC requests to to fetch columns for like a range of uh slots.

Um and on for key chains it becomes very difficult. Um and so we actually change things to where we first request blocks by range and then request data columns by root. So you can request like a set of data columns by the block route. Um and it makes it that made it much easier to um sync on like these non-finality forky chains. Uh there's been a lot of other like uh optimizations that we had to do.

For example, KZG verification is a pretty like computationally expensive process. Uh for super nodes reconstruction is a very computationally expensive process. So we really had to like um like dig in and figure out okay when should a super node reconstruct or with KZG verification how many CPU threads should we be using um to do KZG verification. Um for example during like forward sync you want to sync as quickly as you can. So we'll let the node take as many CPU threads as it needs to uh perform KCG verification.

Um but during backfill sync once the node is already synced and can um you know participate in the network um some nodes will start backfill syncing as well. Um we don't want to use as many CPU threads because it can um uh it can make like validator performance um you know worse because you're taking up all these CPU threads to do KCG verification instead of like a testing or whatever other validator duties you need to do. Um, so yeah, there's been a lot of challenges there and a lot of them have kind of recently been uh we've recently learned about mostly because of like the non-finality issues. Um, which is good that we're figuring them out now and not later. Um, so yeah, we've that's really what we've been doing the last two or three weeks.

Um, and I I think there was a second part to your question, but I don't remember it.

Uh, yeah. Uh what's uh like for the folders? What do you think would be nice to have in the future?

Oh, like make it simple or anything?

Yeah. So, I think I think right now with Pure DAS, we have like this uh BO schedule where we're increasing the blob count um on like an interval. Um, and I'm not sure if like full DAS is really needed at the moment. Uh, based on like the blob counts we're trying to increase to and a lot of the metrics that we've been looking at, I don't think full DAS is going to be needed anytime in the immediate future. Um, I'm not sure at what blob count like it becomes much more important, but with the numbers we're looking at and the numbers we want to get to, um, yeah, uh, full dash isn't like uh, as important uh, at the moment.

And I think um one of the EF researchers actually wrote a post about that on the um uh ETH R&D discord in the uh data sharding channel. Um if I find it, I can try linking to it. Um he explained like exactly like with the numbers like why full isn't needed in the in the immediate immediate future.

Uh yeah, agree. And actually for that also still need a lot of preparation theoretical wise and also uh architectural wise. Uh yeah uh if we have time I'd like to know more about the uh finan issue you have uh faced like that fork issue you have faced during the period last development but it depends on if we have enough time this uh let's call

yeah and if during this call, you can like always feel free to message me and I can, you know, add more details.

Yeah, you can also ask in the discord channel. Um, but uh yeah, I think we have enough time for questions. We have 40 minutes to go, but we have a couple of questions in the the GitHub I wanted to get into. Uh there's three of them. So maybe just let's just take a first one because this is something that you also partially discussed with Shane.

Um since uh second uh yeah since you worked on multiple parts uh of the consensus client during and after EPF which area gave you the most confidence in your closest client knowledge which part of the client would you recommend an EPF participant uh to focus on the most?

Yeah, that's not sure where I'd recommend to focus on the most. Um, so like when I was an EPF follow, the thing I first started focusing on was the was the beacon API and the and the database stuff because those were things I was familiar with in my previous work experience. Like you know, if you're a software dev, you've probably spun up some APIs or you've probably done some database stuff. Um, so like I didn't want to dive into like fork choice or you know because I've just never even had any experience with that and it was kind of like this kind of scary place to look at. Uh but the ap the API stuff was like very familiar.

The database stuff was very familiar and like one of the things that actually gave me the most confidence was actually uh actually maybe like six or eight months ago I don't remember exactly was during the Holski non-finality incident. Um so like Holki because of like uh some client bugs a majority of the network attested to um an invalid block. Um, and that just caused like a lot of chaos. Uh, consensus clients couldn't sync. Um, and Lighthouse itself had a ton of issues.

So, like because we were in non-finity for so long, um, we were hitting like out of memory issues. Um, yeah, our client would just like crashing over and over and over again. And so, um, you know, we as a team kind of sat in a war room and just started trying to like fix all these different issues. Um, and it was kind of like a stressful time. We were kind of treating trying to treat it as if like mainet exploded.

I mean, I think if mainet exploded, we would have probably done maybe worked more hours than we would have, but um, we were like just trying to, you know, play along with it and recover as quickly as we could. Um, and I learned a lot during that process. Um, I think just getting thrown in trying to like debug things that maybe I never even looked at before or thought about um really like gave me like a maybe a deeper understanding of Lighthouse than I had before. Um, and it was really only like a period of like a week or two when we were, you know, doing all of this work. But because it was so like um I guess hands-on and intense in a way and because we were sitting in a war room with each other just like constantly talking and working on like all these fixes a lot of like different parts of the codebase um it just gave me um a lot of confidence and I think I learned more in like those couple weeks and I than I had in like a long time.

Awesome. Thank you so much, Ethan. Uh yeah, that's that's a really that's really great. I mean, it's a great advice to start from with what is familiar. Uh but also like the the Hollleski incident, I think it was a learning experience for everyone.

And the cool part is that we even have one fellow who is now working on the improving the the caching in Lighthouse to help future nonfight incidents and he has a hand up. So auden if you want to ask uh this is this is exactly your time.

Thank you Mario. So hey Ethan I am auden um am I audible firstly?

Good man. How are you?

Yeah I am doing great. So yeah um like I have radio sorry radio weekly updates and like your entire journey from being a cohort member to being in lighthouse and single time. So u a few questions I would like to ask. So like you have been working on the slasherdb implementation when you are in the cohort. So like apart from working on the database how did you like as you said you are working on a like a lot of PRs and issues since you had some time.

So was solving issues and PRs the way you got like familiar to other parts like the validator client and other parts or like you were also going through the entire code base with time like how did you get familiar to other parts like the validator client or other sections? Yeah, I mean I think I think everyone probably has like their own way of learning a codebase. Um for me, I learned the codebase by working on specific issues. So like I actually when I started I was like intentionally ignoring different parts of the codebase. Like if I was working on the beacon API, I was just looking at the beacon API.

I wasn't trying to like learn everything. And uh to be fair, there's probably still parts of the codebase that I'm not like super familiar with. Um I think that that process takes a lot of time. Um, and you know, Lighthouse is a pretty complex piece of software. Um, and so yeah, my my strategy was like you just take it like small pieces at a time.

So like if I'm working one one of the things I worked out was um kind of the the block production flow. So in DANB uh we we moved to like uh like the V3 endpoint for block production and it like kind of changed a few things along the way with uh how Lighthouse does its block production flow. And so like um when I worked on that issue, that's when I learned okay, how does you know block reduction work at a high level and I anytime I was working on an issue, I was just trying to learn about like specifically what I needed to get that issue done instead of like trying to figure out everything. Um, and so like my strategy was, yeah, to literally ignore parts of the codebase, not not to spend too much time trying to learn everything and like, you know, accepting the fact that I'm not going to know everything right away and just like trying to just take it small chunks at a time.

Uh, yeah, that makes sense. And also like speaking of the hoski non-finite incident like that was a like obviously a good learning curve and like you guys work together and apart from that like were there any other like struggles that happened during your the course of time you've been working with sigma prime or lighthouse where like you faced some struggles and you worked a lot and like learned a lot in that process.

Yeah. Yeah. Um there's been a lot of struggles for sure. Um, and another like cool learning experience for me I guess was during uh we had like an interop event in Kenya um where all the client teams like kind of sat in a room and um we're trying to ship like uh the Electra Devet Zero. Um, and that whole week I I had worked on like some of the actistation changes that were uh involved with Electra, which um on the surface looked like a pretty small EIP, but because act because that change touched so many places in the codebase, it actually became like this a much bigger um yeah feature than we thought it was going to be.

And um it was very like delicate in a way. Like small bugs were hard to like it it was hard to figure out like what was broken at first. And so you know we I literally spent most of like the interop in Kenya just like banging my head against the computer trying to get it trying to get things to work. And um what's nice is I have you know co-workers that you know have more experience than me with the codebase. Um and so anytime I get stuck you know I can reach out and like ask questions.

Um, and yeah, because we were all on the same roof, we ended up getting, you know, an Electrode Devet shipped by the end of the interop. Um, but yeah, there's always there's always moments where like I get stuck. Um, or I can't figure out why things are working. And, you know, sometimes it's just like sitting in in front of a computer for like four or five hours just running the same test case again over and over. Um, and looking at logs and trying to figure out why it's not working.

Um, sometimes, you know, it's asking questions. Um, but yeah, it's always like, you know, it always has its like ups and downs. Like, you always think, "Oh, now I finally, you know, figured everything out. Uh, it's going to be smooth sailing from here." And then you pick up another issue and you get stuck again on something.

Um, so yeah.

Yeah, makes sense. like yeah I'm also like struggling right now with something like that since I like implemented the memory aware caching part and I was seeing that yeah the memory like that thing is actually working and then I send the logs to Michael and he said like I'm not calculating most of the beacon states fields so I was like oh I'll have to work on it again

yeah and like the team is very helpful as well I have learned a lot from you already

that's awesome

yeah thank you so much

awesome that's really cool Yeah, and it's really great to see by the way that uh people are actually reading the previous fellows uh uh development updates diaries there. So it works. It's really great to see. Awesome. Awesome.

Um okay, let me get uh I mean feel free to raise your hands guys. Uh but meanwhile, let me read another question from GitHub. This is a long one. It has three questions in it. Maybe let me take it one by one.

uh Lucasin who is working on Tokyo tracing in Grand uh uh has some questions about the tracing implementation. Uh from your experience uh adding tracing to Lighthouse, what have been the most practical benefits? Uh which types of spans of structured fields did you find uh most useful for diagnosing network uh layer peer-to-peer issues? Yeah. So tracing has been really nice for some of the peer desk changes lately that we've been um working on.

Um so uh adding spans to like the whole um um KZG verification process. Well, we call it like uh chain processing during sync where we receive like a chain of blocks and columns and then we do like a bunch of verification. And so spans have like given us like very a very granular view of like how long things are taking, what exactly is going on. Um whereas opposed before tracing we kind of just relied on metrics um which is just it just told you like how long the specific process t took but with like a span you have like this like very detailed view of like all the function calls, how long each one of them took. Um and it's been really really nice.

Um we've used it in like the block production flow. Um I think we want to add spans to some fork choice stuff in the near future. Um so yeah like I think uh initially tracing was actually added by an EPF fellow. Um and it's a very it's it's a big grind to to work on that feature because um yeah there's just a lot a lot of like code changes a lot of like small like little details to look at. Um and when uh when we added tracing uh we we it wasn't just the fellow, it was also us.

We had done a couple things wrong. And so like some of the spans that we had implemented kind of like ended up being these long lived spans that were taking up a bunch of like slowly taking up a bunch of memory in Lighthouse. Um and you know finding that out like took some time. Um and we realized we were kind of using spans in the wrong way. Um, but once we've like started using those spans in the right way and using them in like the right places, um, it's really given us just like a deeper view of like what's going on in Lighthouse and really helped us like debug um, certain issues and yeah, it's I think it's been a really positive addition to the codebase.

Awesome. Yeah, thank you so much. Um, and the question has one more part. Um, was relying on optin opt out mechanisms enough to balance log detail and performance or did you use additional strategies? Did you perform any benchmarks comparing the old logging system with the new tracing implementation?

Um, yeah. So, like really what our our goal was with tracing initially uh was to just kind of like mimic how like our old like logging mechanisms looked like. We really didn't want to like um affect like downstream users. a lot of um a lot of users kind of rely on logs to um monitor you know their val their node health in a lot of ways and so we didn't want to like drastically change things. Uh we really wanted to like keep things as as similar as we could.

Um and it it was yeah it was kind of a challenge because the our previous logging library and tracing just like on a fundamental level work pretty differently. Um, but I think we got we got pretty close. Um, and so yeah, that was one of like the main goals of outside of just like adding more granular details to the codebase was like to kind of not change too many things. Uh, which was a challenge. Um, was there another part to that question?

Sorry.

Uh, yeah, that's all right. Like how did you whether you somehow benchmarked it and compared it to the old system? Uh, was the was the

Yeah. Yeah. So we did we did benchmark and initially we didn't really see any issues like I don't know tracing and logging seemed to be working just fine from like uh all the metrics we were looking at but then as we were running the node for like an extended period of time we saw that like slowly over time uh we were having like memory issues um and at first we didn't think it was tracing we thought it was something else um so we had to like you know kind of uh debug that whole situation and realize that okay, we had these like long live spans that were slowly eating up more and more memory. Um but that yeah that probably took about like a month of just running tracing on some node and um yeah finally realizing like oh crap like this is taking up a bunch of memory. What did we do?

Um and then figuring out that it was we were just using spans in the wrong way.

Awesome. That's interesting. Thank you so much. Um yeah, and we get more questions from the audience. Uh Shane, go ahead.

Yeah, just related to the tracing. Um I know you guys have the Lighthouse metrics repo where you can essentially hook up to some um local node running on the DevNet and it'll expose endpoint that Prometheus can ingest and then uh capture those spans and then you could check out any kind of graphics on graphana uh using that data and I think you have tempo as well now to dive a little bit deeper into uh the time the spans. I'm just curious if you use these pretty regularly. Like currently with the DevNet 3 issues, I can see like people folks like Mark uh like using it to try to figure out issues with like data columns and uh requesting too many data columns from the same block for example as a current issue. I just jumped on call with him to check it out and I'm just curious if this is something that you know you use relatively regularly.

Is it kind of more in this scenario where oh we have a devet we're trying to get up or do you actually use it for like normal development too? Yeah, I would say so. Jimmy added um a lot of the like the tempo um stuff maybe like a month ago and I think ever since he's added it, I've I've been probably using it like almost every day. Um

so it's been it's been really nice. Um yeah, like Mark recently added like some spans around like um validator duty requests um because

yeah, we had an issue where um nodes that weren't synced were being spammed by the validators to calculate like validator duties. Um

and this was actually slowing down the node pretty badly um because it was trying to calculate duties for like states that it didn't have. Um, and yeah, so we like adding those spans like kind of let us figure out like, oh, this is exactly the the endpoint that the validator is requesting and requesting it um, in situations where it's not synced and it's like slowing down things pretty dramatically. Um, and yeah, I I think I've probably been using ever since like we've added tempo and all that stuff, I've been using it almost every day.

Sweet. I was curious how useful it was to learn because I've also like span out lighthouse metrics and got the uh like panda ops devet um hookup between like your nether mine and like how's client going. Uh so anyway, yeah, it's just good to learn and it's good to know also that it um has lasting utility outside just like debugging these deet issues. Thanks.

Yeah, and uh I think the panda's also added support for it in crytosis and they've also added support for it in like DevNet 3 in general uh recently. Um so yeah, it's been really nice for sure. Sweet.

Sweet. Awesome. Yeah, thank you so much. Um, guys, feel free to raise your hand, ask some more questions. We have still plenty of time, 20 minutes to go.

So, feel free to ask. Lee,

yeah, sensory look looks like looks like we still have more time. So I I'd like to continue the previous question about the uh fork issue you have like that long fork during the development of the pears. I want to know more.

Yeah. So

yeah on Devonet 3 we were in like non-finity and uh had like a lot of different chains on the network like we had the canonical chains plus a ton of different side chains that were invalid. Um and the way we had previously done r uh range and backfill sync was um we would find peers that can serve us a range of blocks and columns and request those blocks and columns at the same time uh using um blocks by range and data columns by range RPC requests. um with those RPC requests pretty much just allow you to like list like SL from slot you know X to Y I want this the columns for this indices or I want these blocks and the way we requested blocks and columns was um very like um intertwined I guess um and if any part of that request failed we would throw the whole batch away and re-request it and the issue was with all these like forky chains. Um if like you know we requested from peers that were on the invalid chain which there was a lot of we would try to get the blocks we would try to get the columns we do all the like KZG verification blah blah blah and then figure out okay this is actually maybe like after we request a some number of batches we figure out okay this is not the canonical chain we have to like throw all this work away and um start again and so syncing was like very very slow it actually wasn't progressing in many situations if you got lucky and had good peers um you could sync, but it was very like hit and miss. And so we've changed things to where now we're requesting blocks by range first.

So we request like some set of blocks. And then now that we have those block routes, we can um for each uh slot request the data columns for that slot um using uh columns by root request. So we've like separated uh blocks and column requests. So we always do blocks first and then we fill after we like verify that those blocks are on the right chain and on the canonical chain blah blah blah then um then we request the columns afterwards uh with like some different set of peers uh and we make like a by root request. So for I don't know some block in slot 20 with some block route we can use that specific block route to make a data column request um and it becomes um much more sync becomes much more predictable.

It becomes much easier to also like attribute fault to like specific peers. So like when peers serve you like bad data you can like downscore them and potentially like disconnect if they're you know misbehaving or whatever. And with uh data columns by root request, it's much easier to attribute fault to specific peers. So we know like this peer for this route served us a bad column. Uh we're going to ban them or whatever.

Um and so yeah, with those changes, sync just became much more predictable. Um easier to attribute fault to bad peers and like get rid of bad peers. Um and uh helped us like sync. Um now this this change is still like a work in progress. We haven't merged it to unstable yet, but we've done a lot of testing and that's what we've seen so far and I think there's probably still some small tweaks.

Um, but yeah, um, that's kind of like the high level overview. Uh so to my understanding what you have done is like uh uh try to get the blocks at the first phase and uh to see which blocks is going to be added to the chain and then to query the blocks uh attached to these blocks. Um

yeah yeah yeah so it's so like query some range of blocks um verify those blocks and then after we verify uh fetch like individual you know columns by those block routts instead of doing a a column by range request which just like oh from slot I don't know x to y give us these columns we request columns by specific block routts um so it just becomes much more um

yeah becomes uh it just makes stink much more stable and more predictable.

Uh yeah, that's actually follows the idea of the code does design. Uh but my question is if we like query the blobs later, uh would that actually makes the delay of the confirmation longer or anything? Uh this is for for syncing. So this is when the node is not synced. Um so this is when we're just trying to sync a beacon node.

Um does it de um so like the delays between fetching blocks and columns isn't like super uh important. Um we do need to make sure that that data is available eventually. Um, and so like those delays are much more important when your node is synced and starting to perform validator duties. Like you can't attest to a block if you're not able to um see the the columns for the for that block, for example. Um, and in the, you know, in like a full node case, let's say your custody requirement is you have to custody four columns.

Um, you won't you're not supposed to attest to a block until you've seen those four columns. in a super node case. So those are nodes that have to custody all the columns. Um you're not supposed to attest to a block until you've seen all the columns or or seen at least half of them and then reconstructed the rest of the columns. Um so yeah, the those delays are much more important when your node is synced and performing validator duties versus when you're just syncing because when you're syncing, you're not performing validator duties.

You're just trying to catch up to the chain. Um, so I'm sure there's a bunch of other optimizations to sync um to maybe speed things up or to make things a little simpler. Um, but yeah, hopefully that answered your question.

Yeah. Yeah, that's very clear. That's very nice answer. I mean, yeah, that's true. the I feel the bone neck would be the validator or the super node part if like uh they need to both synchronize the block and the blobs and all the stuff.

So yeah, but that's not a whole network issue. Yeah, thanks a lot.

Cool.

Uh yeah, Shane. Hey, you mentioned you work on the beacon API a bit and um I want to try to standardize um what we'll need to change for in EPBS like you mentioned you worked on produce block a little bit and uh essentially when we uh get the response back we're going to have to remove things like the execution payload probably don't need blobs anymore td commitments uh so it's all just in the return type you know the prams that will pass will still be the same like random whatnot uh do you think that that justifies a like v4 update or do you think we can keep as v3 v3 three just change like the version field uh to you know be glow ass and uh all just the the spot objects.

Yeah, that's that's a good question. Um and I would say like the the people maintaining the beacon API repo are the best people to to ask that question to. But I would at least from

does take you maintain it or is it like somebody else?

Uh no I think it's someone from the EF. Um,

okay.

I I I can double check. I don't I don't remember off the top of my head, but but I would say like all the anytime there's like um a change to like return types for an endpoint, we usually move to a new version.

Um in the EPBS case, I'm not 100% sure.

Um just because I'm not like super familiar with EPBS to be honest. Um,

yeah,

but it sounds like if like return types are changing that usually and historically we've moved to like a new version. Um,

but yeah, you can um if you open up like a PR with those changes in the repo, you'll get pretty quick feedback or you can ask Discord. Um,

and if you if you just look at the repo and look at like the the biggest contributors, those are or I think they probably have like a maintainer tag. Those are the people that maintain into the beacon API repo and would be the best to to ask that question.

Yeah, just quickly it's basically like the valid client is requesting the beacon node to you know come back with a new eco block. So the beacon block body is not going to have the execution payload anymore under EPBS. It's going to be separated out and we'll grab that payload later on in the life cycle of block production. So anyway, yeah, that's why the return type's changing. You're not going to have a payload anymore.

So yeah, I just wanted to summarize that. Yeah, thank you for uh explaining how to move forward.

Yeah, off the off just like from what I'm hearing, it sounds like a new version makes sense, but um maybe those the beacon API maintainers have other ideas about that.

For sure. For sure.

Um can you hear me now? I'm back.

Yeah. Uh so sorry I had to switch to data and uh yeah I want to ask you as the last question from the GitHub there's one more let's um let's do that one then um after integrating tracing into Lighthouse did you ever use tool like flame graph or Tokyo console to analyze performance or async behavior if so did you find them helpful or was the benefit limited I've never I've never used flame graphs, but in graphana there's like a way to like look up tra like specific spans and then it gives you like a kind of like this full picture of like the um function call tree or whatever. Um and you could see how long each call took. Um and so I use that a lot. Um yeah, I never use flame graphs.

Uh maybe that would be cool to use. I'm not sure. Uh but yeah, most of the stuff tracing wise I just look in Grafana at the moment.

Uh yeah, Shane. Yeah, just for um everyone here, we're all very interested in core development and you uh kind of went through the process, ended up getting on core team. And uh one part I'm curious and I'm sure it's different for like every client team is what was the process like going from like doing EPF, maybe presenting uh Dev Connect and or DevCon, whatever. Um and then getting like onto client team, was it like hey, you know, you've been doing all these awesome contributions. you want to maybe apply for a job or did you like signal to them in some way or like yo it would actually be pretty dope if we did this long term.

Uh yeah, just curious like how was how was for you personally? I understand it'll be different for everyone.

Yeah. Yeah. Um so like at the end of EPF when we did uh like presentations we were in I don't know if it was DevCon or Dev Connect one of those in Istanbul.

Um and that was the first time I met like some of the lighthouse people face to face. So, I met Michael Sproul. I met um Age. Um and yeah, just you know, talking getting to know them um at like one of the I think we had like this like little event where we had like the uh where the EPF people were invited to hang out with the the client teams um at like a restaurant bar kind of thing. And then um so yeah, I talked to them a bit there and then during the presentation day they showed up um and listened in on all the presentations um and I kind of presented my work.

Um and um I think age age came up to me and like was like wow I didn't realize you did all of that. That's that's super cool. Um and I had talked to they asked me like you know are would you be interested in working um as a client of I think Mario and like our my exit interview also asked me if like you know what's my next steps and I told them I would love to work for you know a client team um and kind of one of the things that I did um after after all that you know they said like oh that'd be cool maybe we could talk later uh blah blah blah but I didn't hear from them for like maybe a month or two um but during that time I still just kept contributing to Lighthouse. Um I just yeah I thought like if I just keep opening PRs, you know, they just won't be able to say no eventually. Um and then I age reached out to me um and asked if I wanted to interview.

Um and I did and I was super stoked and nervous. Um but uh yeah, the interview went well. Um I think one thing that made the interview uh a little bit easier was because I already had like a body of work. they already knew kind of like my skills as like a software engineer and um so that made like the interview process like pretty easy actually. Um but but yeah that was that was kind of it.

So it was like a few months after the end of EPF where they you know finally reached out and asked to schedule interview and then from there you know after the interview went well and blah blah blah I accepted a position and yeah went on from there.

Do you have any takeaway? keep building.

Yeah. Yeah. Yeah. I think I mean obviously people have like different priorities and schedules in life. So like um I'm not saying like you know give you know full-time like contribute all the time but like if you just chip away at it um and if it's something that you want to do I think it's kind of hard sometimes when you're not getting you know actually paid to to do this type of work.

Um it's kind of tough. Um, but you know, if that's something that you want to do, I think uh as long as you keep contributing, there's there's positions for for almost anyone in in Ethereum in general. Um, so doesn't have to be like, you know, client dev. It could be with the like pandas doing DevOps or maybe with like the EF like helping maintain the beacon API specs or whatever. There's so many like different opportunities and positions.

Um, and if if you really want it, I think as long as you keep chipping away at it, um, there'll be a spot there for you somewhere. Um, hopefully. Yeah.

Awesome. Yeah, great advice. Keep at it, basically. Nice.

Awesome. Yeah, glad to see the conversation still going. Uh, we have last few minutes to go. Um, yeah, so sorry again about my internet, uh, coming off. Um, but I'm fully back now.

Um, we have last few minutes. Uh, do we have maybe one more question somebody would like to to ask? Okay. Uh, oh, we got two of them. Okay.

I think I uh pillows first. Okay.

Sure. Thanks. Uh, quick question. I was just interested in what a interview for core dev position looks like. Uh, is it more like just quizzing you on your knowledge of core development stuff or I I'd be curious to hear.

But yeah. Yeah, I I would say every client team is drastically different. So, my experience is not going to be the experience of other client teams. Um, or even depending who interviews you at, you know, Lighthouse. Uh, maybe I'm not I'm not 100% sure.

So, I know like some clients do like some like technical interview or ask a lot of like technical questions. I think for my situation specifically with Lighthouse because I had I think I had like over 15 PRs at the time that were merged to unstable, they had like a very good idea of like my skill level as a as a software developer. So, it was um really just a conversation. Um but again, like I think it really depends on like your specific situation and what client team you're interviewing and who's interviewing you. Um, so there's no like structured like everyone does it this way um every time.

Um, that was just my experience and I think yeah, everyone's experience. I know um some other EPF fellows from my cohort that interviewed had a pretty like uh technical interview um with um I forget I forget what client team. Um so yeah, I think it just really depends.

Yeah. Awesome. Oh, I mean and that's that's really true like that's the point of EPF. You know, the client teams are mostly familiar with your technical skills and then it's much easier to have a conversation. Okay.

Uh Iran has one last question.

Yes, the last one. So as an ex EPF like fellow and now switching as a as a core dev, what do you think is your most underrated tools or like strategy that you use now switching from an XCPF to a core app?

My most underrated tools or strategies

strategy. Yes. Um, yeah, I think I think um starting starting small is is really important. Like and a lot of these like client code bases, they're very like complicated and technical. Um, and some of the the issues that you want want to work on sometimes could be like pretty tough for someone that's not like familiar with the codebase.

So, I think starting small is really important. It the smallest like good first issue you could find is, you know, my recommendation. Um, and then like building up from there, you know, slowly. It it takes it takes a long time. Like if you think about it, when you start like at a at a new company, um, I mean, the onboarding process takes at least two or three months and you're working like full-time.

And in that time, like man, if you could get like a couple a feature or two out the door, I think that's like awesome progress. Um, and so I always I always try to look at it in that way. Like, okay, I'm trying to onboard to this new codebase. Like imagine I was an employee here. Like what would I do?

you know, I would start small. And obviously, you know, you're not working full-time, so it might take even longer.

Um, that's one of my strategies. Um, also, um, like not, yeah, don't be too hard on yourself, you know, kind of that whole thing like there's there's a lot of things you're not going to know and a lot of things you're going to get stuck on. Like I remember when I first was learning Rust, something that would take me like 30 minutes in Python would take took me like six hours in Rust. Um, and so there's always like maybe the learning curve of like the language you're trying to write in, um, the codebase you're trying to write in. And so, um, yeah, just it's it's a grind and you're just like slowly chipping away at it.

And if you like start small and just like build those building blocks, eventually you can start working on bigger and bigger things and, you know, kind of build that knowledge base. Um, so yeah,

amazing. Thank you.

Yeah, thank you so much. I think that's a that's a great advice to actually wrap it up on. We are right right at a full hour here. Um it was it was really great. Uh thank you so much Ethan.

Yeah. Um yeah very skilled developer, very experienced. So thank you so much for all the knowledge, all the inspiration. I believe that uh uh the fellows learned a lot today. Um awesome.

Yeah.

Yeah. Thanks guys. It was happy to be here. And uh I'm on Discord. from uh uh Uncle Bills on Discord.

Um so you can reach out to me if you have questions or on GitHub, whatever, Telegram. Um yeah,

awesome. Yeah, thank you so much. Really appreciate your time. Uh it looks really great. So um thank you so much for joining today, uh all of you, especially Ethan.

Yeah, awesome.

Thanks.

And yeah, have a great rest of the day everyone and I'll see the follows on Monday standup again as always. Okay. Bye bye everyone.

Bye guys.

All right.