stokes:Okay, thank you. Hi, everyone. Welcome to ACDC 165. This is issue 1716 in the PM repo.
Transcript
stokes:And the main topic for today will be Fusaka, with a few other minor notes for Amsterdam, but yeah, let's go ahead and get into it.
stokes:So, to kick us off.
stokes:It'd be good to touch on Fusaka, DevNet 3, and DevNet 5, and yeah, kind of where we're at there.
stokes:I wonder if, Perry or Barnard,
stokes:Barnabas or Perry have any, updates there, or anything they'd like to call out?
Barnabas:Sure. Let's start with DevNet 3. DevNet3 has been,
Barnabas:mostly very stable during this past week. We are over 96% participation now. Teams are still pushing out fixes into their trunk branches, and they get automatically deployed onto this network.
Barnabas:We haven't been doing any kind of…
Barnabas:attacks or any splits in this past week, because we have been mainly focusing on Cosaca.net 5, in this, past week.
Barnabas:7 at 5?
Barnabas:Started out with,
Barnabas:1,500 nodes, and we did a fulloo last week's Thursday, then we did a BPO a day after the birds.
Barnabas:the… Went up all the way to 72 max blobs, and we did 48 on the day before.
Barnabas:Unfortunately, we had a…
Barnabas:bug in the configuration, which led to the full node getting a super node flag, which basically meant that our full nodes were
Barnabas:At 100% bandwidth utilization pretty much all the time.
Barnabas:Which led to quite some issues.
Barnabas:Yesterday, we fixed this by wiping all the full nodes and re-initializing their databases and their… and this also reset their, CGC value.
Barnabas:And we ran an analysis with, 15 max blobs, 21 max blobs, and then 33 max blobs, one more time. But, because of the time window, it's so short between, that time and ACD,
Barnabas:The numbers are not really, representative of, Of many, many slots.
Barnabas:Full nodes are representing 90% of the number of nodes, but only 10% of the validators on DevNet 5.
Barnabas:Nevertheless, some from our team did an analysis. Pari just linked it in the chat. Hopefully, most of you had the time to take a look before the call.
Barnabas:And the conclusion of our, analysis was that, max values of 15 and 21,
Barnabas:seem to be reasonable for BPO1 and 2.
Barnabas:And next value of 32, might lead to struggles into publishing blobs without any V-release or full nodes, with limited bandwidth.
Barnabas:We certainly will have to do another analysis. It's probably going to be physical.net 6, maybe a couple of weeks from now, but, we feel like we should be good to go with Holosky and with these, BPO numbers.
stokes:Okay. Yeah, thanks for the update. Yeah, this analysis, looks pretty good. I'm just looking at it now, but…
stokes:everyone definitely take a look there, and… yeah, at least I think that will then confirm, these first two EPOs, which I think was the main thing we wanted.
stokes:So… yeah.
stokes:Any other known bugs, with any of the clients, or any issues we should touch on with the DevNets?
Barnabas:Yes, Prism has a known bug that they are investigating right now, that leads to a bunch of, often, blocks.
Barnabas:We are rolling out a fix, or a potential fix right now. I'm not sure if they have anything more to share.
Manu:Yeah, I can share something. So yes, we have a bug in prison with a lot of orphan blocks. It seems to be only the case with high blood count and locally
Manu:build block, so there is no issue if it is… if there is MEV, or no issue with low block count.
Manu:We are still in the process of narrowing down the root cause. What just we know right now is
Manu:When we produce the blog, the blog is produced in very well fashion, like a few hundred milliseconds after the start of the slot. And what we see at the end is when we broadcast, the columns, the tech columns are broadcast very late, into the slot, like 6 or 7 seconds.
Manu:So there is something fishy between… after the end of the blog bidding and before the start of the data collection broadcasting.
Manu:And yes, we are in the process of finding the root cause in this process.
Manu:That's all I have to say.
stokes:It seems like this is a regression. Does that track with your understanding? Or do you think something… do you think something new was introduced that caused this, or is this a regression from what we had in the past?
Manu:Actually, possibly, it's not a regression possibility, it's an issue which is here for a long time. However, if I'm not wrong, we…
Manu:the best… We mainly tested,
Manu:high blood count with MEV, which does not cause any issue, and I think we really started to see the issue when we… when, you know, MEV was deactivated on the…
Manu:On the… on the DevNet 5, with high blood… high blood count. So, possibly, it's not so… it's maybe something totally unrelated to pureness.
Manu:A bug which was here from, you know, a long time, but which was not visible with a low, low blood count.
Manu:Yep.
stokes:Yeah, frozen period?
Raúl Kripalani:Yeah, so I think this could be, just kind of like a potential lead here, could be the fan-out mechanism. So basically, when you're a builder, you are forced to publish to 128 subnets, simultaneously, which means that you need to have peers in all those subnets that are peering… that you're peering with, that are part of your mesh.
Raúl Kripalani:And this usually requires a little bit of preparation ahead of time to make sure that, you know, given a particular peer count, active peer count, you have sufficient density of peers in each of those subnets.
Raúl Kripalani:I am aware, I think we've been looking into the Prism codebase as well, and the peer-to-peer networking team, and we did find out some events in which
Raúl Kripalani:Prism can, like, spend 5 to 6 seconds or something like that, discovering peers when it doesn't have sufficient peers, and if this is part of the critical path, that could justify the delay that you're seeing here.
Raúl Kripalani:And this goes, like, in general, as an advice, I think, to all CL clients, is ensuring that, you're correctly
Raúl Kripalani:Predicting when you're gonna be up for builder duty, for proposal duty, and if you're gonna… if you're a local builder, then ensuring ahead of time that you have sufficient peers in your meshes, so that when the time comes, you don't spend any time actually conducting the discovery during the critical path.
Manu:Yeah. So, so for information in a presumably, as soon as
Manu:We… we have one active validator. We ensure to have, at least 6 peers, in our fanhood for our data collection subnets.
Manu:So we don't compute, nor are we going to…
Manu:Are we going to propose a block in one way? Just, yes, if we know we have validators, we always make sure that we have peers in all the technical subnets.
Manu:Paramount.
Manu:Yeah.
Manu:So normally, I mean, we don't have to search for a new peer when proposing a block.
Manu:And if we do, we have luck for that, and it doesn't seem to be the case. But, yeah, I will double-checked, for sure.
Parithosh Jayanthi:Yeah, one of the things I wanted to mention was, in the past, we would either have MEV completely running throughout the lifespan of the network, or we would have it turned off. We didn't really have a fine-grained mechanism to change this, mid-BPO, for example.
Parithosh Jayanthi:But now we've added a scripting approach, so if a BPO lasts for one day, then 12 hours of the day is gonna be with MEV and 12 hours without MEV. So just in case any client dev is looking at a DevNet and wondering why there are no MEV proposals, it's probably because we've disabled it, so that we can test both local block building as well as MEV workflows for all blob numbers.
Manu:And, yeah, and additionally, and from additional information, currently, I know that I checked one hour ago, DevNet3 had LED on.
Manu:And DevNetri has no issue about offhand blocks for poison.
Manu:And, yeah, the next 5 had MED off, and there was a lot of orphan issue.
Barnabas:DevNet3 has a significantly more share of super nodes, though, and also much less blobs. They have a next blob count of 20 at most, and I think 50% of the network is super nodes.
Barnabas:I'm… on DevNet5, it's also possible that some of your peers will be only peering with other non-super nodes.
Barnabas:So, peering will be significantly more difficult to… to find enough peers on…
Barnabas:enough columns. That has all the columns that you need, basically.
Manu:Okay, I will double-check that.
Manu:Should not be the issue, but yes, worth to check again.
Barnabas:We might need to increase the default peer count on some of the CLs, but I think Lighthouse also increased from 100 to 200.
Manu:Yeah.
stokes:Okay.
stokes:Anything else on the DevNets?
stokes:Okay.
stokes:If not, then, next on the agenda, so there was an issue that came up with CKZG. Justin had a note here in the chat, I don't know if you want to add anything to that, Justin, otherwise I can,
stokes:Give a summary.
Justin Traglia:Nothing really to add, just that…
Justin Traglia:there was a misunderstanding of how, Blast used points at infinity, or… yeah. Pretty much we fixed it with a new release, shouldn't be an issue anymore.
stokes:New release of CKTG.
Justin Traglia:That's correct. Version 2.1.3.
stokes:Okay, gotcha.
stokes:And… The understanding is that everyone should be using 2 and 3.
stokes:Or at least not using 2-12.
Justin Traglia:Correct.
stokes:Okay.
stokes:I don't remember if there's a way to link this in the specs, but there either should be, or… yeah, I think we should make this clear somehow.
stokes:How have people handled this in the past?
stokes:What was that, Felix?
Felix (Geth):Sorry, I don't think it's necessary to, like, link specific library versions in the spec.
stokes:Fair. So then we should just remove 212 from the repo?
Justin Traglia:You mean, like… or what do you mean, exactly?
stokes:Well, it sounds like 2 and 2 is just broken, right?
Justin Traglia:Yeah, sure. You wanted me to, like… Okay.
Justin Traglia:Remove it as a release from TKZG?
stokes:Yeah, I mean, I want to prevent what happened, where I think some clients upgraded to it, because there was a new release, and then it caused some issues.
Justin Traglia:Yeah, I mean, we can discuss it. Personally, I don't think that's necessary, but .
stokes:Right.
stokes:I'll follow up with you later, then.
stokes:Then… I guess that's everything there. Just be aware of that.
stokes:Okay.
stokes:Let's see here, let me look at the agenda. Okay, so, yeah, if that's everything we need to talk about in the DevNuts,
stokes:The next thing I wanted to bring up was the testnet schedule.
stokes:So, I have a proposal here.
stokes:And… let me just run through it, and then we can discuss.
stokes:So… The proposal would be next Monday, which is the 22nd of September, effectively code freeze.
stokes:This could be, like, a branch or some commit on, say, you know, a trunk branch, but ultimately, you know, this would be the code that would go into releases for testnets.
stokes:If we have code freeze then, I believe the PandaOps team is willing to do some validation on this code.
stokes:Just to make sure things, you know.
stokes:look okay, and yeah, maybe once I'm done talking, they'll have something to add here. On that point.
stokes:But essentially, aim for Code Freeze next Monday.
stokes:Then if we aim for testnet releases for all 3 testnets on the 25th, which I believe is next Thursday… let me just double check that…
stokes:It is…
stokes:Then from there, we would aim, following sort of the process stuck that we had, at something like 2 weeks,
stokes:For every testnet from there.
stokes:So then, Heleshky would be on 2nd of October, Sepoli on 16th of October, Hudi on 30th of October, and there are fork epochs that align with all the different parameters we need, here on the repo.
stokes:And… I guess one thing to note is the Sepolia fork that would work on the 16th of October is during ACD, so I'm not sure if we want to move that.
stokes:Even assuming we can, you know, agree to the schedule here.
stokes:But that was something to flag.
stokes:And, yeah, one other note, we would use the BPO schedule from .NEF5. Again, it sounds like from the analysis we had, we can at least validate these first two BPOs with the numbers that were given.
stokes:From there, I think everyone knows the deal. You know, if there's a test network and it doesn't go well.
stokes:This is basically, you know, everything after that's on pause. If everything does go well, then even when we get to the last testnet with Hoodie, mainnet would be, you know, 30 days after that, so…
stokes:That's a proposal, it keeps things moving, and yeah, also generally follows this process document that we've been discussing over the last few ACDs.
stokes:So… I see… yeah, Christine here put this in the chat, thank you for that.
stokes:And, okay, yeah, Barnabas, do you want to discuss your adjustments?
Barnabas:Right, so my adjustments would be that we don't do a fork on a non-Wednesday, basically. Okay. Wednesdays are probably the best, most suitable days for a fork, because it allows us to debug before and afterwards as well.
Barnabas:And it doesn't clash with ACD as well. My proposal would be we would do Holsky.
Barnabas:Fusaka on the 1st of October.
Barnabas:Then, a week after that, we do the first BPO, then a week after that, we do the second BPO. Then, a day after that, we do Sepoya Fusaka.
Barnabas:And, we basically just keep repeating this pattern, throughout all the different testnets. So we wait one week for each BPOs, and, we're gonna ship all the BPOs before we proceed for the next testnet,
Barnabas:book.
Barnabas:So this allows us to run analysis on the different testnets, just to make sure that everything is still looking good before we, pull the trigger.
stokes:Yeah.
stokes:Okay, I mean, yeah, I'll validate the numbers you have here, but… That seems reasonable.
stokes:I think, most importantly, I would like to hear from the clients what they think.
stokes:I reached out to a number of different, I think I reached out to every client team over the last couple days, and I think generally they're okay with this. I think the main thing to figure out for now is just really next week, in terms of code freeze and releases.
stokes:But yeah, I'd love to hear, if anyone
stokes:Has an issue with this, or thinks we should make adjustments somewhere.
stokes:So, I've gotten thumbs up from all the client teams before, so if no one says anything, I will just assume it's still thumbs up.
stokes:Okay, now there's a lot of thumbs up on the Zoom screen.
stokes:So… Okay, that's very exciting.
stokes:Does anyone…
stokes:I guess one question here is… yeah, like, do we just go with Barnab as a schedule?
stokes:I'd have to look and see… But it seems good.
Barnabas:All the epochs are on the boundaries.
Barnabas:So…
stokes:Okay, yeah, that's mine.
Barnabas:The official room fight, also, I made sure that the BPOs are on the boundaries as well.
stokes:Okay.
stokes:Yeah, I guess what we could do, because this might take a little double-checking async, is just also have this finalized by…
stokes:Monday, along with Code Freeze.
stokes:And… I would say lean towards go with this, unless someone wants to say otherwise.
stokes:But yeah, I just…
Parithosh Jayanthi:The next step needs to be a PR to the ETH clients repository, updating and setting the forks, and then we can make sure that all the clients have, looked at that before we merge it in.
stokes:Yeah.
stokes:Okay, yeah.
stokes:Oh, Barnabas says you can do that. Great.
Barnabas:So, cool, yeah. One more thing that,
Barnabas:I suggested that we're gonna have different release deadlines, instead of trying to get everyone to include all the timelines in the release. I'm not sure if they want to.
Barnabas:But, like, we could have a…
Barnabas:you could basically have all three raised deadline by the 24th, then Sepolia raised deadline a week after that, and two weeks after that, the Holy One.
Barnabas:I'm not sure if it's really needed.
Barnabas:Or whether we just want to pre-schedule everything in one release.
stokes:Yeah, I think it's simpler for everyone if there's just one release, but… Yeah, I guess so.
stokes:I'd love to hear from clients how they feel.
stokes:Is that helpful for anyone?
Ameziane Hamlat:So, for Basic side, it would be helpful, because we are working on some performance improvements that we would like to include for Fuseka.
Ameziane Hamlat:So we may need to have a different release between the one for Holesky and…
Ameziane Hamlat:Let's say the one for a hotel, for example.
Łukasz Rozmej:So, I think, like, we can definitely do one for all, but we might,
Łukasz Rozmej:Also, like, do some service releases after that,
Łukasz Rozmej:But I think that's fine, and that should also be fine for Bess, so I don't really see this being a conflict.
Łukasz Rozmej:I think if we can agree that it's probably fine to have a release that supports… as a baseline that supports all testnets, but then have subsequent releases after that. That shouldn't be a problem.
stokes:Okay, so, well, everyone will aim to have one release, but then, yeah, if things update, then that's all perfectly fine.
stokes:Okay.
stokes:Cool. Does everyone feel good about this?
Matthew Keil:Yeah, I can second that sentiment, that, we're ready for a release, and, but the flexibility in case something comes up during test sets.
Matthew Keil:To be able to put one more out before mainnet.
stokes:Yeah, I mean, you can always, I think, Cut a release.
stokes:But just for the test nights, given there's so many, sometimes it's simpler just to aim for one, but…
stokes:Yeah, definitely, we should be able to accommodate, At least 3.
Dustin:I mean…
Dustin:Practically, this means that there's… unless there's a… things go catastrophically wrong, there will not be a schedule delay. That's not necessarily good or not necessarily bad, but… but once those releases are out there, and now…
Dustin:Essentially, the dynamic that happens is it becomes partly as risky to delay the release, even if you think it might go wrong, than it is to… the fork schedule, than it is to keep it going and just make it a…
Dustin:an emergency fix, if all of the releases are scheduled at the same time, and this has happened before. So, I mean, as long as people go into that knowingly, that's…
Dustin:Fine. I don't know. It's a choice.
Dustin:But…
Dustin:And this is… but this is one reason why I know the… in other discussions, this has been…
Dustin:Yeah, a trade-off discussed.
Marius van der Wijden:Yeah, I would say it's fine. I think,
Marius van der Wijden:Because these testnets are only run by a very small
Marius van der Wijden:group of people, we can always ask them to just use the newest release. Yeah, and the important thing is that we don't bake in the mainnet release into the tested release, so that the mainnet release will be
Marius van der Wijden:must be a different release. And I think that's… that's it.
stokes:Yeah, sounds good to me.
stokes:Okay.
stokes:Again, I know this has been a huge lift to get here, so, I guess congrats, everyone.
stokes:And let's hope the test notes go well.
stokes:So… Cool, we'll get this PR up for the schedule…
Barnabas:Should I make the schedule then based on the timelines that I've suggested?
stokes:Yeah, I think so. I'll double check, just to make sure, you know, things are aligned to these boundaries and stuff, but yeah, let's go ahead and go with what you had.
stokes:It looks good.
stokes:Cool, okay. Anything else on Fusaka? I think this is pretty much…
stokes:what we have today, on Fusaka, just because, yeah, we're in the process.
stokes:Anything else?
Parithosh Jayanthi:The audit went live, do we wanna maybe mention that?
stokes:That would be good. Frederick, I don't know if you have something to say there, otherwise…
stokes:I can go try to find a link to it.
stokes:I don't have one handy.
stokes:Here, I'll just… oh yeah, thanks, Justin. Okay, so yeah, I was looking at this blog post here. So…
stokes:Yeah, you know, every upgrade, we usually do one of these security competitions, we have one live for Fusaka.
stokes:There's a $2 million bounty here.
stokes:And, yeah, check out the post, especially if you are in the security space and listening to this recording, or watching it live.
stokes:This is super useful. We've found things in the past that, you know, are direct impacts to mainnet security, so…
stokes:This is a super good program, just to find any lost bugs lingering.
stokes:And… yeah, I think that's all I have to say. I don't know if anyone wants to add anything else.
stokes:Mario says it projects 20,000 submissions by the end of the contest.
stokes:I don't know where that number's coming from.
Marius van der Wijden:Yeah, that… I don't know that…
stokes:odds.
Marius van der Wijden:Yeah, the algorithm is kind of weird, but there have been some submissions, and the security team is looking at them, but we also want to encourage all the client teams to look at them as often as possible.
Marius van der Wijden:So, yeah, these things don't go stale, and we don't receive, like, duplicate submissions because the code hasn't been updated.
stokes:Okay.
stokes:Cool.
stokes:Anything else on Fusaka?
Barnabas:One more thing, like, it would be very good to get an update on, backfill from different clients, because I think this was one of the main points that we, still have to ship.
Barnabas:I think Lighthouse now has working backfill, but I'm not sure about any other client teams that have,
Barnabas:finished implementing Backflow.
stokes:Okay, Bota says Prism has it.
stokes:Yeah, there's an implementation they're working on.
Fredrik:I can say something more about the competition quickly. There is, for the client teams, so they're aware,
Fredrik:There's a bit of an issue right now with the…
Fredrik:Judging aspect, but that's gonna be fixed.
Fredrik:Tomorrow, most likely, so that's… Brian teams can provide comments, and
Fredrik:yeah, feedback on the severity levels, etc. And then, yeah, if you need any more people from your team to have access to the competition, to provide this feedback, then you can just reach out, and we will get everyone added.
stokes:Alright, thanks.
stokes:Yeah, Barnabas, back on the backfilt topic, do you just want to follow up async?
Barnabas:Yeah, sure.
stokes:Do you have any other questions right now? Okay.
stokes:Did we want to touch on this earliest available slot thing? Because that also did come up.
Barnabas:I'm happy to bring it up, but I think that can also be discussed.
Barnabas:Oh, awesome.
stokes:Okay.
stokes:Another comment here in the chat from Solus, Grandine also has backfill, merged very soon.
stokes:Cool.
stokes:Then with that… Glamsterdam, and then, yeah, a few announcements from A.B. Kotap.
stokes:Let's make sure I say that correctly.
stokes:So yeah, with Amsterdam, I think the main thing to keep in mind for now is that we said we want to have non-headliner EIPs proposed.
stokes:by Fusaka Midnight Releases, which, with the schedule today, gives you some more clarity on when that would be.
stokes:So that's good. Otherwise, yeah, I don't think we really need to get into it, just because I think we should all stay focused on Fusaka for the time being.
stokes:Yeah, if there's, like, very brief…
stokes:announcements or something anyone would like to provide around breakouts or something, I think that's admissible, but…
stokes:Otherwise, I think that's all we really need to say there.
stokes:Then, on the announcements I mentioned, so… we do have an ETH Magicians post here.
stokes:for the name for H-star, Which, yeah, I forget…
stokes:Okay, there was a poll… here, let me grab the link here.
stokes:There was a poll here, and I guess, yeah, we just need to actually make the decision at some point.
stokes:So yeah, take a look there if you want to chime in.
stokes:And there's another request here for a name for the replacement testnet for Sepolia.
stokes:I will… Share this, because it was added to the agenda.
stokes:I…
stokes:Myself, I'm not convinced we need to replace Apolia, even though I know that was sort of what was written down with, this LTS situation.
stokes:But I think that might be a conversation for a different day.
stokes:So, in any case… Following the schedule, yeah, we would basically sunset Sepolia,
stokes:Soon-ish, and then there would be some replacement, and this post is just asking for a name for the replacement.
Parithosh Jayanthi:I do want to maybe make a point here, we're still helping a lot of people migrate to… from Holsky to Houdi, so we don't want to add to the chaos and immediately have everyone also migrate from Sepolia. So, even though Sepolia is supposed to be end-of-the-year
Parithosh Jayanthi:terminated, I think we will add, like, a decent grace period. We'll probably update wording, and even if there is a replacement test for support here, we'll make sure that both are running at least for a considerable time, so that no one's, overburdened with having to migrate their entire stack.
Parithosh Jayanthi:Multiple times in a year.
stokes:Yeah, that sounds good to me. I mean…
stokes:Yeah, again, maybe a topic for a different day, but I am not sure why we need to actually tear down Sepolia at this moment.
stokes:But yeah, again, different, conversation.
stokes:In any case, yeah, I think what Perry said stands, so there'd be plenty of grace period, just so there's not, yeah, more pressure there.
stokes:Okay, with that, this was everything we had on the agenda today.
stokes:Anything else?
Barnabas:I would like to discuss, Glamsterdam testing, and, possibly splitting up, all DevNet 0 and EPPS DevNet 0, and keeping them
Barnabas:separate as long as we basically can, and we want to. Personal has a 90% done implementation that would support,
Barnabas:the block access stuff, and I think, we can just basically do a prism-only,
Barnabas:DevNet for all the ELs that would be supporting it, and that way we don't have to burden any of the other CR devs with this.
Barnabas:And they can implement it whenever they want.
Barnabas:So, at least for the first couple of iterations of, while DevNets, I think we can just have it with Prism.
Barnabas:And then, on the other side…
Barnabas:So that would not trigger any EPPS testing, and then we could have the EPPS testing without the ball testing, basically.
Barnabas:I just want to make sure that all the CR teams are focused on EPBS testing and writing EPBS stuff, and so we can start testing in parallel as soon as possible.
stokes:Yeah, that part makes sense. I was gonna ask, is this okay with the Prism devs? Like, are…
stokes:Can they produce something that's good for BLs?
Barnabas:It's already done. Like, it's 90% done.
stokes:Okay.
stokes:Okay.
stokes:Okay, and yeah, just to be clear, that would have the EPPS features in it, and they would do BLs on top of that with EPPS?
Barnabas:No, it's without the PPS.
stokes:Okay, interesting.
stokes:Cutters?
potuz:Yeah, so the issue is that, block access lists, without EPBS requires a change in the beacon state, and this is the problem, and why we were petitioned on having different branches.
potuz:With DPBS, block access list doesn't require any changes on the beacon state, because we've removed the header, the execution payload header from the state. So now, in the future forks, whenever we change the payload, we do not need to change the beacon state anymore.
stokes:Okay, Francesca's also asking in the chat here, we might not need CL changes?
stokes:For BL.
Francesco:Yeah, it's just what was just said by Purdus.
stokes:Oh, sorry, okay.
stokes:My bad.
Barnabas:Yeah, but the question is, can we… can we check, ER functionality without… like, testing any EPBS functionality.
stokes:So what would that look like? It would be a Fusaka release with no PPS on top?
Barnabas:Yeah, so the PR that I linked is that.
Barnabas:It's basically implement EIP7928 as gloss fork without concentrating.
stokes:Okay. Does this work for the AL teams, then? Because then, yeah, they can use this for the CL, and then we can move ahead on VL DevNets.
stokes:Podos, did you raise your hand again?
potuz:Yeah, so I was… I wanted to suggest, I mean, it's absolutely fine to use this branch now, but I would suggest that as soon as we have some minimal implementation of EPBS,
potuz:that we can… if we could, switch to doing this on top of EPBS, because maintaining this branch, even though it's mostly generated code, it's going to be a mess. It's… I mean, it's… it changes everywhere. Whenever you change the beacon state, it changes everywhere.
potuz:So I think it would be better if we switched to doing block access lists on top of EPBS so that we maintain a single… and we actually start merging into our main branches these changes.
Parithosh Jayanthi:Yeah, we just need it for now, so that we can unblock stuff. We don't need to maintain this long term. The moment EPBS has minimum viable
Parithosh Jayanthi:client support, we're gonna switch over to that. We just want to ASAP start with block-level access lists, and also just for the other CL teams, you guys don't have to do anything. We just need one CL team to take on the topic, because that will unblock us, and the rest of you can continue work on EPBS, and
Parithosh Jayanthi:Mudge everything later.
stokes:Right.
stokes:So it sounds like we have a plan, then. We can use this PR here, 15712.
stokes:And move ahead with BLs, and then, yeah, I guess another sort of point here is not having this become, like, a persistent branch.
stokes:Plan being to get things merged sooner rather than later.
stokes:And… yeah, okay.
stokes:Are we good there?
stokes:Marvis?
Barnabas:Yeah, I have just one more thing. So, if we can have, client releases, done by maybe next week, Monday or Tuesday or something like that, and we plan to do a shallow fork.
Barnabas:And this shadow fork is going to be for Hudi, one for Sepulia, and one for, Koski.
Barnabas:And then… probably we can also do a mainnet shadow fork, but maybe not next week. So ideally, as soon as the client teams are ready, please let us know, because we would like to have this shadow fork up and running as soon as possible.
stokes:Yeah, there was also a question here from James, about this, so…
stokes:I don't think it matters to you, but would you prefer a release, or, you know, is, like, a commit okay?
stokes:Or is it more up to the team?
Barnabas:I would personally prefer a release, yeah.
stokes:Okay. That might take a few more days, though. And is that okay?
Barnabas:When do client teams plan to have OSCI releases, if we plan to roll it out by the 1st of October?
stokes:So, I think, generally, we're aiming for Thursday next week.
Barnabas:Okay.
Barnabas:Yeah, I mean, we can do… we can do that. If we have, releases by next Thursday, we can do host t-shadow work next week, Friday.
Parithosh Jayanthi:Yeah, so maybe some, context on that. We already have all of our Shadow Fork, playbooks up and running. I'll run them through a mock version tomorrow, so that when the client releases are done, it should take us a matter of, let's say, hours before we'll have the Shadow Fork update.
stokes:Okay, so I guess the point is that…
stokes:Pandops will do some validation here with shadow forks and things, as soon as they have, you know, something from your client team,
stokes:Yeah, I think from there, like, if we can hit Monday, that's great.
stokes:the reason to separate, like, code freeze on Monday from, say, a release on Thursday was just as… was just because some of the clients, I think, have a few days to, like, soak the code before they make a release.
stokes:So… Yeah, but then I guess you're also saying it's fine if you don't get that…
stokes:And implementation until Thursday, right, Barnabas?
Barnabas:Yeah, that's fine.
stokes:Your thumb is still up.
stokes:Okay, cool.
stokes:Everyone clear on that?
stokes:Any questions?
Dustin:I'm wondering on Guido, I mean, has this been tested at scale? I mean, I say as Nimbus was what Nimbus and Techu were a couple of the clients who discovered this, Nimbus was…
Dustin:exposed to this in the DevNets for a while, and we were performing worse because of this. This was fine-ish on the DevNets, that's what they're for. We don't… but typically, we would not want to do a release with…
Dustin:who knows? I mean, especially if it's BLAST and CKCG at the same time. I would say, except for emergency situations, this is precisely the kind of
Dustin:dependency bumps that I would wait till just after a release like this to do, so we had not just two days, but weeks, or at least a couple of weeks, which is clearly not going to happen in this case, for this to, soak, to use the verb people are using here.
Dustin:So, that… there's some wariness on my end from that, I'll say.
stokes:Yeah, because you're concerned with an issue in the newest CK, did you release?
stokes:Or just simply doesn't that time?
Dustin:Become Lindy. Yeah.
Dustin:Yeah, that… well, if it were only the CKZ Geo, it actually wouldn't be as bad, because, if it… that only really affects Fusakra to begin with, then that's a lower stakes thing for now. That's test ad only.
Dustin:But we don't do test and only releases if we don't have to, and we won't have to otherwise here. And so this will also be a mainnet release, not the Fusaka fork mainnet release, but the mainnet release. And BLAST is a absolutely critical dependency.
Dustin:we don't update with P2P, like, 3 days before a release either. That's… Stupid high risk.
Dustin:why… I don't know that we would update Blast, and that's why I asked before, and got the answer from, Justin, saying, well, no, like, they go together. And they don't… Blast doesn't have a release out yet.
Justin Traglia:That's true. This is just hybrid from my end.
Justin Traglia:Would you prefer if we make a new CKG release that adds back the pointed infinity filtering to CKZG, so that you do not have to update BLAST?
Dustin:Yes.
Justin Traglia:Okay.
Justin Traglia:We'll do that. I think that'd make… Everyone felt better, so…
Justin Traglia:Yeah, consider that done. I'll do it today.
Dustin:Okay, great.
stokes:Okay.
stokes:Anything else?
stokes:Okay, if not, then let's go ahead and wrap up today.
stokes:Good luck with MSACA, everyone.
stokes:And I will see you next time.
stokes:Oh, this is a fun question from Perry. Did anyone do anything cool to celebrate March Day?
stokes:I imagine we were all working off Osaka. I did.
stokes:Yeah.
stokes:Okay, I'll see y'all next time. Thank you.
potuz:Bye, guys.
Justin Traglia:Bye, Ron.
Chat Logs
00:00:33
stokes:https://github.com/ethereum/pm/issues/1716
00:01:24
Justin Traglia:Gm
00:02:59
Parithosh Jayanthi:https://ethpandaops.io/posts/fusaka-devnet-5-bpo-analysis/
00:05:58
Barnabas:https://notes.ethereum.org/@ethpandaops/fusaka-syncing-bugs
00:06:04
Barnabas:known current bug list ^
00:06:19
Raúl Kripalani:This could be related to the fanout/discovery mechanism
00:06:22
Justin Traglia:Worth mentioning that there was a bug in ckzg v2.1.2 due to a misunderstanding of how blst handled points-at-infinity in their pippenger msm function. We’ve fixed this with a v2.1.3 release. Existing spec tests didn’t find this because there was a different code path for higher blob counts. We’ve added a new test for this too.
00:06:33
stokes:Replying to "Worth mentioning tha..."
Yeah this is next on the agenda
00:06:47
Dustin:Replying to "Worth mentioning t..."
Can this be used with existing BLST 0.3.15?
00:07:16
Justin Traglia:Replying to "Worth mentioning tha..."
Nope. Needs the latest commit of blst. We’ve been asking them to make a new release.
00:07:21
Dustin:Replying to "Worth mentioning t..."
ok
00:07:55
Ansgar Dietrichs:Replying to "https://ethpandaops...."
amazing analyis, love the formatting
00:09:03
potuz:yeah I doubt fannouts peers is the issue here
00:09:23
Ansgar Dietrichs:Replying to "https://ethpandaops...."
question though, for test 1: I am pretty sure we already knew that local building for low-bw nodes (so 50 Mbit) would have feasibility limits, but that we would be okay with locally set caps on max included blobs.
So that should specifically not be a bottleneck to go to 32 blobs or beyond
00:11:20
Parithosh Jayanthi:Replying to "https://ethpandaops...."
Yup, I think we had a few other issues that made 32 and beyond hard to quantify. I think we’re missing a few optimisations that clients haven’t had time for.
00:11:38
Raúl Kripalani:Replying to "https://ethpandaops.…"
Yeah this what I meant with “not enough topic density amongst your peers”
00:11:50
Raúl Kripalani:Replying to "https://ethpandaops.…"
Generally CLs should be reserving some quota to peer with supernodes
00:12:15
Ansgar Dietrichs:Replying to "https://ethpandaops...."
yes totally fair, and I don’t think we need to get there for BPOs 1 & 2.
just wanted to point out that local building specifically should not be considered a bottleneck for 32 or beyond.
00:12:33
Dustin:hmmm, we'd prefer not to update to a non-release of BLST
00:13:08
felipe:Should v2.1.2 be yanked?
00:13:10
Barnabas:Replying to "hmmm, we'd prefer no..."
I think thats in the works
00:13:15
Felix (Geth):we can all just upgrade
00:13:27
Felix (Geth):to the latest
00:13:35
Felix (Geth):it's a normal procedure
00:13:39
Dustin:Replying to "hmmm, we'd prefer ..."
ok but until then that's a soft-blocker for updating to 2.1.3
00:14:16
Parithosh Jayanthi:Replying to "https://notes.ethere..."
Could all CLs please help us keep this up to date? It would help people conducting audits
00:14:34
J Sunnyside Labs:Replying to "https://ethpandaops...."
Just to add: we ran LH and Teku fullnodes which capped at 50/25Mbps instead. I think they can perform local block building better with higher caps.
00:14:36
Fredrik:< 2.1.2 also does not have the security fixes from the audit: https://github.com/ethereum/c-kzg-4844/blob/main/audits/2025_zksec.pdf , so 2.1.3 would be recommended
00:14:49
Justin Traglia:Replying to "hmmm, we'd prefer no..."
Yeah it’s an annoying situation. We could make another release which just adds back point-at-infinity filtering in ckzg if you all want this.
00:15:05
Marius van der Wijden:Could you put the dates here?
00:21:42 Christine Kim: proposal:
22 Sep: code freeze
PandaOps team can run some validation on frozen code that would serve as release candidate; can discuss best strategy here given devnet-5 outcomes
25 Sep: testnet releases for (holesky, sepolia, hoodi)
2 Oct: Holesky fork at epoch 165376 (2025-10-02 12:06:24 UTC)
16 Oct: Sepolia fork at epoch 273152 (2025-10-16 14:12:48 UTC); note: during ACD
30 Oct: Hoodi fork at epoch 50944 (2025-10-29 22:11:36 UTC)
mainnet would be 30 days after Hoodi fork at the earliest
00:15:33
Barnabas:I have a slight adjustment recommendation (include forks only on wed)
https://notes.ethereum.org/@bbusa/fusaka-bpo-timeline
00:15:40
Barnabas:Also I have included all BPOs in it too
00:16:09
Dustin:"You don't have permission to access this resource.Sign in or head back to home."
00:16:23
Barnabas:Replying to ""You don't have perm..."
try now
00:17:30
Dustin:General note, whatever is chosen, Nimbus needs the holesky/sepolia/hoodi repos in eth-clients actually updated, they're the sole source of truth Nimbus uses, we don't copy/paste the numbers in
00:17:40
Dustin:in time for this code freeze
00:24:13 Christine Kim: Holesky Release deadline:
2025-09-24
Sepolia Release deadline:
2025-10-01
Hoodi Release deadline:
2025-10-14
Mainnet Release deadline:
2025-11-03
BPO 1 - target/max 10/15
BPO 2 - target/max 14/21
00:18:01
Parithosh Jayanthi:Replying to "General note, whatev..."
We can do that as soon as the dates are decided
00:18:33
kingy_sigp:we're ok with that timeline at lighthouse
00:19:04
Parithosh Jayanthi:Decided for Barnabas’ dates or the original?
00:19:15
Marius van der Wijden:Does it align to the boundaries
00:19:17
Marius van der Wijden:Ah perfect
00:19:20
Parithosh Jayanthi:yes
00:19:22
Marius van der Wijden:then yeah sgtm
00:19:28
Parithosh Jayanthi:We used the symphonious website for it
00:19:53
Tim Beiko:🛳️
00:19:53
Barnabas:I can prep that
00:20:34
Tim Beiko:I would prefer a single release for all testnets, but weak view
00:20:36
Justin Traglia:Replying to "Holesky Release dead..."
Where do the 10/15 numbers come from? I think this should be 9/14
00:20:57
Dustin:tradeoff is if anything goes wrong, then oops
00:20:59
Dustin:now it's emergency
00:21:04
Barnabas:Replying to "Holesky Release dead..."
9/14 is not 2/3
00:21:14
Dustin:with single release
00:21:27
Dustin:(i.e. if Holesky goes badly, and a pause is needed)
00:21:34
Justin Traglia:Replying to "Holesky Release dead..."
I mean not exactly but that’s fine.
00:21:48
Barnabas:Replying to "Holesky Release dead..."
We want exactly 2/3s
00:21:56
Parithosh Jayanthi:Replying to "Holesky Release dead..."
Well we can modify the update fraction if needed
00:21:59
Parithosh Jayanthi:Replying to "Holesky Release dead..."
But yes we should test it
00:22:01
Barnabas:Replying to "Holesky Release dead..."
Ideally not.
00:22:11
Barnabas:Replying to "Holesky Release dead..."
Update fraction works nicely with 2/3s ratios
00:22:12
Parithosh Jayanthi:Replying to "Holesky Release dead..."
(We have tested in the past, prefer not changing)
00:22:14
Barnabas:Replying to "Holesky Release dead..."
thats why we chose these numbers
00:22:20
Justin Traglia:Replying to "Holesky Release dead..."
Ah okay got it.
00:23:34
Trent:Updated the BPO graph based on barnabas’ doc
https://dune.com/queries/5603580/9115224
00:24:32
Barnabas:💯
00:24:36
Parithosh Jayanthi:Yeah we can do mainnet releases in oct and allow us to gain confidence
00:25:06
Marius van der Wijden:Excited, lets go 🚀
00:25:56
Justin Traglia:https://blog.ethereum.org/2025/09/15/fusaka-audit-content
00:26:08
Marius van der Wijden:The website projects 20k submissions by the end of the contest :D
00:27:00
Parithosh Jayanthi:10k from Marius right?
00:27:08
Trent:All typo fixes
00:27:11
felix (eest):19.9k use em dashes
00:27:25
Barnabas:Replying to "19.9k use em dashes"
—
00:27:54
Łukasz Rozmej:We got sumbission for estimateGas and tx limit, not sure if we should or not enforce it there
00:28:06
Parithosh Jayanthi:Replying to "We got sumbission fo..."
Yeah pk brought it up on ACDT too
00:28:06
potuz:Prysm has it
00:28:09
kasey:We have an implementation that we are testing and code reviewing.
00:28:10
Parithosh Jayanthi:Replying to "We got sumbission fo..."
There’s a thread on discord for discusion
00:28:18
kasey:*Prysm
00:28:20
kingy_sigp:we've finished it -- will be in unstable by end of week
00:28:30
Barnabas:Fredrik uses internet explorer :D
00:29:19
Łukasz Rozmej:Replying to "We got sumbission fo..."
where? :D
00:29:24
saulius:Grandine also has backfill, it will be merged very soon.
00:29:58
Fredrik:Sorry if I missed something in the comp stuff, had to take a call 🙂
00:30:04
Christine Kim:Replying to "We got sumbission fo..."
https://discord.com/channels/595666850260713488/688075293562503241/1416131247251521556
00:30:52
Łukasz Rozmej:Replying to "We got sumbission fo..."
nah thats eth_call
00:30:57
Łukasz Rozmej:Replying to "We got sumbission fo..."
that shouldn't have this cap
00:31:04
stokes:https://ethereum-magicians.org/t/h-star-name-for-consensus-layer-upgrade-after-glamsterdam/24298
00:31:25
stokes:https://ethereum-magicians.org/t/testnet-name-needed-for-sepolia-replacement/23221
00:32:08
Christine Kim:Replying to "We got sumbission fo..."
ah gotcha, my bad
00:34:00
James He:question, why do we need code freeze if we are expected to release next Thursday 🤔 is the commit needed for something else?
00:34:27
stokes:Replying to "question, why do we ..."
I think pandaops wants to run some validation on the code
00:34:30
stokes:Replying to "question, why do we ..."
Ahead of releases
00:35:12
Parithosh Jayanthi:Replying to "question, why do we ..."
We can run it on releases if you prefer
00:35:16
Francesco:The CL changes needed for BALs might not be needed after all, correct? @potuz
00:35:17
Parithosh Jayanthi:Replying to "question, why do we ..."
Depends on client team confidence tbh
00:36:00
Barnabas:https://github.com/OffchainLabs/prysm/pull/15712
00:36:01
potuz:correct
00:36:04
James He:Replying to "question, why do we ..."
Yeah I think on the releases makes more sense to us
00:36:12
potuz:we need changes in the engine API
00:36:17
Ansgar Dietrichs:it does seem desirable to only merge epbs and bal devnets once they are both stable - how much extra effort would these one-time changes for bals be on CL side?
00:36:57
Barnabas:Replying to "it does seem desirab..."
we only need a single CL impl.
00:37:24
Dustin:Replying to "it does seem desir..."
for Nimbus: nontrivial
00:37:39
Dustin:Replying to "it does seem desir..."
Much, much better to sequence things linearly
00:37:49
Barnabas:We plan to launch bal devnets next week.
00:37:53
Barnabas:I don’t think epbs will be ready
00:38:24
Łukasz Rozmej:Replying to "question, why do we ..."
run on release makes more sense for us too
00:40:03
Ansgar Dietrichs:Replying to "it does seem desirab..."
ah, right, makes sense we only need one CL client for the devnets
00:40:05
James He:I thought we wanted release by next thursday
00:43:51
Toni Wahrstätter:Replying to "it does seem desirab..."
Those changes have been started by terence and are continued by jihoon now
00:44:30
Francesco:Replying to "it does seem desirab..."
been started by terence
Many such cases
00:44:33
Parithosh Jayanthi:Did anyone do anything cool to celebrate merge day? 😄
00:44:46
Barnabas:Replying to "Did anyone do anythi..."
work on fusaka
00:44:49
Justin Traglia:Replying to "Did anyone do anythi..."
same
00:44:53
Parithosh Jayanthi:Replying to "Did anyone do anythi..."
lmfao
00:44:57
spencer-tb:Replying to "Did anyone do anythi..."
Whats the merge?
00:44:57
Barnabas:Replying to "Did anyone do anythi..."
lame
00:45:00
Łukasz Rozmej:Replying to "Did anyone do anythi..."
sleep
00:45:02
Parithosh Jayanthi:Replying to "Did anyone do anythi..."
weld
00:45:04
Louis:Replying to "Did anyone do anythi..."
The weld
00:45:06
Parithosh Jayanthi:Replying to "Did anyone do anythi..."
@steel