AllCoreDevs - Testing #073

|2026-03-09| Agenda: #1956| canonical JSON

Transcript

danceratopz:Alright, good morning, everyone. Welcome to ACDT number 73. It's the March 9th of 2026.

danceratopz:I will just drop the agenda in the chat for everyone to have a look at.

danceratopz:And we've got a lot of DevNets running, or about to launch, so I think that'll take up a lot of time today. Let's get to it.

danceratopz:First off, the status of blob DevNet Zero.

danceratopz:Would someone like to speak up about that, who has a bit more context?

Parithosh Jayanthi:That should usually be Barnabus, but he's out today. Last I've heard, there hasn't been any particular change, so I would say it's roughly the same as last week.

Parithosh Jayanthi:I know that he was looking a bit into bottlenecks with higher blob counts, but I think we have an agenda item later to talk about, engine API and SLZ.

danceratopz:Yeah, thanks a lot, Perry. That would actually be more or less now, if we want to talk about it now, if someone would like to add a… I know that Barnabas did a bit of benchmarking.

danceratopz:If someone else would like to pick that up, there was quite a bit of discussion in Ethereum R&D on this.

Stefan Starflinger:I mean, just to summarize the issue, so there has been tracing, like, OTEL tracing.

Stefan Starflinger:OpenTelemetry added to the clients, and then Barnabas looked at the bottlenecks, and he saw that

Stefan Starflinger:Serializing is quite the bottleneck.

Stefan Starflinger:And since we want to switch to SSC anyways, it would make sense to

Stefan Starflinger:have that as a way to communicate between the EO and CO.

Stefan Starflinger:And that's kind of the proposal. And through the capabilities, it's not really a breaking thing, just if the clients support it, then they can, basically use that, otherwise fall back.

Stefan Starflinger:And that's kind of necessary to scale it further. That's kind of the context of the PR.

danceratopz:Awesome. Thanks a lot, Stefan.

danceratopz:I guess this would be a decision that would be deferred to ACDE or ACDC? Would any client teams like to speak up to this suggestion and strongly oppose it, or welcome it?

Giulio:I will.

Giulio:I think, because…

Giulio:I have been doing some work on this over the weekend. I actually… I just… I… this is not, of course, this is a discussion for ACD, so I'm not gonna take time too much, but I did implement the spec for…

Giulio:Prysm, Lodestar, Lighthouse, Teku, Erigon, Nethermind, and Geth?

Giulio:I also run some benchmark… some benchmarks through Kurtosis.

Giulio:It works. It can do… it can be done very easily in,

Giulio:It can be done very easily.

Giulio:Optionally, so… yeah, I just wanted to give this quick update. I'm gonna present it maybe more on ACD, but it's basically

Giulio:done… I mean, it's not done, but there is already… there are already a lot of… there is already a proof of concept, and probably if you want to test it on blobdevNet0, you can take one of my PRs from one CL and one EL, and then just yellow it there, and you should see if that helps or not.

danceratopz:Awesome, thanks a lot for that, Julose.

danceratopz:I think that's really helpful, to help make a decision.

Parithosh Jayanthi:The proposal was also do it as an optional thing, right? So, worst case, we can roll it out asynchronously, and whichever clients have it implemented, it would be used, otherwise it doesn't.

Giulio:So, the way it works is that when you send a request to anywhere via HTTP or via the web, you can put your content type that you expect in the header. So, if you put the application JSON, what happens is that you're going to be redirected to JSON RPC,

Giulio:This is all execution layer side, right? So if the engine API, if the layer puts that JSON RPC, it will be routed to the JSON RPC. If not, it will be… if it puts Octream, it will be redirected to a REST full API,

Giulio:that uses SSD instead.

Giulio:So that's why it's optional. So the CL can, you know, can test if there is the… if there is…

Giulio:the CSS support by seeing if it doesn't get an error, if he puts OctetStream, and then he can, you know, he can decide himself where to go by default.

Giulio:You know, maybe at the beginning, of course, since this will require maybe Hive tests to be adapted, you will have it basically disabled, even in CL, and be enablable by a flag.

Giulio:But, yeah, I think it's, you know, but…

Giulio:But it is optional. It's very easy to make it optional, and it's already optional as I already implemented them.

danceratopz:Alright.

danceratopz:Thank you.

danceratopz:I think we'll leave that there, perhaps, to be taken up for further discussion, another call.

danceratopz:The only other topic we had there was… there was some,

danceratopz:issues on E69, but I think it was just some mainnet peers sneaking in. I don't think… I don't know if anyone wants to say anything to that. I think the problem's solved. Would there be any other topics for Blob DevNet Zero that anyone wants to talk about today?

danceratopz:Okay.

danceratopz:Thanks a lot, Gilo and Parry, for the context. Moving on to Glamsterdam.

danceratopz:Bavnet, sorry, BAL DevNet2 is still, running, and I think Stefan just posted, in the issue below.

danceratopz:That you'd like to, hi.

Stefan Starflinger:I can go over… I can go over it, if you would like. So basically, I got around to adding some more nodes. These nodes won't have any validators attached to them, they're just different configurations with the prefetch enabled, sequential, or with all of the valid optimizations enabled, I've now rolled that out on DevNet2

Stefan Starflinger:just get and Bezo, so we can capture some benchmarks. I will update also the Grafana dashboard with these nodes. They're currently still syncing, and that will give us some insights. I'm not sure how long I will be running them, but we will have a little bit of data for

Stefan Starflinger:how good the optimizations are, and I think DevNet2 also has a

Stefan Starflinger:decent state by now. I mean, it's not mainnet, but it's something since we've been spamming continuously on the network.

Stefan Starflinger:Since it started.

Stefan Starflinger:But otherwise, I think, regarding nodes, it's going pretty smoothly so far. Just one issue with the Bezu tracing that is already being looked at.

Stefan Starflinger:And Erigon, hasn't, given an update regarding, their progress on DevNet 2. I think they switched over, focusing on DevNet 3.

Stefan Starflinger:about Definite 3.

Stefan Starflinger:But I think that's all from…

Stefan Starflinger:a definite tool, and I think we can…

Stefan Starflinger:If it's okay if anyone else has something to say for Definite 2, otherwise we can move on to Definite 3.

Stefan Starflinger:Okay, then, for Definite 3, we've been doing a lot of, local testing and kurtosis, trying to, find any inconsistencies with the spec or, with the different implementations.

Stefan Starflinger:So it's still an ongoing, process, especially for 837. These gas changes also kind of make it difficult, for…

Stefan Starflinger:the ball, because quickly there are consensus issues with the blocker access list and gas accounting, so that makes it a little bit harder, but it would be good if we could get an update from the clients regarding the progress.

Stefan Starflinger:on, about definite 3.

Stefan Starflinger:Maybe we can start with, Geth?

danceratopz:I see your PR got merged, Stefan. Does anyone from the call want to… from Geth want to speak up to that?

danceratopz:Looks like Marius merged the fix.

danceratopz:Okay, if not, shall we move on to Besu?

Daniel Lehrner (Besu):I… I can give an update. So with the latest spec test, 5.3…

Daniel Lehrner (Besu):We have, 7 filling tests.

Daniel Lehrner (Besu):Two of them are false positives, where BESU recognizes the block as invalid, which is expected. We just returned the wrong error message, so I need to…

Daniel Lehrner (Besu):to make a PR for the execution specs to fix that.

Daniel Lehrner (Besu):The other 5 were 8037 differences between the 5.2 test and the 5.3.

Daniel Lehrner (Besu):I have been fixing them this morning. I think it should work now, so…

Daniel Lehrner (Besu):If everything is correct, I think we should pass all the tests, but I need to… I need to double-check. As I said, except maybe for some false positives, but for the consensus-specific stuff.

Daniel Lehrner (Besu):It should be… it should be correct.

danceratopz:Awesome.

danceratopz:Thank you, Daniel. Just for context,

danceratopz:Ben detected some gaps in AT37 coverage, created an issue, and added some tests himself, and Spencer added a few more tests, for AT37, which resulted in another execution specs release, 5.3, which was released, like, late on Friday.

danceratopz:And just a heads up that that release, includes all the tests filled.

danceratopz:Well, it includes Osaka, but it actually includes, all forks. So not just Amsterdam.

danceratopz:So yeah, then, if you see any, issues with the tests, please ping us in the Discord and let us know what needs to be changed.

danceratopz:Then… Stefan, you also found an issue in Nimbus that…

danceratopz:PR also seems to be merged.

Stefan Starflinger:Yes, we're making some…

Stefan Starflinger:some small progress, but I've also been testing with all the clients with a basic payload of creating a large contract and EVM fuzzing, and those networks don't seem very healthy yet, so there are still a lot of things to uncover.

Stefan Starflinger:That, I'm also working on, and would be great if, client teams

Stefan Starflinger:Can, run kurtosis locally to see

Stefan Starflinger:If they can find some issues as well, still. I updated the Kurtosis configuration in the spec sheet.

Stefan Starflinger:To the latest one, and also updated the Ethereum Genesis generator to include the Nethermind, Timestamps.

Stefan Starflinger:as well.

Stefan Starflinger:But I think, is there anyone from Reth, that could give an update as well?

Stefan Starflinger:That would be great.

Emma:We've got Dragon working on 8.37, I think most of the changes are there, it's just, like, testing and some fixes, but otherwise, I think, on the Reth side, we're passing all the tests, so…

Emma:I think we're looking good.

Stefan Starflinger:Okay, another mind, a quick update, that'll be great.

Ben Adams:Yeah, we have all the features implemented, we're planning some tests at the moment.

Ben Adams:And the BALS optimizations we're thinking, will come a bit later.

Stefan Starflinger:Okay, great. I think then we're making pretty good progress. It's mostly about ironing out all the…

Stefan Starflinger:small issues with gas accounting that basically straight away result in consensus issues because of buzz.

Stefan Starflinger:Is there anything we missed on DevNet3?

danceratopz:Do you have a feeling, Stefan… sorry, do we… do you have a feeling, Stefan, for when we could potentially launch? Is there any, date?

danceratopz:Targeted for that.

Stefan Starflinger:I think Marius just raised his hand, but regarding the deadline, I'll try to get it for Wednesday, again. Let's get it working well.

Stefan Starflinger:Marius?

Marius van der Wijden:Yeah, so, we just sat in a call for, like.

Marius van der Wijden:an hour, and I think they said for two more hours before that. Looking over the 1837 changes, it's…

Marius van der Wijden:like, the stuff that we have is partly implemented by me, which are the good parts, of course, and half of it is implemented by AI, and it's just so garbage that we basically have to, yeah, sit down and redo a lot of these.

Marius van der Wijden:I suspect we will not be able to finish, or I will not be able to finish that by 2.

Marius van der Wijden:by, by Wednesday.

Marius van der Wijden:The… yeah, the version we have right now, it has a…

Marius van der Wijden:still not passing a bunch of test cases, like a thousand, but, yeah, might be able to participate on the network, except for, yeah, if there are any transactions on it, basically. No, if there are, if there are,

Marius van der Wijden:complex, interactions.

Marius van der Wijden:So, yeah, I've tried to do my best to, to get there, but, yeah, it's not super easy.

Marius van der Wijden:And, yeah, the whole… logic of 1837 is also…

Marius van der Wijden:It's… yeah, it's… it's just not super nice, and it…

Marius van der Wijden:Every time I think more about it, I…

Marius van der Wijden:I want to change stuff, but then I think more about it, and I don't want to change them anymore, and so it's… yeah.

Marius van der Wijden:It's not an easy process.

danceratopz:Okay, thank you.

Stefan Starflinger:So, yeah, it's hard to say what the deadline will be. I think we're… we'll make…

Stefan Starflinger:some more progress until Wednesday, and I think then, we can see, and maybe… Have some more specifics.

danceratopz:none?

Ben Adams:Yeah, just to follow up on that, the implementation for us has been, like, an extremely deep change that, like, has tendrils that affect everything.

Ben Adams:And so we don't like it from that respect.

Ben Adams:But then, from the… from another aspect.

Ben Adams:Don't really think of anything.

Ben Adams:That's fair.

Ben Adams:So, yeah.

Ben Adams:is what it is, I suppose.

danceratopz:Okay, alright, thanks.

danceratopz:for the additional context, Marius and Ben, I think Wednesday does seem a bit ambitious. As always, I think

danceratopz:we can lean on Highview and the test results to see how clients are doing, this week.

danceratopz:And obviously, Stefan's amazing ketosis testing as well.

danceratopz:To see if it reveals some more gaps in coverage.

danceratopz:Okay,

danceratopz:Is there anything else anyone would like to discuss for BAL DevNet 3? Or we… can we wrap that up now?

Toni Wahrstätter:Maybe, maybe just a quick question, regarding the PAL optimizations.

Toni Wahrstätter:So Geth and Basil have been ready already with the parallel I.O. I wanted to quickly get a feeling how far other clients are with regards to the parallel I.O. optimization. I think I saw in the chat

Toni Wahrstätter:that, was it Ben or Lukash? Someone mentioned that Nethermind is… It's getting there.

Toni Wahrstätter:Very soon, so I'm wondering, is that already, like, Definite Free ready?

Toni Wahrstätter:And also hearing from other clients, like Erigon and Rap, how far they are with regards to the hardware.

Łukasz Rozmej:So, parallel transaction processing is almost ready, and we are experimenting with parallel I.O, but our current experimentations are not really netting any major benefit, and almost any benefit, to be honest, so…

Łukasz Rozmej:But we are not sure, maybe we still will.

Łukasz Rozmej:Managed to get something.

Ben Adams:Yeah, we're more focused on correctness than optimization at this stage, I think.

Toni Wahrstätter:Makes sense, yeah, thanks a lot for the update.

Toni Wahrstätter:Is there also someone from REF that could quickly… I guess Dragon is not on the call, but is there someone else who has insights into parallel I.O. on REF?

Emma:We've got the batch I.O. work in a PR.

Emma:Otherwise… .

Łukasz Rozmej:Does it give you any… mmm… benefit?

Emma:Oh, no, we're just doing pure implementation at this point, no optimizations.

Łukasz Rozmej:But I mean, have you benchmarked it? Does it give you any benefit?

Łukasz Rozmej:No, we haven't been to it. Okay.

Emma:But I can do that today and report back.

Toni Wahrstätter:Awesome, thanks a lot for the update. Yeah, and we have some tests prepared, some in the… it's in the…

Toni Wahrstätter:In the testing repo.

Toni Wahrstätter:There are some tests specifically written for benchmarking.

Toni Wahrstätter:That could be used, I can, I can post them into the Discord, so we could.

Toni Wahrstätter:Yup.

Toni Wahrstätter:Catch up async.

Emma:Coop.

danceratopz:Awesome. Thanks a lot, Tony and everyone, for chiming in there.

danceratopz:I'm thinking that's it for about DevNet 3?

danceratopz:moving to the CL, Would someone like to give an update on the ePPS DevNet?

Parithosh Jayanthi:Yeah, so EPBS DevNet Zero launched last week, and we had a couple of early issues.

Parithosh Jayanthi:Portos has a write-up on his blog as to what the issues were.

Parithosh Jayanthi:I've linked it over here. But besides that, we aren't particularly testing or stressing the network, it's just making sure that the nodes are up-to-date and rolling out patches from clients.

Parithosh Jayanthi:I guess more serious testing would kind of happen on EPBS DevNet 1.

danceratopz:Thanks, Perry. I think POTUS wanted to talk about forcing the EL to trigger block production on older heads.

danceratopz:And the issue of PTC lookout.

danceratopz:POTUS, would you like to speak up on those topics?

Potuz:Yeah, so one is sort of, like, mandatory, and the other one, we need to make a decision what… how to deal with. So the first one is on the EL side. Today, there's, on the engine API,

Potuz:There is a May that I think needs to become a must.

Potuz:Which is the fact that when we send a four-choice updated message to the EL for a previous head.

Potuz:then the EL just ignores it and gives us a payload… an empty payload ID.

Potuz:But on EPBS, the problem is that we actually might have an honest reorg on the CL

Potuz:That, does a four-choice update to a previous head on the current branch.

Potuz:So I… currently, because of the issues that, well, we have, like, partial fork choice in the DevNet, this is actually being triggered on Prysm essentially every few blocks.

Potuz:And yes, so, well, that's the comment. I need input from the EL. Marius already replied something, but then that sort of died on Discord. I need to know from the EL if this is hard, if this is trivial, if we can have it soon, so that we can actually start testing, these kind of things.

Łukasz Rozmej:And alleles work like that?

Potuz:Well…

Potuz:All the else, as far as I know, give us an empty payload ID if I give you an FCU to a previous head.

Potuz:At least every time that we test, the EL gives us an empty payload ID,

Potuz:they ignore an FCU to a previous head.

Potuz:And they don't trigger blood production. When I do an FCU for an old head, And that breaks CPBS.

Marius van der Wijden:Yeah, the program.

stokes:thing, or… Like, do you think it'll get better with multiple choice progress?

Potuz:No, no, no, I mean, this has to be the case. I mean, a reorg… I might reorg to empty, like…

Potuz:there's an honest behavior, which is voting for the empty block, and in that case, the CL needs to reorg.

Potuz:Right. A current head, and it needs to be able to produce a blog on top of a previous head.

stokes:I bet.

Potuz:The thing that is triggering this… I mean, this shouldn't happen on production, on mainnet, regularly, right? So we want to continue building on full, but the problem is that Lighthouse and Lodestar are voting for empty everywhere.

Potuz:And they, they outweight us, and so we take this as a signal that we need to reorder the payload, and we try to build on empty, but then when we try to build on empty, the EL just grabs an app on us, and we get reordered.

danceratopz:Marius, did you want to add something?

Marius van der Wijden:Yes, so… The problem is that we, the spec does not allow FCU

Marius van der Wijden:To, a… a parent of…

Marius van der Wijden:little block, because that basically means we are reorging back the chain. The payload production is a different service.

Marius van der Wijden:To allow… Dude…

Potuz:Yeah, it's… I can't hear Marius.

Łukasz Rozmej:Yeah, we cannot hear anything.

Marius van der Wijden:So… No?

Marius van der Wijden:Goddammit.

Łukasz Rozmej:Every Friday.

Marius van der Wijden:Good Lord.

Łukasz Rozmej:Maybe.

Marius van der Wijden:Okay, different…

Łukasz Rozmej:Might be the internet.

danceratopz:Okay, we'll see if Marius can resolve his connection issues. Are there any other EL devs that would like to speak up to this in the meantime?

Łukasz Rozmej:Yeah, I'm taking a look, and I explicitly see this in our codebase, that yes, if we have,

Łukasz Rozmej:Fork choice to a…

Łukasz Rozmej:payload that is on the same chain as the current head, we ignore it, so… not sure what should we do, and how's the specification.

Łukasz Rozmej:Phew.

Marius van der Wijden:Yes, this was part of the spec. I hope you can hear me better now. I'm in a different Wi-Fi. Hopefully, I will get better internet soon. So, the problem is that we are forbidding these reorging the

Marius van der Wijden:reorging the same chain. What we could allow is, if you specify older, if you specify the new FOC choice.

Marius van der Wijden:but older payload parameters that we are building, these older blocks for you. We could do that, or we could allow for reorging the chain. But I think there was an issue with reorging the chain, why we specified it this way, but I'm not sure.

Łukasz Rozmej:I will add interesting… context in our…

Łukasz Rozmej:HiCore plugin, we actually have this feature disabled, so you can reorg to all the train.

Łukasz Rozmej:Which is interesting.

Łukasz Rozmej:But it's not in the mainnet implementation.

danceratopz:Okay, it seems like this topic will need to be followed up on.

Potuz:I just want to stress something that… I mean, it's not a matter of something that might be optional. There are reorgs on the EL that will happen where we roll back the chain.

Potuz:And that… that will be the head. Even if no block is produced.

Potuz:We are going to call the yel, saying, you just need to roll back your head.

Marius van der Wijden:Yeah, this is so horrible, UX. I… I… I know that this…

Marius van der Wijden:the way it's going to work now, but this is so horrible to UX, because we are telling the users that, hey, this is the new head, oh no, actually, the previous head is the new head. Or actually, no, we have now another sidechain that is the new head.

Potuz:But isn't that the way that it works now?

Marius van der Wijden:Yes, but it never happens.

Potuz:Is this behavior when you reorg a block?

Marius van der Wijden:Yes, but reorgs never happened.

Marius van der Wijden:And it seems that with.

Potuz:Yeah, reorgs will never happen on mainnet either, but we need to deal with them.

Marius van der Wijden:Okay, okay, then, then it's fine. Like, we can just prepare a branch that removes this check.

Marius van der Wijden:For… for Bell DevNet 1, or 0, or wherever you guys are at. I think that is… that is not a problem. Like, I can do that right now.

danceratopz:Okay, are there any other young, client devs who would like to speak up to that, or…

danceratopz:Does this discussion need to be taken in more detail elsewhere?

danceratopz:BOTUS, are you happy with the comments you've received?

Potuz:Yeah, definitely. If we just have an implementation that we can test, that'll allow us to test for choice. I mean, we don't expect to see many of these kind of reorgs, but we need to deal with them, and we need to test our for choice, so we need to have a branch that does this.

danceratopz:Okay.

Ben Adams:I do it.

Ben Adams:Where are you?

Potuz:no rush at all. We are not going to test for choice until two DevNets from now, so there's no problem at all not having this behavior now. We're not thinking on testing these things, even on DevNet 1.

Ben Adams:Is it… is it purely a testing thing, or would it need to… We expect change for me.

Potuz:It happens that we reorder blocks on mainnet. It will happen that we will reorder payloads on mainnet, which is done…

Potuz:We don't want to see it regularly, as they just don't happen today regularly. But it does happen that we have reorgs on mainnet.

danceratopz:All right, POTUS, when's the next EPBS breakout?

Potuz:We don't have breakouts since we have this meeting.

danceratopz:Okay, good. But I think the message was pretty clear. Thank you.

danceratopz:Did you want to talk about the PTC look-ahead as well?

Potuz:Yeah, this is an issue that Nico found, which… which is, yeah, it's a flaw in our spec. We…

Potuz:We need to… PTC attestations are sent during one slot.

Potuz:And then they are included in the next slot's block, and they are validated when we process those… that block.

Potuz:The problem is that, the PTC committee is computed with the balances of the given state.

Potuz:And that means that if you attested being on the PTC on slot 31,

Potuz:And your attestation got included in slot 32.

Potuz:you are validating the block that included that PTC attestation, and you're looking at the effective balances

Potuz:Of the current slot, which is 32, it's in a different epoch.

Potuz:And those are… that may have changed the PTC, the full committee. So an attestation that was included in the blog, because it was violent then, might just become invalid now.

Potuz:And to deal with this, we have a few options. I opened the most naive of the possible PRs, which is including

Potuz:a sort of like a look behind. We just cache

Potuz:the full PTC committee for the current epoch, well, the previous epoch and the current epoch on the beacon state.

Potuz:It is mostly useless, because this only happens at that particular slot. It's just one slot that is needed.

Potuz:But…

Potuz:Clients, anyway, seem to cache the full epoch to have duties, so it seems… on a first run, it seems that clients don't mind this.

Potuz:But I don't know. I wanted to have a small discussion, at least on what are the options, or what people prefer.

Potuz:I really have no strong opinion on the topic.

Potuz:Perhaps Nico is here, who found the topic, if you can just say something.

nflaig:Yeah, so I'm not sure what's the cleanest spec design here, but as for Loadstar, we currently cache the current and previous epoch, so for us, it wouldn't be extra memory usage. Also, I think it's the equivalent of

nflaig:I think 64 validators or something, so it's pretty small memory-wise.

Potuz:Yeah, so the current PR, I think it's 256 kilobytes per state that is added.

stokes:Does the spec right now say…

stokes:Like, I'm sure, like, the spec is correct, right? In terms of who you validate when and where?

Potuz:No, no, the spec is incorrect, actually. The spec, the specs… I mean, it's clear on whom you need to validate, but the problem is that, some.

stokes:It's using it wrong.

Potuz:It's valid in the previous epoch is not valid in the current epoch.

stokes:Right. Okay.

Potuz:This is a bugger.

stokes:So it's… it's using the wrong state at that point in time.

Potuz:Yep.

stokes:Okay.

stokes:then yeah, we should fix the spec. But I think what you suggested makes sense. Yeah, like, I don't think we need to be more invasive, have some cash explicit in the state.

stokes:Clients can just cache it locally.

stokes:But I should just be clear, yeah, who you want to validate against.

Potuz:The cache is actually quite trivial. We are… it turns out that Prysm is not caching the full epoch. Prysm is only caching the committee and computing slot by slot, but moving to the full epoch is trivial for us, so it's fine.

stokes:Do you have a link to the PR?

Potuz:Oh, yeah, let me find it.

stokes:Okay.

Justin Traglia:To be clear, this solution that you guys just talked about, would that mean not adding anything to the state?

Justin Traglia:Like, caching it in the clients outside of the state.

Potuz:No, no, no, this needs to be added somewhere. It needs to be clear what is the state that you're validating with, so…

Potuz:the PR caches it in the state.

Justin Traglia:I see.

stokes:Yeah, big 10.

Justin Traglia:256 kilobytes is kind of a lot to be added to the state each.

Justin Traglia:Is it a slaughter epoch?

stokes:I think it's each epoch, but yeah, why can't it just say, you know, use the state?

stokes:In this case, I have the last slide.

stokes:Like, you just should just use the state that corresponds to the validators you're.

Potuz:Well, the state transition function is a function that takes one state, one block, and recovers a new state.

Potuz:If you… change the signature, that's a much deeper change. You're gonna be saying now, take two states.

stokes:Yeah, but we're seeing…

Potuz:Give me a new stake.

stokes:Hmm…

stokes:Yeah, I don't know… Is there nowhere else we don't do this?

Potuz:Yeah, there's no… no, no, this is… there's no way we're doing this, and… and even… we tried, like, milder changes before, of having one state and two blocks, like, to use the previous block as part of a state transition function, and…

Potuz:Danny almost kills us, and forced us to, like, cache… whenever we are in this kind of situation, we add an extra cache to the state. That's why we have headers, and we have a bunch of stuff in the states that we could just easily take from the previous block.

Justin Traglia:I think I'd prefer a solution that just adds the least amount of data to the state, whatever that is.

Potuz:So there is a solution that only adds, So, two committees?

Potuz:That is… that will be,

Potuz:a thousand, indices, so it's 8 kilobytes, I think.

Potuz:And that works.

Potuz:But then that forces clients to be, like, updating this

Potuz:So, to keeping the cache in memory, and then updating this state every slot. I can open that PR if you want. It's easy to open.

Potuz:I don't know what CL clients would think it's… I mean, there's a trade-off of complexity versus, like, having the space in the state or not.

Justin Traglia:Yeah, I would hold off on adding a PR for that. I think we should just continue discussing this more.

Justin Traglia:There you go.

Potuz:This one is a little bit more urgent than the previous one, because we do want to test PTC attestations, and we suspect that if we send PTC attestations now on DevNet0, we're gonna split. So I think that's the last thing we're gonna do. When we tested all of the existing structure.

Potuz:we are going to say, well, okay, so now we're fine by splitting, let's just send PTC attestations and see if we actually split, or everyone implemented the same kind of sort of cache.

Justin Traglia:Got it, yeah, let's make a decision on this, this week, and I can also…

Justin Traglia:make a SPAC release with this change as well.

danceratopz:Alright.

danceratopz:Thanks a lot, Justin.

danceratopz:I think we can continue that conversation async. Just in case you missed it, POTUS, Mario's already pushed a fix to the, GEF branch, the fix for reorgan to parents in the child wealth.

danceratopz:Is there anything else that we'd like to discuss on the ePBS DevNet before moving on?

Potuz:Can we quickly, like, decide parties here of what sort of, like, things we can…

Potuz:We can send, or… or it's fine if we just, like, comment on the chat.

Potuz:We want to, like, stress now EPBS.net zero. We can probably kill it by… but we can… we should, like, test

Potuz:I don't know, withdrawals and these kind of things. So for this, we just need to make sure that all… that we have clients that can handle, or that

Potuz:Think that they can handle, and what structures they can handle.

Parithosh Jayanthi:Can we make a thread on EPBS, and then we can have a list of things, and we'll take care of it.

Potuz:I'm sorry, Barry, I didn't hear you.

Parithosh Jayanthi:Can we make a list on the ETH R&D EPBS channel? And then, once we have a list, we can just go through it one by one? I'd rather first test it out locally before just breaking the live network.

Justin Traglia:Maybe it'd be nice to get some verbal,

Justin Traglia:Discussion on this really quickly, though? Like, what can clients not do? Is there anything?

Justin Traglia:work.

Justin Traglia:I don't know. I kind of agree with Botus that it might be nice to talk about.

Parithosh Jayanthi:Yeah, maybe withdrawals is the easiest first one. Is there any client that can handle withdrawals on EPPS DevNet Zero right now?

Potuz:Well… Withdrawals will happen once we have, Yeah, I don't know.

Potuz:We need to… so withdrawal request, you mean, right?

Parithosh Jayanthi:Yep.

Potuz:I think we should be able to ha- to handle them. Not sure. We can try.

Parithosh Jayanthi:Okay, we can create a local test and then share the outcome.

Parithosh Jayanthi:I would assume it's the same for deposits, blobs, transaction load, everything, so we can just try one test locally with everything and see what breaks?

Potuz:Right, so I'd like to try,

Potuz:Current structures that exist, whether or not we can continue to handle them, handling them.

Potuz:Especially if there is a missed slot, or a reorg of a payload.

Potuz:So, now the depth net is perfect, because there aren't that many missed lots. So… So it's good.

Parithosh Jayanthi:Yeah. Yeah, what we can probably do is Asserta can turn on and off block production, and then we can trigger missed slots via that, and then make sure that the network's still fine.

Parithosh Jayanthi:Yeah, I'll set up a test for it.

Justin Traglia:When you say builder deposits, or sorry, when you say deposits, does that include builder deposits?

Potuz:No, no, no, just the usual structures that exist now, and then once we know that we can handle those, like normal deposits, then we can just start dealing with builder deposits.

Potuz:I think that should be easy.

Potuz:And after that, we can just kill everything with BTC attestations, or trying to reorder just payloads. Like, we can start, like, withholding payloads, and these kind of things, and see if people, like, reorder the full block, or just build on empty.

Potuz:And then I'm sure we're gonna kill the devnet. I'm sure we're gonna be split then, but that's fine.

Justin Traglia:Okay, we'll continue discussing this on the ETHRNG Discord.

Justin Traglia:Dan, are you there?

danceratopz:Yeah, sorry, everyone. I had a bit of a sound problem for a minute, but I think I'm back now. I could hear you, Justin. Sorry about that. I lost the last minute of whatever was said or not said.

danceratopz:So, I have one short… in my agenda, I mentioned that Teco and Nimbus are not running on the DevNet. Is that… is that not true? Are all clients running on the DevNets at the moment?

Potuz:Techo and Nimbus are not running.

danceratopz:Okay.

Potuz:But, Stefan, I don't know if Stefan is here, but Stefan said that he might be able to hack a branch to sync.

Stefan Bratanov:So if we, if we start…

Potuz:from Genesis, I think, both work, but syncing is a problem, or checkpoint syncing is a problem.

Stefan Bratanov:Yeah, I'm working on it at the moment, but haven't done it yet.

danceratopz:Okay.

danceratopz:I was just wondering whether, it's time to up the ante on testing on the DevNet when some client teams haven't joined yet, but…

danceratopz:I'll let you, decide that based on local testing, if that's how you're gonna move forward.

danceratopz:Are there any other topics for EPS DevNet Zero before we move on?

danceratopz:Alright.

danceratopz:Then, Tony brought up on the agenda

danceratopz:he wanted to talk about a networking protocol change for Snap 2. Tony, would you like to speak to that?

Toni Wahrstätter:Yes, I will be quick with that one. Gary and me have,

Toni Wahrstätter:began inspecting SNAP2, which is, SNAP protocol using block access list.

Toni Wahrstätter:It's an early draft, so I would love to have more eyes on it.

Toni Wahrstätter:Basically, instead of, using getTryNotes, it will use the block level access list to…

Toni Wahrstätter:do the healing phase in SnapSync. That's kind of the TLDR.

Toni Wahrstätter:And, yeah.

Toni Wahrstätter:More to become.

Toni Wahrstätter:But would be great to get some feedback on it.

danceratopz:Sorry, yeah, thanks a lot, Tony. Tony just dropped the link in the chat if anyone wants to provide feedback.

danceratopz:Please take a look.

danceratopz:And I think Joachim's question moves on to the next agenda point, so if, just quickly.

danceratopz:If anyone would like to mention something now about SNAP2, please speak up. Otherwise, we'll move on to EV70, EV71.

danceratopz:Okay.

danceratopz:So, Tony had another agenda point for ETH71.

danceratopz:To clarify the response size cap, would you like to say something to that?

Toni Wahrstätter:Yeah, this was, this was a mistake I made, so I was just, thinking of the 10MB limit, which we had in EVE70 with the receipts, but, thanks to Besu, who pointed out on EVE Magicians, that GEF is using, for example, GEF is using a 2MB softcap.

Toni Wahrstätter:And EVE71 for block cloud access lists now also have… has the same soft cap of 2MB. So this was a…

Toni Wahrstätter:Very recent change I just did today, just as a heads up.

danceratopz:Awesome. Thanks a lot for the information, Tony.

danceratopz:Is there anything to discuss for EV70?

danceratopz:Does anyone have context on this?

danceratopz:Is this blocking anyone?

danceratopz:Okay. If no one… Has anything to say there?

danceratopz:then I would move on to…

danceratopz:Benchmarking and state bloat, in particular performance DevNet 3.

danceratopz:Would someone like to give an update on the Perf DevNet 3?

Louis:I can give a quick update. So, for the blotting process, we already completed blotting for EOA account and ERC20. We have a smaller size ERC20 that targeted, necessary benchmark.

Louis:And, we have an updated version of a related benchmark, including account access benchmark and Storage Operation Related Benchmark.

Louis:And we're now moving forward to generate the payload using guest benchmark tools from Nita and my team.

Louis:And collecting the data using Benchmarker.

Louis:On the agenda, I see there's some questions regarding the snapshot management. I'm not entirely sure about the details.

Louis:I see Johong on the call, maybe Johong could chime in for more details?

jochem-brouwer:Yeah, so on the… can everyone hear?

jochem-brouwer:That's true.

jochem-brouwer:Yes?

jochem-brouwer:Oh, great, great. So, on the DevNet, well, to take a snapshot, you of course have to have a synced client.

jochem-brouwer:As of now, we have, the clients Rev, Nethermind, Besu, and Kev.

jochem-brouwer:Here we go as a problem, but the plan is, if it is synced, then we will take another snapshot, and we will use that snapshot for the snapshot item.

danceratopz:Alright, thank you for the updates.

danceratopz:Lewis, did you want to add any other… did you have any other questions for client teams regarding…

Louis:Adding note from my side. Thank you.

danceratopz:Okay.

danceratopz:Jocham?

jochem-brouwer:Yes, one more point. So we are currently targeting the benchmarks for the repricing.

jochem-brouwer:But, I will also ask, to start learning the block-level access list benchmarks.

jochem-brouwer:We have some…

jochem-brouwer:benchmarks for that already. I think they are not merged yet, but I will focus on that this week to get these merged, and to also add additional benchmarks to also target the block level access lists.

jochem-brouwer:I think we will target the Perth DevNet, because then we can also focus on the state.

jochem-brouwer:To see what the impact is on the block access list there. So, also building clients with the block access list optimizations and without them to see the,

jochem-brouwer:Well, what they optimize and what the impact of these is.

jochem-brouwer:Yep, thank you.

danceratopz:Awesome, thank you, Joachim.

danceratopz:Is our all client synced to the DevNet now?

danceratopz:Or is it…

jochem-brouwer:No. Is there still some points missing?

jochem-brouwer:Here we go, it's not fixed, but I see on Discord that there is a fix ready.

jochem-brouwer:Except that it needs a resync, so this

jochem-brouwer:is going to take some time, but this is not a blocker for the benchmarks, we will bring them on the synced clients, and we will try to get everyone synced, and to also add those to the benchmarks.

jochem-brouwer:And one of the reasons, just to add this, is that the, well, not the reason, but one of the causes is that only DevNet, we really, well, we raise, like, the gas limit by a lot, and this stresses the clients, of course, and, well, this might run into unforeseen issues.

jochem-brouwer:Like, I'm not going to speculate, but one of these things are, like, out-of-memory issues where,

jochem-brouwer:well, there's, like, a lot of state changes happening, and these state changes are all loaded into memory, and this causes an out-of-memory. And this, of course, only a thing which happens if you, like, raise the guesstimate by a lot, and then start to spam this network with a lot of these bugs.

jochem-brouwer:But it's, I think, still nice to see… to find these kind of bugs and to fix these now. So we are basically ready for, like, a 5-giga-gas network based on these kind of stats, yeah.

jochem-brouwer:But I'm not saying we should move to 5G, I guess, on mainnet, of course, yeah.

danceratopz:Awesome, thanks a lot for the clarification, Joachim.

danceratopz:All right, any other topics for Perf DevNet 3, or for… Other benchmarking topics in total?

danceratopz:There's, of course, an informal gaslighting committee meeting tomorrow, if client teams wants to join.

danceratopz:Just drop a line to Lewis.

danceratopz:Or Maria, or Jocham?

danceratopz:And us to join.

danceratopz:Otherwise?

danceratopz:Any other topics on benchmarking or stay below?

danceratopz:Okay.

danceratopz:So, we're very close to the end of agenda. The only other topic

danceratopz:that I would, that I added to the agenda was basically there was some discussion about the ACDT cadence, and whether we should reduce this. I think…

danceratopz:I would just say, given the amount of discussion we have on this call at the moment, I think it's… it's merited to have it weekly.

danceratopz:But of course, it's… I understand that it's a diverse scope of topics, and that not all topics are directly relevant to a lot of people on the call.

danceratopz:I added the link to the thread in the ETH R&D Discord server thread.

danceratopz:To the agenda, please chime in there, but if anyone wants to add quickly their two cents before we finish up for today.

danceratopz:Please feel free to speak up.

Parithosh Jayanthi:Yeah, maybe one thing I was thinking of was taking a call about next week's ACDT by roughly, like, Wednesday or Thursday.

Parithosh Jayanthi:If it feels like there's mostly just topics to discuss on one specific, feature, for example, if it's EPBS or whether it's block-level access list, then we could just defer the call to an EPBS breakout, so that the teams can go deeper into those, those niches.

Parithosh Jayanthi:Yeah, portal.

Potuz:I'd suggest that we have a call that doesn't have a topic, and whose topic is chosen every week. So, for example, now we could say,

Potuz:benchmarking this or that feature on block-level access list requires a discussion, let's just take that spot on Wednesday.

Potuz:So that that call actually is a technical call, and only is attended by people that know that they want to be there.

Potuz:No, Nick, so it's not really ACDT, because ACDT now has

Potuz:I don't know, 20, 30, 40 people talking about completely separate topics, and none of them actually gets to be discussed in detail.

Potuz:Well, at least none of the ones that I know about.

danceratopz:So, this suggestion would be similar to how ACD got split into the EL and the CL calls. We could do the same for ACDT and rotate.

Potuz:No, because that… that keeps them… that keeps them every week, already forced, everyone from ACDC to go there. What I say is.

Potuz:we have a topic, we decide now, today, or an ACD, or whatever, or even an async, we have this topic, let's cover it in that call. And let's just have a focused call that is a very technical call.

danceratopz:So just reserve the spot for breakouts, and define the breakout subject based on demand.

danceratopz:Okay, Tony, did you want to add something?

Toni Wahrstätter:Yes, we also wanted to stop doing the Brock Cloud Access List breakout calls, and move them to ACDT.

Toni Wahrstätter:So as of… as of today, we are still on Wednesday.

Toni Wahrstätter:Every two weeks is the Block Club Accessless breakout call, and every other week there is the repricing breakout call.

Toni Wahrstätter:And we could also use that spot, so I think that's basically what Podos said. For example, with all the benchmarking that is still needed and might still need a breakout call, we could move it to do it weekly, or have that new spot that the block of access list breakout call had to do other, breakout sessions.

danceratopz:Okay. All right, thanks a lot. Yep, sorry, did someone want to speak up?

Parithosh Jayanthi:Yeah, I think part of the issue might just be that it's a seasonal thing. Maybe right now, the better option is to seed ACD in terms of,

Parithosh Jayanthi:sorry, ACDT in terms of more breakout sessions, because it seems like there's more interest in diving deeper into topics rather than making quicker decisions. Maybe in two months, that's not the case anymore, and we just need to meet to make quicker decisions.

Parithosh Jayanthi:So yeah, we… Mario and I can maybe take a call on it on a weekly basis. If it feels like there's a lot of piled up decisions across different streams, we can say that we do ACDT, but on weeks that there aren't, then we can, cede to…

Parithosh Jayanthi:switch to a breakout format. Does that sound like a fair outcome?

danceratopz:That sounds fine.

danceratopz:Thank you, Perry. I think we can leave it there. Continue a discussion async, perhaps in that thread. Thanks, Justin, also for offering to help, that's awesome.

danceratopz:We're at the hour, so thanks a lot for everyone for coming.

danceratopz:And for your contributions, been a really good call.

danceratopz:Thanks a lot, see you next time.

Potuz:Bye-bye.

Louis:Thank you.

Łukasz Rozmej:Bye, thank you.

Toni Wahrstätter:Thank you, bye-bye.

Chat Logs

danceratopz:https://github.com/ethereum/pm/issues/1956

Parithosh Jayanthi:https://github.com/ethereum/execution-apis/pull/764

Giulio:https://ethresear.ch/t/binary-ssz-transport-for-the-engine-api-an-initial-benchmark-on-a-live-network-with-kurtosis/24324

Łukasz Rozmej:not basically done -> in Nethermind it needs to be redone from scratch properly.

Łukasz Rozmej:Have we experimented with also using compression between EL<->CL?

FLCL:compression makes it slower

Parithosh Jayanthi:Also amount of data isn’t the issue right? Its just serializing/deserializing overhead

Giulio:yeah the best case is literally passing pointer to the data struct

Łukasz Rozmej:you mean shared memory or something?

Giulio:yeah

Giulio:but that's for the future, shared memory beetwen processes is hacky

Giulio:possible ofc

Łukasz Rozmej:it's quite simple, I've coded those in the past

Potuz:The best is not to pass anything

Giulio:the best is to pray

Potuz:not even a pointer, just a root to sign

Łukasz Rozmej:you can have simple messages for communication and shared memory for the payload

Łukasz Rozmej:makes it simple

Ben Adams:Replying to "The best is not to p..." vibe validation

Giulio:some machine may run CL and EL on different machines

Potuz:Replying to "The best is not to p..." no, we don't need anything, and on notify_newPayload, we should pass the snappy we get

Giulio:*some operators

Łukasz Rozmej:yeah we need a fallback then

Potuz:Replying to "The best is not to p..." and you deal with it directly

Giulio:the best case is to pass pointer :)

Łukasz Rozmej:so it is only optional optimization

Maria Silva:Sorry if I missed it, but what is the status on BAL optimizations for the various clients?

Giulio:yeah like SSZ, we can do that in the future

Potuz:Replying to "The best is not to p..." requests are essentially the only thing we need

Potuz:Replying to "The best is not to p..." withdrawals could be just the hash

Giulio:I recall there was some feature for fast jsonrpc through shared memory but forgot about it

Łukasz Rozmej:Replying to "Sorry if I missed it..." still in progress on Nethermind side, sorry

Toni Wahrstätter:no updates,but Geth and Besu have been ready and devnet-2 is running with the optimizations enabled/disabled on different nodes

Stefan Starflinger:Replying to "Sorry if I missed it..." Ah yes I forgot to ask, reth has some flags updates otherewise nethermind and etherex we don't have any flags yet. Thanks for the update Lukasz

Łukasz Rozmej:just FYI the more we implemented 8037 the more we disliked it :(

Stefan Starflinger:Replying to "Sorry if I missed it..." https://notes.ethereum.org/@ethpandaops/bal-devnet-3 flags documented here

Dragan Rakita:Does it include refill of reservoir?

Łukasz Rozmej:Replying to "Does it include refi..." @Ben Adams?

Stefan Starflinger:https://notes.ethereum.org/@ethpandaops/bal-devnet-3

Dragan Rakita:Replying to "Does it include refi..." On revert/halt return child spent state_gas to reservoir.

Dragan Rakita:Replying to "Does it include refi..." Let me ping you in discord

Maria Silva:Replying to "just FYI the more we..." What are the messy parts? Anything we should take into the design and improve on?

Dragan Rakita:Replying to "Does it include refi..." https://discord.com/channels/595666850260713488/1476205965895536782/1476208158421549096

spencer:Replying to "Does it include re..." Yeah it does! It is up to date with EIPs repo now. This commit: https://github.com/ethereum/EIPs/pull/11328/changes/ec612217fd2fda7409369a98d160a929829cb66f

Ben Adams:Replying to "just FYI the more we..." Is a very deep change that effects everything

Stefan Starflinger:AI slob testing :D

Ben Adams:Replying to "just FYI the more we..." @Marius van der Wijden would be interested in what you'd change

felix (eest):Replying to "AI slob testing :D" 0-90% feels so nice with AI, but the last 10%..

spencer:Replying to "AI slob testing :D" Can we create an agent called slob?

felipe:I have some more BAL benchmark tests lined up that we should sync on @Toni Wahrstätter

Potuz:I'd like to discuss the two topics I added in the end to the agenda

Parithosh Jayanthi:https://www.potuz.net/posts/epbs-devnet-0/

Louis:Replying to "I have some more ben..." @Toni Wahrstätter @felipe https://github.com/jochem-brouwer/execution-specs/pull/1 This is for BAL benchmark by Felipe

Louis:Replying to "I have some more ben..." Sorry I’ve not yet reviewed it

Stefan Starflinger:Replying to "AI slob testing :D" Its very good when there are 1 or 2 things wrong, but when there are like 3+ things wrong its a real mess

Toni Wahrstätter:Replying to "I have some more ben..." Yeah def! That's the one focusing on parallel IO I meant: https://github.com/ethereum/execution-specs/pull/2033

Potuz:The spec does allow it

Potuz:it just says MAY

Łukasz Rozmej:is Marius breaking up?

NC:Replying to "is Marius breaking u..." yes

felix (eest):german internet back at it again

Toni Wahrstätter:Replying to "I have some more BAL..." @felipe lmk how you want to proceed. My PR is still a draft but happy to convert it :)

Potuz:Thanks :heart:

Marius van der Wijden:Pushed a fix to our epbs-devnet-0 branch that allows for reorging to parents

Francesco:The current PR seems fine if clients are fine with the occasional last minute change for the first slot of the epoch

Justin Traglia:https://github.com/ethereum/consensus-specs/pull/4979

Ben Adams:Will we need the re-org behaviour spec'd somewhere? Since isn't allowed atm

Marius van der Wijden:Replying to "Will we need the re-..." Yes we should. I remember there were engine api tests that demanded this behavior

NC:We are adding 256kb to the state, but we really only 4kb (1 slot of PTC) of it. Is that right?

nflaig:Replying to "The current PR seems..." I think this is fine, we fetch duties at the start of the epoch and ptc duty is quite late in the slot, imo we don't really need 2 epoch lookahead

Potuz:I love you @Marius van der Wijden

Eitan Seri-Levi:I think LH might be able to handle them

Potuz:@Parithosh Jayanthi that's mostly a call for you, I suspect that deposits should not affect neither Teku's or Nimbus ability to catch up

Parithosh Jayanthi:Replying to "@Parithosh Jayanthi ..." Yeah I agree, I think its worth already breaking/finding out what’s broken in existing clients

Toni Wahrstätter:snap/2: https://github.com/nerolation/EIPs/blob/toni/snap-2/EIPS/eip-9999.md

jochem-brouwer:Does this remove the same msgs from eth/71?

jochem-brouwer:Antwoord verzenden naar "Does this remove t..." I will discuss this async

Potuz:@stokes want to discuss the honest reorg feature here in light of FCU changes?

Toni Wahrstätter:That's the PR for the eth/71 soft cap: https://github.com/ethereum/EIPs/pull/11387

Ameziane Hamlat:So the idea is to test the BAL optimisations on top of perf-devnet-3 (Bloat network) ?

Maria Silva:Replying to "So the idea is to te..." Yes, because we need bloatnet for the stageful repricings tests

Maria Silva:Replying to "So the idea is to te..." *stateful

Toni Wahrstätter:Replying to "So the idea is to te..." The pure exec optimizations (no state) can already be tested on devnet-2. For state related things we need to use a mainnet-state fork or bloatnet

Ansgar Dietrichs:I heard 5 Gigagas on mainnet? brb notifying stakers

Ameziane Hamlat:Replying to "So the idea is to te..." So we want to do the tests on one snapshot instead of having a snapshot of mainnet and another one of the bloat network

Ameziane Hamlat:Replying to "So the idea is to te..." I’m saying this because I don’t see the link between BAL and Bloat network

nixo:isn’t that… acdt

jochem-brouwer:Antwoord verzenden naar "isn’t that… acdt" acd-topic call? :)

Dragan Rakita:Can we do async reporting?

felipe:acd-TBD

Potuz:yeah TBD is a good topic :)

Justin Traglia:I can help here too

Summary

24 highlights · 3 decisions · 2 action itemsExperimental

fusaka blob devnet 0

Engine API serialization bottleneck identified; SSZ transport proposal under review00:04:43
SSZ can be implemented optionally via content-type header routing00:06:10
Giulio implemented SSZ proof-of-concept for Prism/Lodester/Lighthouse/Teku/Aragon/Nethermind/Geth00:06:47

glamsterdam bal devnet 2

BAL DevNet2 running with prefetch/optimization benchmarking nodes added00:09:46
Aragon switching focus to DevNet3; Besu tracing issue under investigation00:11:06

glamsterdam bal devnet 3

EIP-8037 cross-client bug hunt ongoing; multiple consensus issues found00:11:42
Besu: 7 failing tests, 2 false positives; fixes in progress00:12:47
Marius: EIP-8037 implementation deeply complex; AI-generated code causing issues00:17:35
Ben/Nethermind: 8037 implementation has tendrils affecting everything; disliked from complexity standpoint00:19:57
Nethermind/Reth: parallel I/O not showing major benefits yet00:21:02

epbs devnet 0

ePBS forces EL to trigger block production on older heads; spec currently forbids00:24:49
Marius confirms horrible UX but accepts reorgs will happen; can prepare branch00:30:48
Marius pushed fix allowing reorging to parents on epbs-devnet-0 branch00:37:45
PTC lookahead issue: attestations valid in epoch N invalid in N+100:38:16
Proposed fix: cache 256KB of PTC committees in beacon state00:39:31
Decision on PTC fix needed this week before testing attestations00:41:24

testing infrastructure

Execution specs v5.3.0 released: expanded EIP-8037 coverage plus all Osaka tests00:14:26
Perf DevNet3: 50GB ERC20 complete, EOA bloating in progress00:50:46
Geth synced on perf-devnet-3; Reth/Nethermind/Besu sync issues being resolved00:52:49
5 Gigagas testing revealing out-of-memory issues in some clients00:54:33

networking proposals

snap/2 draft: BAL-based state healing using block access lists00:48:19
eth/71 response size cap clarified: 2MB soft cap, not 10MB00:49:22

organizational

ACDT cadence discussion: proposal to rotate topics or convert to breakouts00:56:33
Proposal: Reserve ACDT slot for focused technical breakouts on-demand01:00:01

Decisions

SSZ engine API will be optional via content-type headers; deferred to ACD for final approval00:07:21
EL must support building on older heads for ePBS; spec will be updated00:31:05
ACDT cadence will be evaluated weekly; cede to breakouts when no cross-cutting topics01:00:49

Action Items

CL teams / Justin Traglia: Make decision on PTC committee caching approach this week00:41:20
Parithosh / DevOps: Test withdrawals, deposits, blobs on ePBS DevNet0 locally before live testing00:44:25

Targets

BAL DevNet3 launch: targeting March 11, pending client readiness00:17:31