Parithosh Jayanthi:Okay, let's get started. First up is an update from the testing team. Is Dan already on the call?
Transcript
Parithosh Jayanthi:Yes. Dan, do you want to give the structure update? I can share my screen so that people have something to look at.
danceratopz:Yeah, hi, thanks, Perry. Yeah, it's just a quick heads up. I hope it doesn't affect anyone badly, but this is basically, a heads up for client teams or anyone that consumes
danceratopz:our test fixture releases.
danceratopz:that previously, when we filled a test, for example, for Cancun, we dumped all the fixtures filled for all forks.
danceratopz:and also all the subsequent forks into the same JSON file, which meant that these JSON files grew with every fork. And in order to stop that growth, we've now reorganized our
danceratopz:Fixture directory layout, or fixtures, so that, essentially, we only ever include, fixtures for one fork, or field targeting one fork.
danceratopz:into one JSON.
danceratopz:And to enable it, we've now added a top-level, fork directory, so for Prague, for Fusaka, in order to organize this.
danceratopz:And,
danceratopz:For the benchmarking tests, they're also additionally organized by gas limit or opcode count, whichever mode is used to fill them.
danceratopz:If anyone has any serious objections, then please do let us know. It was announced in EFAR&D, which is linked in today's ACDT issue.
danceratopz:Feel free to… Chime in now, or drop a comment on the,
danceratopz:on the thread in EF R&D.
danceratopz:Alright, thanks a lot.
Parithosh Jayanthi:Well, are there any questions, or anything for Dan?
Parithosh Jayanthi:Great. Then, if anyone has something, then please just, yeah, drop a message to them.
Parithosh Jayanthi:Let's move on to the next topic. We have, Fusaka Blob DevNet updates. Do you wanna go, Barnabas?
Barnabas:Yeah, sure. So, last week, Thursday, we discussed that,
Barnabas:We're going to be keeping the Blob DevNet around for all the CL teams to, join in, but for all the CL teams that are feeling ready to go, they can, make a hidden flag, or a flag that is disabled by default, and, push it to their,
Barnabas:stable releases, basically, and possibly included in the next,
Barnabas:Next releases, we're gonna be also testing it on, on a bigger network, and this is basically just to be able to see that everything is working as expected on a
Barnabas:A limited number of validators. And then, if everything's working, then,
Barnabas:We can do a bigger rollout.
Barnabas:The good thing is that Sepolia and Holesky are both,
Barnabas:Mostly controlled by us, and we were able to basically tweak these flags, enable or disabled, quite easily.
Barnabas:So there's no big, problem there.
Parithosh Jayanthi:Okay, are there any asks from the client teams, or…
Parithosh Jayanthi:Is everyone ready to go there?
Barnabas:I think everybody that, was on the call on Thursday agreed with this approach. We didn't spend a bunch of money on, trying to test this on a…
Barnabas:on a DevNet kind of way, but we all thought that this would be just a waste of money, and instead we should just be testing on Hoodie.
Barnabas:Which is basically as big as it gets, for that map.
Barnabas:I think we were missing Nimbus and, and Teku, to give us an update. I'm not sure if they would be ready to give us an update on this call, or maybe sometime on Thursday.
Barnabas:In the previous call?
Parithosh Jayanthi:Okay, then I guess we move on to Glamsterdam, if there's no other update there?
Parithosh Jayanthi:Yeah, okay. Stefan, do you want to give us an update on block-level, DevNet 2 and DevNet 3 readiness?
Stefan Starflinger:Yeah, sure. So, BioDevNet2 has been running.
Stefan Starflinger:quite some time now. We've got Erigon now, and Erigon.
Stefan Starflinger:And it seems that Erigon still needs to iron out some, small details, to be proposing.
Stefan Starflinger:And then we have still the benchmarks that we want to do, for DevNet2, images, branches. We want to see that,
Stefan Starflinger:Nethermind, and Erigon.
Stefan Starflinger:And EtherX as well, if they have it available, have flags, that we can configure. The bulb prefetching.
Stefan Starflinger:The sequential, and the… pretty much everything, all the optimizations.
Stefan Starflinger:That would be great, if maybe Nethermind could give us an update on Erigon.
Stefan Starflinger:On the flags for benchmarking. I think they're still missing.
Marc:Yeah, for… Nethermind. In my testing, I got the, parallel execution and, state root application working.
Marc:So I just need to do a bit more, testing of that, before we merge it, and then I'm focusing on the, battery implementation.
Stefan Starflinger:Okay, so you should have it done soon, is what I hear.
Marc:Yep.
Stefan Starflinger:Okay, so, from my side, I think…
Stefan Starflinger:Regarding the plans for Shadow Fork, when we have 3 clients, then I will, start,
Stefan Starflinger:Putting effort into that, for the benchmarking.
Stefan Starflinger:And otherwise, it would also be great to get an update, yeah?
Parithosh Jayanthi:Have we done stateful tests against this?
Parithosh Jayanthi:Or not yet.
Stefan Starflinger:Yeah, that was the next thing I would want to ask.
Stefan Starflinger:Kind of, if there's an update on the benchmarking, testing side.
jochem-brouwer:Yeah, I can give an intake. So, one of the blockers was that if we want to run stateful tests for block developer access lists, we need to have a client which supports the testing endpoint
jochem-brouwer:in Glamsterdam.
jochem-brouwer:and Besu supports this. I have not yet looked into this, but I think
jochem-brouwer:Or no, I don't think. Like, I know that we can now start to integrate these benchmarks. So that means that we can run the block of access list benchmarks on top of a state, so, like, a maintenance shutter fork, or, like, Holesky.
jochem-brouwer:And… Let's see, it's Commune…
jochem-brouwer:Oh, wait, no, he's not here today. But what we will do is, yeah, I'm…
jochem-brouwer:The main point is that we can now start to run these tests at, I was talking. Yep.
jochem-brouwer:And maybe, maybe, maybe to, to, to just add to that, this…
jochem-brouwer:What we will do is, we will run these tests on the clients with different configurations, so we have these optimizations. I think the most
jochem-brouwer:important one now is the batch I.O.
jochem-brouwer:So we can actually see how much, impact this has.
jochem-brouwer:So what we will do is we run exactly the same client, but with a different configuration. The first has the batch I.O. enabled, and the other does not have the batch I.O. enabled, and then we will see how much impact this has on the runtime.
Toni Wahrstätter:Are you now… are you now talking about the running the batch. on the shadow fork, or on a local definite? Or on a local empty definite?
jochem-brouwer:No, not on an empty, like, so on a shadow fork, so mainnet or blog net.
Toni Wahrstätter:Perfect, yeah, great.
Parithosh Jayanthi:But when you guys are saying Shadow Fork, do you just mean the benchmarks?
Parithosh Jayanthi:tool, the benchmarker tool, or are you running a live network?
jochem-brouwer:So, these, stateful tests, what you do is you have, like, a normal test, but now you will, run these, or you will fill these on top of, like, different state.
jochem-brouwer:And, well, we have two state candidates. The first is, like, mainnet, like a snapshot. Maybe, like, you need to shadow focus is not a good name for this. It's actually, we take a snapshot from mainnet, or a snapshot from Holesky, and then we build the tests on top of that, and then we start to run them.
Parithosh Jayanthi:Got it, so it's running on the benchmark suite. Got it, yeah.
jochem-brouwer:Yes.
Parithosh Jayanthi:Okay, makes sense.
Parithosh Jayanthi:Yeah, anything else on the benchmark topic for block-level access too?
Stefan Starflinger:I think…
Toni Wahrstätter:I think regarding… sorry, go ahead. Stefan, do you want to add something?
Stefan Starflinger:No, no, you go ahead.
Toni Wahrstätter:I was just, saying, like, for the execution parallelization, we can basically do it on DEFNET2 already. I think all clients are ready on that front.
Toni Wahrstätter:And we don't need state for it.
Toni Wahrstätter:So, if we want to already benchmark, Block Lab Access List itself.
Toni Wahrstätter:Just with regards to execution, then we should be ready to go.
Parithosh Jayanthi:Yeah, that makes sense. The only sort of thing you should be careful about is the machines we're using for…
Parithosh Jayanthi:DevNet2 should be Hetzna, I'm not sure if we have dedicated CPU on them or not, just in case this isn't funky.
Stefan Starflinger:PR ready, that we could add dedicated instances to DevNet2, and that's something we can consider doing.
Stefan Starflinger:That we let them run dedicated instances maybe for a day or two. I don't think they need validators, they don't need to propose anything, and then we can capture that information.
Stefan Starflinger:Running for one or two days.
Parithosh Jayanthi:Yep, that sounds perfect, yeah.
Stefan Starflinger:But otherwise, I think, from my side, that's it for DevNet 2.
Parithosh Jayanthi:Okay.
Maria Silva:Cool. For Devin… Can I just…
Parithosh Jayanthi:Maria?
Maria Silva:Yeah, just a quick question. So, I'm curious, so we could be running all the… also the repricing
Maria Silva:benchmarks for compute, right? Where we, at least for,
Maria Silva:for the parallel execution, we have a sense of the difference between the values we are seeing now and the values we are seeing after balls. Will that data be stored somewhere that I could vary and check?
Parithosh Jayanthi:I think there's two layers of benchmarking happening. So one is on the live DevNet, and on the live DevNet, you won't be able to change the opcode pricing. But on the other side, what Joachim was talking about earlier is just on Benchmarker. So that's the same infrastructure you're using for repricing testing.
Parithosh Jayanthi:So in that one, as long as clients provide us an image with the updated opcord pricing, then we can just read on the benchmarks, and the data will be in the same place you're testing all the other, repricing data.
Maria Silva:Right, and just to say, for this one, I wouldn't need to change upload pricing, it's just,
Maria Silva:What is the runtime for given tests?
Parithosh Jayanthi:Yep.
Parithosh Jayanthi:Okay, the… looking at the spec sheet for block-level access to Stemnet 3, there seems to be a couple of open PRs.
Parithosh Jayanthi:The first one's on the execution APIs. Are there any blockers for this one? This adds the block-level accessless JSON RPC methods.
Parithosh Jayanthi:If not, then can we get this merged in this week?
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:I see that whatever review was required is done.
Parithosh Jayanthi:So we'll get merged in today.
Parithosh Jayanthi:The next question was, merging in the next API change, which is for max use gas field on each simulate V1.
Parithosh Jayanthi:No blockers here either?
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:In that case, we'll get that merged in today as well. And then we had a comment about text vector… test vector metadata consideration. Does someone from the testing team want to explain this one?
Mario Vega:I'm out of the loop here, Spencer, you commented on this issue, I had not seen it before.
Mario Vega:Do you have any idea what it is about?
spencer-tb:Yeah, sorry, I'm just, re-reading it now. So, as far as I understand,
spencer-tb:Basically, we want to add, within the fixture format, a metadata field.
spencer-tb:And basically… Use that field, to place Non-consensus-related, Items.
spencer-tb:into that… into this new metadata field, so I guess,
spencer-tb:Yeah, the question is if, people are okay with us doing that, because I think if we do that, we could break some of the…
spencer-tb:ways EL clients are parsing the fixtures.
spencer-tb:Yeah.
spencer-tb:I think it's something we should do, Carson.
spencer-tb:It's a… it's a nice way to clean up our fixture format.
Parithosh Jayanthi:So the main objection would be from client teams who are consuming the fixtures, right?
Mario Vega:Exactly, I think it might break their consumers, but I don't think it should.
Mario Vega:Basically, the question is, adding extra fields to the block test?
Mario Vega:format, is it a problem for consumers of, in the clients? That's basically the only remaining question, I think. We can do it really easily.
Parithosh Jayanthi:I don't think it's a problem, so never mind.
spencer-tb:I guess just to add more context, the…
spencer-tb:the extra kind of metadata field we'd want to add is, the block access list.
spencer-tb:So I think,
spencer-tb:I think with this change, we'd have a metadata field within the… each block of the fixture, and then the metadata field would contain the RLP, the block number, and the block access list. So I think that's the full change, now that I've ever read it.
Parithosh Jayanthi:Okay, I… Yes, we'll do this async as well, and then consider it shipped, yeah?
Parithosh Jayanthi:We can ask again on the Steel Discord server. If there's no objection, then I'd say this week we make the change.
Parithosh Jayanthi:Okay, that sounds good. The next open point was, EATH71 and inclusion on block-level access list DevNet 3.
Parithosh Jayanthi:Does someone have a bit more context on this? Does this require each 70 to first be shipped, and then 71 to go on top, or how's the sequencing here, and is it optional or not?
Parithosh Jayanthi:Or maybe an earlier question, has any client implemented each 71?
Parithosh Jayanthi:Okay, I'll take that as a sign that it's not going in this cabinet. Yeah, Tony?
Toni Wahrstätter:Yeah, I just recall that Felix was saying a few weeks ago that, of course, the proper way would be to first, roll out EVE70 and then 71. I think for DevNet 3, it's not a blocker, though, if we don't have EV71 yet.
Parithosh Jayanthi:Okay, sounds good. So we will just make sure it's not on the spec sheet, and yeah, if it's there in any advantage, it seems it's optional.
Parithosh Jayanthi:Anything else for DevNet 3? What are the status updates on clients? What's the timeline looking like?
Parithosh Jayanthi:Understandable, client by client, I guess. Someone from GET?
Parithosh Jayanthi:I see Marcus in the chat.
Parithosh Jayanthi:Or, yeah, never mind?
Łukasz Rozmej:Yeah, so… I think we're a bit down yet on the…
Łukasz Rozmej:State creation gas costs increased. We'll hope we'll get it done by, Wednesday. And we… I think we also need to do the increased maximum contract size, but that is very simple, I will do it today.
Łukasz Rozmej:Yeah, but that state creation gas cost increase might, slow us down a bit.
Parithosh Jayanthi:Okay, Marius says, forget they were released today, and then they should be able to merge some stuff in. Is anyone from Bezel around?
Parithosh Jayanthi:Yeah. Yeah.
Daniel Lehrner (Besu):Yeah, I can give an update. So we have everything in the Bar Defnet 3 branch, except for 8037. AT37, we have a PR.
Daniel Lehrner (Besu):And today, in the morning, I made the change with the fixed, cost per state byte that is required for Bar De Fnet 3.
Daniel Lehrner (Besu):So we need to review it, merge it.
Daniel Lehrner (Besu):And I did not run yet the new execution spec test, I need to run those as well, so there might still be bugs.
Daniel Lehrner (Besu):But I think… I think we're good to go for Wednesday.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:That's awesome. Erigon?
Parithosh Jayanthi:Okay, I've known from Eragon, then, do we have anyone from RET?
Jen:Yes, we should be good for DevNetly this week.
Jen:We're just missing 8037.
Parithosh Jayanthi:Yeah.
Parithosh Jayanthi:I guess that means we still want to aim for DevNet 3 launch this week, right, Sukhan?
Stefan Starflinger:I will do some, or I will start doing some kurtosis testing, as soon as clients are ready.
Stefan Starflinger:And then we can, if everything goes well, we can aim for a launch this week, yeah.
Parithosh Jayanthi:Okay, sounds good.
Parithosh Jayanthi:Next one on the list was EPBS DevNet Zero. Barnabas, or Stefan, or PK, do you want to give an update?
Barnabas:We are looking to get an update from the client teams first, and then, hopefully we can schedule it in for Wednesday as well.
Parithosh Jayanthi:Okay, yeah, do we want to start with CL teams? How's it looking?
Barnabas:Maybe you can also call out the teams specifically.
Parithosh Jayanthi:Yeah, sure. Prism, someone from Prism want to speak up?
Potuz:This is an EPVS.
Parithosh Jayanthi:Yes. The…
Potuz:We are passing Kurtosis test, we're finalizing
Potuz:We seem to be robust.
Potuz:We really can't test against other clients because, well, we try it with Lighthouse, and we break immediately, because Lighthouse cannot handle when the payload arrives before the block.
Potuz:And I think a DevNet without this is just going to break quickly.
Potuz:We are working on sync. We cannot sync, I mean, we can only do forward sync, so if we fall behind, or if we need to restart, we're gonna be broken. We hope to have this, like, either today or tomorrow, ready, at least on a DevNet.
Potuz:And, I believe other clients are more or less in the same situation with regards to sync. Otherwise, I think we're fine. We… we are finalizing in a robust manner. We can handle payload reorgs, we can handle reorgs. I'm testing against an evil node to see if we can… if we can handle, like, more strict situations, and…
Potuz:I'll have those tests today, I suspect we're fine.
Parithosh Jayanthi:Awesome. Is your branch already up to date, or are you working off a different branch?
Parithosh Jayanthi:PBS DevNet Zero branch.
Potuz:It is… there is the EPBS DevNet Zero branch. Terrence is going to push changes towards sync. Sync changes are on a different branch, and EPBS DevNet Zero is essentially the same as developed today, so it shouldn't be too far to… to actually merge this on the main branch.
Potuz:That's why we're kind of late, because we were trying to push everything to develop.
Potuz:Yeah, makes sense.
Potuz:It's fine to test, but as I said, sync is important, and we're currently broken on sync.
Parithosh Jayanthi:Yeah, no, I was mainly asking because we also have our malicious prism node, so we wanted to rebase on top of yours, and then we can have kurtosis tests for, delayed blocks or withholding stuff, etc. So, we'll probably do that over the next day or two and get back to you with a few more scenarios.
Parithosh Jayanthi:Yeah, and to maybe Tony's question, should the payload ever arrive before the block?
Potuz:Yeah, so this is actually… so this happens a lot on DevNets, because… especially because we're doing self-building, and we're just broadcasting everything together. Lodestar and Lighthouse didn't break before, because Lighthouse adds a delay before… I mean, it forces the payload to go later.
Potuz:But in real life scenario, this actually may happen, especially if your payload is very low, and people that are co-located with the builders, they might actually get the payload earlier than the… than the block.
Parithosh Jayanthi:Okay, yeah, Barnabis?
Barnabas:Yeah, so ideally, EPBS net zero should be as stable as possible. I don't think we should do malicious.
Barnabas:Testing, at least not in the first week or two.
Barnabas:What I want to do is have an interrupt network that is very stable, doesn't require, clients to be able to sync, because everybody's gonna be joined in at Genesis.
Barnabas:And, then this is going to give us a nice base to work at, and then clients can start fixing their sync test, and their syncing issues, and then they can, roll out as they get ready.
Barnabas:And then, maybe in definite one, we can start testing, edge cases.
Parithosh Jayanthi:Yeah, that sounds reasonable to me.
Parithosh Jayanthi:Do we still want to do updates from the other CLs as well? Taeku, do you want to go next?
Stefan Bratanov:Yeah, sure, for Deco… We're in kind of a similar situation with an IVLC.
Stefan Bratanov:Where… I'm running some tests against Lightfall, so…
Stefan Bratanov:There seems to be an issue when we send the payload very fast.
Stefan Bratanov:But I'm gonna run a bit more tests.
Stefan Bratanov:See… See if everything else is well.
Stefan Bratanov:Apart from that, we're also working on storing the finalized payloads.
Stefan Bratanov:So right now, our… implementation doesn't have that, so we don't serve finalized payloads over our PC.
Stefan Bratanov:Yeah. That's pretty much it, Jake.
Parithosh Jayanthi:Cool. Anyone from Lighthouse?
Parithosh Jayanthi:We might not have anyone… How about Lodestar?
nflaig:So we are mostly reviewing code right now, and getting stuff merged to our main branch.
Parithosh Jayanthi:Okay, cool, so then we should be able to at least attempt, Lodestar, TechU, and Prism interop, right?
Potuz:I think Lighthouse is also… should be more or less ready to…
Potuz:I mean, there's this issue of the payload, but I think they already had a queue that they were working on.
Potuz:I think we should be fine for two tests this week.
Parithosh Jayanthi:Okay, cool.
Parithosh Jayanthi:And Grandine.
Saulius Grigaitis | Grandine:Yeah, so… so we are still working on the franchise, so I think,
Saulius Grigaitis | Grandine:We will need maybe a couple of weeks more until we can try to interrupt with other clients.
Parithosh Jayanthi:Yeah, and, do we have someone from Nimbus?
Caleb:Yes.
Caleb:So…
Caleb:In Nimbus, we're testing with, like, a Nimbus-only devnet, like on Cutosis, we are pretty good, finalizing, pretty fine, like.
Caleb:We work well. We have a networking issue where we try to interrupt, so we work pretty well for a while, then…
Caleb:kind of downscore pairs after a while, then kind of breaks the change, so we're working on a fix for that. We hope to push a fix by today. Apart from that, we are good.
Caleb:like… We're currently… so we are thinking of, like, disabling the…
Caleb:scoring of peers and all, because after a while, we lose peers. That's the only issue we have at the moment, but…
Caleb:Hopefully, today, tomorrow, we… we are working on a fix already, so… It's so much.
Caleb:That's it.
Parithosh Jayanthi:Thank you.
Parithosh Jayanthi:So that sounds like we should be good to go later this week for a DevNet, like how Barnabas was describing, and if we're able to get you a sync test script for ketosis, I know that Prism has one, but probably
Parithosh Jayanthi:all the teams want to test that, then we should be good to go for this week's testing on EPBS.
Parithosh Jayanthi:Is there anything else that's worth immediately tackling?
Parithosh Jayanthi:Okay, sounds good. Thank you.
Parithosh Jayanthi:Yeah, Portos?
Potuz:I just want to reply to Stefan. I think we agreed that we shouldn't be broken by Baylor attestations.
Potuz:What kind of worries me is that, clients might have bugs and downscore peers, if they are not broadcasting them.
Potuz:So we should probably check on this.
Potuz:But other than that, I don't see why there should be a problem in, like, not
Potuz:In, in, in packing and sending, BTC at stations.
Potuz:We were hoping, actually, to add them as well. We have this old branch on… when we first interrupted with TechWh a couple of years ago, and I think that that hasn't really changed much, so it should be easy to add again.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:Anything else for EPBS?
Justin Traglia:Nothing spec-wise, but do let me know if there's anything we can do to help you, the clients.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:Then the next topic we had was, gas limit. Do we have…
Parithosh Jayanthi:Any updates on EAT70 implementations on plans?
Parithosh Jayanthi:Or any blockers, rather.
Fabio Di Fabio:Just to confirm that for Bethood, 8870 is ready?
Fabio Di Fabio:We need just to merge it on main, but it's ready for… Also, the DevNet.
Parithosh Jayanthi:Awesome. And yeah, Jen also posted that RET is ready for each 70.
Parithosh Jayanthi:Yeah, if that's the case, then I think we can make sure that the DevNet has both of those running.
Parithosh Jayanthi:Do we, at our parables of PK, do we have explicit tests for each 70 already?
Barnabas:Yeah. I think the idea was that, clients would need to sync to the chain, and if they can, then that's basically saying that they're able to handle 870, because we already introduced some, big blocks in the network.
Barnabas:Which would basically stop, some, 869 clients, do you think?
Barnabas:Fabio, can you please let me know when it's all in master?
Barnabas:I mean, did I get a ride out?
Fabio Di Fabio:Yes, on, which network are you… We're talking about which network?
Barnabas:So we are testing this on NFT devot them. I can give you a link for that.
Fabio Di Fabio:Okay, battery is already running, though.
Barnabas:Okay.
Fabio Di Fabio:Yeah, it is not running with the main branch, but it's running out of the development branch.
Fabio Di Fabio:But he's there since… One week, at least.
Parithosh Jayanthi:Okay, cool. Next topic on the agenda was, state float. So, do we have updates on Puff.net 3?
CPerezz:Sorry, I couldn't find the mic to unmute. I'm deploying the latest, scenario that we will ship, the next snapshot with, which is the depth, depth-based worst cases, so it will be done today.
CPerezz:After that, we will be sitting at about 1.5 slash 2TB of state, and with that, we will… we will extract all the snapshots, and probably the next stop will be at 4TB.
CPerezz:Anyways, with these numbers, the idea is to start running all of the analysis, because there's no more scenarios now to deploy it, and there's no more foreseeable worst cases that we can
CPerezz:See, or come up with, for state.
CPerezz:So, yeah, besides that, 4TB and see how we can handle that state size is likely the last thing. One thing that I wanted to ask is, all the teams have already either merged or have the PRs ready for the slow block tracing.
CPerezz:I have to review the one from Besu, I'm aware of it, but from Nethermind, it's the only client that, the PR is sitting there, and I would like to either understand if I should continue to work on it, or they will take over, such that we can unblock this, because for the next round of benchmarks, we…
CPerezz:We really need this tool to be able to check across clients all the metrics.
CPerezz:Maybe Ben, or Lukash?
Łukasz Rozmej:Well, sorry, I wasn't keeping up with the state load that Camille should be.
CPerezz:I mean…
Łukasz Rozmej:You both revealed the PR, not Camille, so that's why I'm.
Ben Adams:Oh, you mean… you mean the, Large block output.
CPerezz:Yeah, the one that shows… let me search the PR once again, it will be easier.
Ben Adams:Yeah, I'm working on, reworking our internal metrics to build it better.
CPerezz:Okay, so you're working on it. Okay, that's… that's perfect then. If that's the status, then there's nothing… nothing blocking and nothing pending, so we should just take the snapshots in, and then start running the whole… the whole benchmarking, which will probably start today, or tomorrow. Yeah, thank you.
jochem-brouwer:Yeah, full day.
Parithosh Jayanthi:But we had a couple of issues with nodes synced on First DevNet 3, right? Like, we were blocked on doing snapshots because the nodes weren't healthy. Could we get an update on that?
CPerezz:I think Nethermind is the only one that is having issues at this point, though I could be wrong, but this is what I've seen today in the chat. But Camille doesn't seem to be here, so I'm not sure if we will get any updates.
Parithosh Jayanthi:Okay, but for the Netherland issue, I think we were investigating the physical host now, so Barnabis has put in a request to, get a mem test done on the host, because Camille asked us to verify that the host is actually in the
Parithosh Jayanthi:But, Erigon and Wrath are fine otherwise, because I remember they were still sinking last week.
CPerezz:Yeah, I mean, REST will take… will take a lot of time to sync, because it needs to re-execute absolutely everything, so I mean… so I will check the logs now, but I would… I would expect to… to take a couple more days even, maybe, to… to finalize.
pk910:The last time I checked Wrath, it was in a crash loop because it was trying to, hash the storages and was, running into, out of memory that way.
pk910:So, I've pinged, Rafim, and, Champa was,
pk910:Saying he was looking into it, but I'm not sure what's the status of that right now.
Parithosh Jayanthi:Okay, yeah, maybe, Jen, can you follow up with the Red's team and let us know what's going on there?
Jen:Yep, will do.
Jen:That was the thing on the State Bloods Discord, right?
Parithosh Jayanthi:Yes.
Jen:Yeah, Mojeko.
Parithosh Jayanthi:Okay, perfect. Yeah, and Joachim, you had a, agenda item today as well. Do you want to discuss, state at all?
jochem-brouwer:Yes, first I had one more question about RoadNet, because for the repricings, we actually need this snapshot.
jochem-brouwer:So, my question will be… Mmm… That's… yeah.
jochem-brouwer:I don't know,
jochem-brouwer:Because, Carlos, we have all this stuff, but you wanted to get in, right? Wait, the problem is that the notes are not synced.
jochem-brouwer:Yeah, okay.
Parithosh Jayanthi:Exactly, we need at least all the nodes to be healthy before we can take the snapshots.
jochem-brouwer:Yeah, yeah, that makes a lot of sense, yeah.
jochem-brouwer:Yeah, it's just that for the repricings, so for the analysis of the repricing, so for 8038, we need these snapshots to run them on, because they have to state what we actually want to test on.
jochem-brouwer:And… Don't need that. I just wanted to share that.
jochem-brouwer:Okay, then I will move on to my other topic. So…
jochem-brouwer:This is about state actor, and State Actor is a two…
jochem-brouwer:what Tony, Carlos, and Camille, are creating. And this… the end goal, or, well, the thing that it tries to solve is that we want to, to give clients away, to start with a very big state,
jochem-brouwer:well, right from block zero, or from whatever, but to start the client with this very big state.
jochem-brouwer:So you can do that with a Genesis OGs, because this felt, like, way too big.
jochem-brouwer:The idea is that we create, like, a, configuration.
jochem-brouwer:Where we start to create a state. So you also don't have to download, like, one or two terabytes of blocks. And the nice property is also that you can actually say, like, if you need, like, a state and date, you can just say, okay, yeah, I want some more contacts here, I want some more storage there, et cetera, et cetera.
jochem-brouwer:But I think that the realization is, or at least my realization, is that… well, that might seem a little bit stupid, but that every client has, like, their own database layout.
jochem-brouwer:And this is, this is our challenge.
jochem-brouwer:And bots…
jochem-brouwer:we are mainly wondering about is… I think that, like, for state research, this would be, like, very helpful.
jochem-brouwer:And I, yeah, it would be nice, like, if there are maybe clients also, like, interested in this, or, like, can give us, like, a point, like, okay, watch out, like, you can do this, or can, you should watch out for that.
jochem-brouwer:Because we have to initialize the client with this big state, and we likely have to go, like, in the inner methods of these clients to actually start to route dump this stuff in the database.
jochem-brouwer:I hope all of those also appear, yeah.
Parithosh Jayanthi:Yeah, Tony?
Toni Wahrstätter:Maybe just to add to that, so the idea was basically to bloat the state outside of the network, basically being only bound by physics, what your SSD can handle, and then start a network from it, because then, of course, you can bloat very fast.
Toni Wahrstätter:One thing you can't do afterwards is replaying mainnet transactions against this type of state, because, of course, it's very random.
Toni Wahrstätter:So it's very useful for benchmarking, and so far, I think we have GAF and Erigon implemented, so basically reusing the code of GAF and Erigon to create the state. We have to look into other clients yet, how it differs there.
Toni Wahrstätter:But, yeah.
Toni Wahrstätter:basically creating a very flexible way to create a network. It's also very flexible in the depth of the tri, so you can create very deep branches.
Toni Wahrstätter:Very, fast, and this is the goal of it.
Parithosh Jayanthi:Okay, so this will be, like, a parallel workstream alongside BloatNet and keeping BloatNet running, right?
jochem-brouwer:I would too.
Toni Wahrstätter:Yep.
jochem-brouwer:Yeah, I would think that that's how it would work in my mind, is that BloatNet is also used, like, for the networking part of this.
jochem-brouwer:And this, this, like, this random state doom is… well, it's just focused, like, on big state. So, like, run tests on that directly, and then if there's something wrong with the state, edit it and do it again.
Parithosh Jayanthi:Cool. Yeah, that sounds like a very nice effort. Any questions or comments?
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:I think that was everything on the agenda. Anyone has any open topics they want to discuss?
Parithosh Jayanthi:Cool. And thank you all for coming, and see you next week.
jochem-brouwer:Cheers, I bet.
Toni Wahrstätter:Thank you, bye-bye.
Chat Logs
00:05:13
Ben Adams:ssz tests :)
00:14:35
jochem-brouwer:Can also do it on empty state for pure compute ones
00:16:32
jochem-brouwer:Benchmarkoor uses a snapshot so its not a live network. We can fill on Osaka and Glamsterdam
00:16:50
Parithosh Jayanthi:https://github.com/ethereum/execution-apis/pull/726
00:16:53
CPerezz:Replying to "Benchmarkoor uses a ..."
Except for execute-remote dependent tests
00:17:13
Barnabas:today *
00:17:30
Parithosh Jayanthi:https://github.com/ethereum/execution-apis/pull/746
00:17:55
Parithosh Jayanthi:https://github.com/ethereum/execution-specs/issues/1938
00:18:59
jochem-brouwer:Antwoord verzenden naar "Benchmarkoor uses ..."
This is a good point, I think it's there but I have to check that we can choose the fork to build these benchs on
00:20:02
jochem-brouwer:Antwoord verzenden naar "https://github.com..."
broken url :(
00:20:12
Parithosh Jayanthi:Replying to "https://github.com/e..."
fixed
00:20:15
Parithosh Jayanthi:Replying to "https://github.com/e..."
Trailing )
00:20:50
spencer-tb:Ship it?
00:21:52
Marius:we haven't yet
00:21:55
Barnabas:very quiet audience 😂
00:22:15
Marius:Its optional, so its not a block
00:22:29
Barnabas:I listed on the spec sheet as optional
00:23:00
Marius:sorry again?
00:23:05
Parithosh Jayanthi:Wen bal-devnet-3
00:23:07
wolovim:Replying to "I listed on the spec..."
https://notes.ethereum.org/@ethpandaops/bal-devnet-3
00:23:25
Marius:We are working on a release today, afterwards we will be able to merge some stuff
00:23:37
Marius:And then create teh devnet-3 branch
00:26:12
Mario Vega:Replying to "Wen bal-devnet-3"
EELS test release: https://github.com/ethereum/execution-spec-tests/releases/tag/bal@v5.2.0
00:26:33
Barnabas:Potuz: “we seem to be robust”
Potuz: “we cannot sync”
00:27:23
felix (eest):Replying to "Potuz: “we seem t..."
this is fine
00:27:23
Toni Wahrstätter:Should the payload ever arrive before the block? I guess we wouldn't expect this to happen in the happy path?
00:28:57
Stefan Bratanov:we (Teku) have a delay of 500 ms after the block when we publish self built payload
00:30:41
Potuz:@Stefan Bratanov what’s the branch? You can test against Prysm, it should handle early payloads
00:30:48
Potuz:I can test against you guys if you do handle them
00:31:05
Barnabas:teku is master
00:31:29
Stefan Bratanov:I need to merge one PR, then we can test
00:33:14
Stefan Bratanov:by the way we send and pack payload attestations in blocks, that's not issue for other clients right. they will be ignored for devnet-0? Just want to make sure it doesn't break clients
00:33:49
nflaig:Replying to "I can test against y..."
lodestar should be able to handle early payload
00:34:37
Stefan Bratanov:kk thanks, I can turn it off easily if it causes issues on the gossip side when we run the devnets
00:35:01
Potuz:Replying to "I can test against y..."
Ah nice, what is the image I should use in Kurtosis to test against you guys?
00:35:22
nflaig:Replying to "I can test against y..."
ethpandops/lodestar:epbs-devnet-0
00:35:31
Jen:Reth is ready w/ eth70
00:35:37
Potuz:Replying to "I can test against y..."
Thanks will test today after I cleanup my open PRs
00:35:44
Barnabas:we can include besu and reth today
00:36:52
Parithosh Jayanthi:nft-devnet-10
00:38:15
Barnabas:nethermind’s host is still being investigated for now
00:38:19
Barnabas:Can we hold off another day?
00:39:39
CPerezz:https://github.com/NethermindEth/nethermind/pull/10288
00:41:03
Barnabas:are we syncing reth in full or archive mode?
00:41:19
Jen:Can you guys share the logs with us?
00:41:29
CPerezz:Did Jenn not finish with it?
00:41:29
Barnabas:you should have access to the servers
00:41:36
CPerezz:Kk MB.
00:41:43
CPerezz:Ty @pk910
00:43:23
CPerezz:https://github.com/nerolation/state-actor
LINK TO THE TOOL
00:43:27
CPerezz:FYI
00:43:30
Ameziane Hamlat:What is the status of besu nodes on bloat network, I see that only geth is proposing blocks ? I’m trying to sync again from my side as a besu follower node but how about validator nodes ?
00:44:47
CPerezz:Replying to "What is the status o..."
Will check now. Still, you should have access. All IPs to ssh are here: https://github.com/ethpandaops/perf-devnets/blob/master/ansible/inventories/devnet-3/inventory.ini
00:46:46
CPerezz:Bloatnet covers main net compatibility, syncing testing
00:46:53
CPerezz:Rest can mostly be done with this tool
00:47:09
CPerezz:Like Geth’s Bintrie
Summary
22 highlights
· 5 decisions · 3 action itemsExperimental
Summary
22 highlights · 5 decisions · 3 action itemsExperimentaltesting infrastructure
fusaka updates
glamsterdam bal devnet 2
glamsterdam bal devnet 3
epbs devnet 0
- Prysm passing Kurtosis, finalizing robustly; sync support by today/tomorrow00:25:56
- Lighthouse breaks when payload arrives before block; needs queuing fix00:27:30
- Teku similar to Prysm status; finalized payload storage work ongoing00:29:51
- Lodestar reviewing code for main branch merge00:30:51
- Grandine needs couple more weeks before client interop readiness00:31:36
- Nimbus finalizing well; fixing peer scoring/downscoring networking issue00:32:30
gas limit eth70
- Besu eth/70 ready; Reth eth/70 ready; both can join nft-devnet-1000:35:27
state bloat
- perf-devnet-3 deploying depth-based worst cases today; reaching 1.5-2TB state00:37:44
- Next stop 4TB; no more foreseeable worst-case scenarios after this00:37:57
- Slow block tracing PRs ready across clients; Nethermind reworking internal metrics00:38:31
- Nethermind syncing issues under investigation; Reth in crash loop on storage hashing00:40:12
- State Actor tool creates big state from block zero without Genesis downloads00:44:33
Decisions
- Skip expensive Blob DevNet; test Fusaka features on Holesky/Sepolia instead00:07:39
- Use dedicated CPU instances for 1-2 days on bal-devnet-2 for benchmarking00:12:29
- EPBS DevNet 0: stable network without sync/malicious testing initially00:28:56
- Payload attestations will be ignored in EPBS DevNet 0; not breaking clients00:33:49
- State Actor tool runs parallel to BloatNet for different testing purposes00:46:22