Mario Vega:Hi everyone, welcome to ACDT number 61. Today's November 10th.
Transcript
Mario Vega:we have, the same, basically the same items of the agenda as every week, and I think we should just get started.
Mario Vega:Yeah, starting with Fusaka, we had BP01 activation last week.
Mario Vega:So, the first topic I wanted to discuss is whether there were any events that were significant enough that we need to discuss this week.
Mario Vega:Barnabas, do you know if we have,
Mario Vega:If it was a good transition?
Barnabas:We had no problems on the BPO2, but we have discovered a bug in NetherMind. Maybe someone from the NetherMind team can reach out and discuss what has happened.
parithosh:Do you mean a bug relating with BPO2, or the release bug?
Barnabas:the release bug. Sorry, regarding BPO2, I think we were good.
Barnabas:BP01, BPO2 is the trick.
Marcin Sobczak:Yeah, so…
Barnabas:So…
Marcin Sobczak:Yeah, we had, we added some bug fixes on top of,
Marcin Sobczak:the version 1.35.0. We released 0.1 with bug fixes, but these bug fixes introduced even worse bugs, so we then reverted to… we released 0.2 version, which is on top of 0.0.
Marcin Sobczak:And we already, solved this… the bugs, which were in
Marcin Sobczak:version 0.1, but it was, like, under-tested. We just released it,
Marcin Sobczak:It was under-tested, and, we are, we are, going to, to do a release after mainnet fork with, with, some known optimization and fixes.
Marcin Sobczak:And right now, we go with version, .2, which, we consider as, good enough, and,
Marcin Sobczak:And, more stable, and at this point, the stability is more important than having fixes of very rare bugs, etc.
Marcin Sobczak:And, yeah, we introduced, we had a broken snap sync, because we introduced, regression in downloading block receipts.
Marcin Sobczak:And, other, code, like a consensus bug, but, yeah, it's,
Marcin Sobczak:We are going with version 2.2 now.
Mario Vega:Alright, thank you.
Mario Vega:Also, Barry mentions that there was also a red, bug. Is someone from Red in the call that would like to chime in?
Mario Vega:Alright, I think there's no one from Red. Do you guys know, Barry or Venomas, do you know the context on this?
parithosh:Yeah, I think the TLDR is they merged some changes that were supposed to be meant for block-level access lists, but, the changes ended up affecting the mainnet release before the fork.
parithosh:But I don't have any more details on the topic.
Mario Vega:Alright, thank you. Yeah, I think, I think it's basically the same thing, right? From Nevermind.
Mario Vega:It would be nice if I…
Mario Vega:if we can reach out to Reth at some point to see if they're planning on making another release, just to make sure that the blog post is updated, I can… I can… I can follow up with them.
Mario Vega:Cool.
Mario Vega:Anything else from… the HUDI VPO activation, or the mainnet, client release.
parithosh:Bill.
Mario Vega:Not too sure.
parithosh:There's one more that Sam, found out, which was, Lighthouse, it's a non-consensus critical issue, but, Lighthouse is downloading the blobs from the checkpoint sync provider, rather than fetching from peer-to-peer. So, the team has mentioned that they'll fix it in a subsequent release, but it's non-consensus critical, so it's fine.
Mario Vega:Alright, alright, thanks. Enrico?
Enrico Del Fante (tbenr):Just one thing that I noticed, 15… kind of 15 minutes ago,
Enrico Del Fante (tbenr):Our nodes are detecting some columns by route request via PC on mainnet at the moment. It's kind of curious to see.
Enrico Del Fante (tbenr):I don't know if anyone else noticed those, RPC calls at the moment.
Enrico Del Fante (tbenr):Yeah, nothing important, but, just, if anyone adds…
Enrico Del Fante (tbenr):It's just a curious thing, maybe some… there is some client, buggy client, doing this RPC request.
Enrico Del Fante (tbenr):Beforehand.
Mario Vega:Do you have more info on where these requests are coming from, or…
Enrico Del Fante (tbenr):It's really…
Enrico Del Fante (tbenr):10, 15 minutes ago, I haven't dig into that, at the moment, but I don't have the peer, peer ID at the moment. I was just seeing these requests coming, coming…
Enrico Del Fante (tbenr):Yeah, just a curiosity.
Mario Vega:Alright, definitely needs to be investigated. Anyone else has… Seen this?
Mario Vega:Okay, no, I think it's not recorded by anybody else,
Mario Vega:But I think we should follow up on this.
Enrico Del Fante (tbenr):I will, write a message somewhere in Telegram or Discord, and maybe run… A thread on it.
Mario Vega:Alright, thank you.
Mario Vega:Yeah, and moving on, the next,
Mario Vega:There's a comment in the chat that there was a halt.
Mario Vega:From Nimbus on Fusaka 113.
Barnabas:Yeah. I noticed that on, pretty much all Nimbus super nodes, the CL just stopped.
Barnabas:It is not crashing, it just stops sending anything, stops receiving anything, stops logging anything, and this is with the latest minute release.
Barnabas:So it's definitely something that they need to investigate.
Mario Vega:Alright, anyone from Nimbus in the call that,
Mario Vega:Just to make sure if you are aware of this issue.
Mario Vega:I cannot see anyone from anyone's in the call.
Mario Vega:Can… can we follow up with this on… with them on… with the Inimus team?
Mario Vega:Via message or some… or something?
Barnabas:Oh, we are already talking with them.
Mario Vega:Alright, okay, thanks. And just to say on that topic, is there any… anything else that's happening on… of DevNet, on the 3?
Mario Vega:Currently.
Barnabas:Yeah, a bunch of nodes, ran out of disk. I did some maintenance today, I cleaned up some databases, I reduced the number of blobs, so hopefully, we're not gonna run into the same issue, because the blob, database size was over 50030GB.
Barnabas:So we were just spamming too many blobs. I reduced it now, significantly, so hopefully…
Barnabas:We're not gonna have machines running out of this for the next month. And we… we should consider shutting it down, once, mainnet is worked.
Mario Vega:Okay. Okay, thanks. Does it mean that…
Mario Vega:So we're spamming a lot of blobs in this network. Does it mean that we are not cleaning up correctly after the…
Mario Vega:The time period where they should be cleaning.
Barnabas:Not really, because, at 20 blobs per block, a 10-day period is…
Barnabas:It's supposed to be around 640 gigabytes, I think.
Barnabas:And because we were spamming 20 blobs a block, it's… yeah, all the nodes slowly started to run out of disk. All the super nodes. Pull nodes had no issues.
Barnabas:Of course.
Mario Vega:Alright.
Mario Vega:Thanks,
Mario Vega:Okay, anything else regarding,
Mario Vega:Alright, we have someone from Reth. Yand, do you mind,
Mario Vega:Sharing a little bit about the, the bug that you guys hunt.
Mario Vega:Last week?
Jen:Yeah, I just added the… sorry, can you guys hear me? I had also issues with my mic. I just shared… I just shared also the release notes, so the V191.
Jen:That should be the release for Pazaka, which includes a patch. This came in with, like, an EVM regression, as mentioned before, that was already introduced for preparation for polls. And yeah, we're taking measures to make sure we don't have any further search regressions. I've been talking to the testing team.
Jen:I'm on tests to include.
Mario Vega:Perfect, thank you. And I can see that the blog is already, updated for this, so we should be good there.
Jen:Thank you, Feng.
Mario Vega:Excellent, thank you.
Mario Vega:Alright. Anything else Fusaka-related? I think, the only remaining things are we have to investigate the… strange requests in mainnet.
Mario Vega:But anything else apart from that?
Barnabas:It would be good to hear from Prism team, whether they would have a stable release this week.
Barnabas:Because, right now they have a Treasury lease only for Milo.
Mario Vega:Anyone from prison that would care to shame in right now?
Mario Vega:Alright, I guess not.
Mario Vega:Could we follow up with them on AC… I mean, it's… is it too late to follow up with them on ACDC, or…
Mario Vega:What other avenue do we have on there?
Barnabas:Yeah, I already reached out to them, but I got no response, so… I'm not sure where everyone is from, personally.
Mario Vega:Alright, yeah, I think we have to maybe keep on trying to reach them. Otherwise, ACVC, I think, is the next,
Mario Vega:A spot where we can find them.
Mario Vega:Hi, Casey, yes, exactly. We're, we're, we were wanting to hear about the Prism release.
Mario Vega:Here, you can share.
kasey:Sure, yeah, Preston right now is, working on putting a release together, so we've been soaking over the weekend, and things are looking good, so we'll make sure to give an update in the, the Steel server. But yeah, we're looking at,
kasey:Building that out into a release as soon as possible.
kasey:Hopefully… hopefully today. So, yeah, we'll give an update in the… in Discord.
Mario Vega:Great, thank you, thank you.
Mario Vega:Okay.
Mario Vega:Alright, anything else related to Fusaka?
Mario Vega:Alright, if not, I think we can move to the next topic.
Mario Vega:So 60 million gas, I think,
Mario Vega:We already discussed in the previous weeks that all clients are ready for $60 million a minute.
Mario Vega:So, we can basically just jump into benchmark testing. Do we have anyone from NetherMind that can update on this?
Marcin Sobczak:Yeah, so, we have,
Marcin Sobczak:Brantis from Nethermine and GAF with EIP7904.
Marcin Sobczak:And, we did some, like, initial testing, and
Marcin Sobczak:And, yes, simply intestine benchmarks are, working as expected.
Marcin Sobczak:And at this point, we need to generate all the new payloads to match the consensus rules after this EIP.
Marcin Sobczak:So it's our next step, and after finishing this, we will be able to run a dashboard with,
Marcin Sobczak:with, like, instances of, standardomite, standard GAF, reprised Netheramide, and reprice GAF, and compare the results. So, like, we are at the step… at the step where, we need to generate, new…
Marcin Sobczak:Like, new payloads for these benchmarks.
Mario Vega:Alright, thank you.
Mario Vega:And there was… and there's one comment, .
Ameziane Hamlat:Yeah, I can, I can give the details. So…
Ameziane Hamlat:Testing7904, so if you keep the same dashboards,
Ameziane Hamlat:you may see, maybe a drop in the M gas per second, so I guess what could be interesting
Ameziane Hamlat:If, if you want to check the difference between, like, current schedule and…
Ameziane Hamlat:7904, you may need to, to include TPS, which could be, an interesting…
Ameziane Hamlat:Yeah, and at least two metrics.
Mario Vega:Thank you.
Mario Vega:Okay, anybody else, that needs to shameen?
Mario Vega:Either under repricing or benchmarking testing?
Mario Vega:Alright, we also have,
Mario Vega:a call, I think, this week, or is it next week, about gastropricing.
Mario Vega:So please also… Let me see…
Mario Vega:When is the next, repricing call? I think it's next week? Do we have?
Maria Silva:Sorry, are you… are you asking about the breakouts?
Mario Vega:Yeah, I mean, just generally about gastrophe.
Maria Silva:seem very good.
Mario Vega:Yeah.
Maria Silva:Yeah, so we had two. We had one last week and one two weeks before, and now we don't have any else planned.
Maria Silva:because it was an overview of all the EIPs, I mean, if people find that the specific breakout or repricings would be helpful, I'm happy to schedule a few more, but with DevConnect coming, I think
Maria Silva:Maybe it wouldn't make sense to do it right, right away?
Maria Silva:But yeah, we don't have any breakout schedule at the moment.
Mario Vega:Excellent, okay. Yeah, yeah, I was just wondering. Thank you, thank you for chamining. Is there recordings of this, general pricing, breakout?
Maria Silva:There are. I'm happy to share after.
Mario Vega:Excellent. Thank you.
Mario Vega:Alright, yeah.
Mario Vega:If nothing else on gas or pricing, or estate testing, or benchmark testing, I think we can move on to Glamsterdam.
Mario Vega:So the first topic is, lock-level access lists.
Mario Vega:Anyone care to chime in on… Updating the current status.
Mario Vega:I think the main question is whether we are planning on launching,
Mario Vega:DevNet at the end of the month, still.
parithosh:Yeah, so that is still the plan, and Stefan is on his way to DevConnect right now, so he can't make it, but he asked us to get two questions resolved. The first one is if the block-level access list tests are fixed, and I think Spencer is looking into it, and we might have been using the wrong release, so that's probably the cause, is…
parithosh:Yeah, Spencer, do you have any other information for that?
spencer-tb:Yeah, so they're… they're,
spencer-tb:Basically, the most recent block access list release, the version 1.4.0, is different from the previous releases. Basically.
spencer-tb:Previous releases were block access list specific tests.
spencer-tb:But this new, I guess, latest, the one that's, that needs fixed, contains the block access list specific tests, and then also…
spencer-tb:All of our existing tests refilled full block access lists.
spencer-tb:So I think, there was a…
spencer-tb:7702-related issue, with that.
spencer-tb:release, there will be a patch for that in the next release. Also, we're running these tests right now in Hive.
spencer-tb:And all the blood tests are failing because we need to fix the…
spencer-tb:We need to fix the, BPO-related configs.
spencer-tb:On how we're starting up the clients with these tests.
spencer-tb:Anyway, I guess, just a TLDR is that…
spencer-tb:Hopefully we'll have a next… the next release either tomorrow
spencer-tb:or the very latest on Wednesday, and we'll update… we'll update Hive as well. And yeah, these should be fixed.
spencer-tb:So I think Felipe from Steel has been working on this the most, and he…
spencer-tb:is out of office today, but he'll be back tomorrow, so, there'll be some action, for sure, either tomorrow or Wednesday.
parithosh:Perfect, thank you. And, the second thing he mentioned was that, so Spammer has a basic EVM fuzzing scenario in there now.
parithosh:And even just running the basic spammer with, with this config in a local block-level access list network kind of makes it immediately.
parithosh:There's an example of a failure on Nethermind. Stefan said he'll also repost the entire Kurtosis config so that clients can look into it, but it sounds like it's very brittle right now, so it would be great if we can get some fixes in and
parithosh:we're ready to launch immediately, the entire infrastructure set up. As soon as it's not as brittle, then we'll spin up the network. We were hoping to still do it on Wednesday, but I'm not sure if clients get time to fix it that soon.
Mario Vega:Thank you. So what I can see from the results at the moment is that… so we… basically, the previous
Mario Vega:the previous releases, they did not contain other… other things that… than targeted tests, right? So what happened now is that we… we are filling every single test that we had in the repository with block-level access turned on, and now we have, basically.
Mario Vega:1,000 tests at the minimum failing for every client.
Mario Vega:Something I noticed is that,
Mario Vega:I… from debugging in the logs, in the clients, I don't see that many of them are printing the full block-level access list that they are expecting.
Mario Vega:Right? So, I think, I don't know if this was discussed on the, on the local accesses breakout, but I think we need to make sure that all clients do print, if they find…
Mario Vega:the block level access does not match what they were expecting. It would be nice if every single client, at least on debugging, they would bring the entire block level access, so we can easily debug and compare
Mario Vega:What the test expects versus what the client expects.
Mario Vega:So yeah, that was, that was something that, I, I, wanted, I want to share.
Mario Vega:Cool.
Mario Vega:Yeah, but yeah, come in, bye.
parithosh:It's kind of the same ask we had with the debug, bad blocks endpoint. I think there, also, the request was to have not just the block-level access list that you think is bad, but also what you generated. And I think Bezos already done this, and Jared looked into it and said he will do it for Geth. I'm not sure about NetherMind and the other clients, though.
Toni Wahrstätter:Yeah, everyone will edit. This was part of the block of access list breaker call result result. At the very end, all the clients agreed to adding that.
Mario Vega:Amazing. Thank you. And coming back to the point of the DevNets, I don't think we should launch until we get this, this number of failures on Hive to a very… to a very low number, more manageable, because we're certainly gonna find more things.
Mario Vega:And it's gonna be, like, breaking all the time, at this point.
parithosh:Yeah, for sure. We also don't want to launch it when, like, a basic fuzzler can take it down in half a minute or something.
Mario Vega:Alright.
Mario Vega:Yeah, thanks, thanks for the input. Anything else on block-level access lists that we want to share in this meeting?
Mario Vega:Okay, yeah, okay, so, and TLDR, basically just go to the Hive, Hive results page, look at your client, look at the logs, and please chime in. Also, of course, if there's anything wrong with the tests.
Mario Vega:So if you think you are in the right, in the correct, regarding rejecting a block load access list, please also chime in, and with the, with the testing team.
Mario Vega:either with, Felipe, Tony, or Rahul, we can, we can take a look at the test and also, also try to fix it if, if you guys, think that the test is incorrect.
Mario Vega:Alright. Good. Moving on, to ePBS,
Mario Vega:Is there anything, anything worth of discussion today?
Justin Traglia:I can chat for a second. Hello? We had the ePBS breakout call on Friday last week. It was pretty quick, but we discussed adding an off-protocol value field to the execution payload bid structure.
Justin Traglia:I think the name for this is Execution Payment. We also discussed including a previous Randall field in the bid, which is necessary.
Justin Traglia:And calling, process builder payments earlier and process epoch, that's still undecided.
Justin Traglia:We plan to…
Justin Traglia:Include these changes in the next spec release, maybe within, like, a week or two, and then DevNet Zero will be based on that.
Justin Traglia:I personally don't think that DevNet Zero for EPBS will happen probably until mid-January, at the earliest, due to DevConnect and the holidays, so just keep that in mind.
Justin Traglia:I know clients are still making good progress on implementations,
Justin Traglia:And the next, breakout call, which was scheduled for, I think, November 21st, has been canceled due to DevConnect, so…
Justin Traglia:I think the next one… was, it is December 5th.
Justin Traglia:I don't… that's pretty much all I have, unless clients wants to give… wants to give.
Justin Traglia:More detailed updates.
Mario Vega:Thank you. Okay, any clients?
Mario Vega:For any other updates?
Mario Vega:Yeah, I think that was… that was basically it. Thank you, thank you, Justin, for the update.
Mario Vega:Alright, I think that's basically the entire list of topics. The only other item in the agenda today is whether we are going to proceed or cancel with the
Mario Vega:next two weeks of calls. The next week is a little bit easier. Basically, is that kind of week. We are… a lot of us are gonna be over there.
Mario Vega:and discussing in person, so I don't think we should hold that meeting. The only open question is the week after that, because mainly… mainly because most people are gonna be traveling.
Mario Vega:Some, part of the IPANDOps team are also gonna be offline the week after that. So, I'm talking about the… specifically, the…
Mario Vega:24 of November, the call on that. Is there any opposition on canceling that meeting? Should we… should we… or…
Mario Vega:What's the… what's the sentiment?
Saulius Grigaitis | Grandine:maybe not directly answer to this, but I think most of the… or at least a significant amount of the teams are actually experiencing some, let's say, bugs or not polished releases, and
Saulius Grigaitis | Grandine:And my general question is, how do you feel… how do you feel about that? I mean, there isn't that much time left to the main it, and… well, the bugs are still there, you know?
Saulius Grigaitis | Grandine:So… Any thoughts on this?
parithosh:Yeah, maybe just reiterating what Barnabas is mentioning, the question is how many of these are really Fusaka-related versus could have happened in any major release. I do think…
parithosh:For at least recent few months, we've kind of kept off on making major releases so that we can have everything merged into trunk branches, so…
parithosh:kind of makes sense that the first release would have some issues or the other. The only thing we can hope for is we're working on a mainnet shadow fork, so that we should be able to catch a few more of them before they go live, and I think it's also expected that clients would have a hotfix release if they do find anything. That's also one of the reasons why we
parithosh:Have releases about a month before the fork, so that releases can soak, we can have extensive rounds of testing and catch everything.
Saulius Grigaitis | Grandine:I'm… I'm just wondering, maybe there is some statistic on, how…
Saulius Grigaitis | Grandine:Much of, like, these large, stakers or, or staking entities, have a redundant, setups, because just for example,
Saulius Grigaitis | Grandine:data that was a bit nether mind on Hudi. At least on my instance, I needed to resync, the EL, so…
Saulius Grigaitis | Grandine:So, I mean, if we…
Saulius Grigaitis | Grandine:if the setups are highly redundant, in terms of, you know, multiple clients are used by the same staker, redundant setups, then I think these, these releases that, has, some issues, like we…
Saulius Grigaitis | Grandine:have now. Maybe it's not a big problem, but if the setups are not that redundant and are not that diversified, then
Saulius Grigaitis | Grandine:There could be some turbulence after the hard work, at least based on the issues that we have now.
parithosh:Yeah, for sure. We don't have exact numbers from everyone running every validator on mainnet, but at least for large staking pools, they tend to have a backup beaker node that they can fail over to.
parithosh:A lot of them use a testint vouch slash similar style of middleware proxies, where if there is a failure in one node, you can define a strategy to failover to a different node.
parithosh:And there's also SSV and OBOL-style clients on mainnet, so I don't think the answer is 100% redundant, but it's not 0%.
Saulius Grigaitis | Grandine:Okay, sounds good then. Likely these issues will be covered by these redundant setups.
Mario Vega:Alright, and one comment from Barnum is that this is…
Mario Vega:One of the smoothest test networks since the March.
Barnabas:Yeah, I think so. Like, we had many forks since the merge, and every single,
Barnabas:or had some kind of bug here or there. I think Flipica has been one of the most stable since, since we started forking the testimonets.
Barnabas:Yeah, smoother than actual merger as well.
Mario Vega:Alright.
Barnabas:Yeah, I don't think there's much to worry about, to be honest.
Mario Vega:Any other concerns that, or opinions from other clients?
Mario Vega:Never mind, I mean, or Reth, did you guys, experience the, the issues? If you want to share anything, or…
Ben Adams:Yeah, we…
Ben Adams:We released a fix for it, like, 12 hours after the release. We were a little bit ambitious in the
Ben Adams:What we included in the, in that initial release.
Ben Adams:But yeah, it's fixed now, and we're… we're being a bit more cautious.
parithosh:Maybe a follow-up question is, we do still have some testing bandwidth now that the releases are shipped. We are focusing on Shadow Fork, and then making sure that there are no high failures, but are there any other angles or…
parithosh:Specific things that we should be looking at that client teams are worried about, because we can then try to prioritize that over the next weeks.
Saulius Grigaitis | Grandine:Do you guys do a non-finality testing?
Saulius Grigaitis | Grandine:On a larger network.
parithosh:Do you mean the, like we had in September?
parithosh:We did an opportunity test round then, but we haven't repeated that right now.
Saulius Grigaitis | Grandine:Yeah, I think it would be valuable to have a… Last time, just before the…
Saulius Grigaitis | Grandine:Before the hard work on the mainnet.
parithosh:Okay, I think we can plan that in for after DevConnect in that case.
parithosh:Yeah, we will discuss that and get back to you.
Saulius Grigaitis | Grandine:Great, thanks.
Łukasz Rozmej:Yeah, I have a question. We probably do have it somewhere, but, some stats for when hammering blobs for post-peer DAS in terms of, like, blob pull. Do we have that… confirm that everything is fine there?
parithosh:Yeah, Sam's been looking into that, and hasn't found anything worth surfacing right now, but part of the problem is the blobs kind of originate from us, because we're doing the spamming, so it's hard to tell if there's a bug that is hidden, because we're basically reusing testnet infrastructure.
Mario Vega:Oh, I think, I think, going to Lucas' question, do you, do you guys have, like, any dashboard where we can… I mean, is there, is there anything worth, like…
Mario Vega:Or is this internal, and you only surface if you find any bugs, or do you have, like, public metrics, or…
parithosh:It… mostly just ZATU dataset data. We can share something, but…
parithosh:Yeah, the question is, like, what we're looking for.
Łukasz Rozmej:So my, my question was mostly about performance degradation, or, like,
Łukasz Rozmej:Starvation, or… bandwidth starvation, or latency spiking, things like that.
Łukasz Rozmej:Because of just the high memory… high bandwidth usage of blobs per post-Fusaka, and mempool.
parithosh:Okay, so, so, okay, got it. I think the Sunnyside Labs has some dashboards for that. We can, share that, yeah.
Mario Vega:Awesome.
Mario Vega:Alright, any other concerns from Clintense? Regarding current status?
Mario Vega:And there's a question from Casey about, private blobs and rear-heavy situations.
Barnabas:We also didn't notice any issues with private blobs. We did many tests, when…
Barnabas:The relay was the only one seeing the blobs, and no other nodes were able to use, get blob V2.
Barnabas:So… We didn't see any issues with that. Rear communications might cause issues, with some specific ER clients.
Barnabas:I'm not sure if…
Barnabas:we have fixed all of that. I remember Netherland had the pre-org issues, they push out fixes. We haven't had a non-finity test since
Barnabas:I would say maybe a month and a half now, so… yeah, we can definitely redo that on Devon Street.
kasey:Yeah, and if I can jump in, one concern I have on the CL is that when we reorg, you know, we exercise some pending block queue code paths that are not usually really, exercised in tests that don't have these reorgs, so, you know, we have
kasey:a missed, block, not only do we need to get the block, now we need to go get, like, a super node might need to get 128, column sidecars, you know, and that can just put a lot more strain on the node. So I think that would be… if we have a way to test that in.
kasey:some of the testnets, that would be good. And, and for the private blobs, I'm wondering, are we… are we continuing to test that, like, as we go through, like, Hoodie and so forth?
Barnabas:We are not. We're doing only on DevNet 3. We don't…
Barnabas:Basically, participate on testnet, manipulating testnet data.
Barnabas:We don't run any relays, so this would be up to the relay operators to submit private transactions, and
Barnabas:Yeah, I mean, they… they might have already done it,
Barnabas:We do not have control over those.
kasey:Gotcha.
Mario Vega:So if I… sorry. Yeah, go ahead, Praszh, sorry.
parithosh:There was, one more, bug that we were talking about, relays and, map boosts in itself.
parithosh:I've posted it here, Bharat, along with the new construct folks, dug into this. The issue was that if you have multiple, relays configured, then the… there's a race condition. It's now been fixed, and there should be a release soon. But yeah, just keep an eye out for that one.
Mario Vega:Okay, thanks. Just going back to the question of non-finality, are we doing that on DevNetri? At some point, it was at the consensus?
Barnabas:Oh, we can do that, yeah. We can redo another one.
Mario Vega:Alright.
Barnabas:I would just ideally want to have the numbers bug,
Barnabas:Figured out before we trigger an infinity.
Barnabas:Because as far as I know, that's the only place where Nimbus has detected issues with this.
Mario Vega:Alright, for sure, yeah, yeah, okay, so we wait until it's… that one's fixed. If we already reach out to Nimbus, I think we should be fine, and then we try to trigger the non-finality.
Mario Vega:Dimensory. Perfect.
Mario Vega:Alright.
Mario Vega:Any other comments from client teams?
Mario Vega:Alright, if not, going back to the original question, so do we keep… The column 24…
Mario Vega:For a CDT, or do we cancel? Is that…
Mario Vega:Or should we leave it open? Maybe you can leave it open and come back to it, and…
Mario Vega:Alright, there's, boarding right now, do we cancel or keep it? It seems like… Alright.
Mario Vega:Yeah, it seems like we have to… we… we are leaning towards cancel, so…
Mario Vega:Yeah, let's just cancel. So, just to summarize, the week of the… sorry, the call of the 17th is going to be canceled, and the call of the 24 is also going to be canceled.
Mario Vega:We are back on, December 1st, for ACDT.
Mario Vega:Alright.
Mario Vega:Good. Any other thoughts?
Mario Vega:On, on… that need to be discussed today, or… because that's everything in the agenda for today.
Mario Vega:Alright, if not, I think we can, we can, we can end it here.
Mario Vega:Thanks everyone for joining. See you on… One… one raised hand. Casey, do you have last comments?
kasey:Yeah, I just… since I said before that I would follow up on the Prism release, I did hear from Preston that, he is planning on getting the release out today, so I think we can't expect that to be done today.
kasey:That's it.
Mario Vega:Excellent, thank you so much.
Mario Vega:Alright, thanks everyone, and thanks for joining, and see you in DevConite, or hear you in, on December 1st.
Mario Vega:Goodbye. Thank you.
Antoine James:Thanks, everyone. Bye-bye.
Pooja Ranjan:Thank you.
Marius van der Wijden:What?
Louis:Thank you.
Chat Logs
00:03:16
parithosh:There was also a reth release bug
00:05:41
Barnabas:1.9.0 1.9.1
00:05:44
Barnabas:basically this ^
00:06:09
wolovim:In case unclear for anyone (or any AI summarizer ;)) hoodi BPO2 will happen this wednesday
00:06:13
parithosh:https://github.com/sigp/lighthouse/issues/6837
00:08:07
Barnabas:we have seen a halting issue with nimbus on fusaka-devnet-3
00:08:23
Barnabas:they are still investigating
00:11:19
Jen:Sorry, I joined late. For Reth, we included the bug fix in v1.9.1 which we patched on Friday https://github.com/paradigmxyz/reth/releases/tag/v1.9.1
00:13:56
kasey:I just joined - I think you are asking about prysm release?
00:16:43
Ameziane Hamlat:You may need to include TPS as mgas/s may be even lower
00:18:36
Louis:I think it is on Wed?
00:19:13
Francesco:Could make sense to have another one this wednesday to go a bit beyond the overview stage, at least for some EIPs?
00:19:14
Toni Wahrstätter:We can share the recordings
00:19:20
Louis:On youtube
00:20:53
Mario Vega:https://github.com/ethereum/execution-spec-tests/releases/tag/bal%40v1.4.0
00:21:59
Mario Vega:https://hive.ethpandaops.io/#/group/bal
00:22:22
parithosh:Nethermind fails: https://discord.com/channels/1359927674746835211/1424818910846386357/1435999369919725659
00:22:39
Ameziane Hamlat:Replying to "You may need to incl..."
I meant in the grafana dashboard, as by EIP-7904, the nodes should include more transactions for the same gas limit
00:24:54
spencer-tb:Replying to "https://hive.ethpand..."
1000s should be fixed once I sort the bpo configs on start up for Amsterdam, hive config related fails, then we’ll have a truer overview on the fails.
00:29:14
Justin Traglia:I vote to cancel.
00:29:59
wolovim:For bookmarking, the two gas repricing calls: https://www.youtube.com/watch?v=f_1sTYTY9DY&list=PLJqWcTqh_zKE8xsbR8cPEDQr01e2L0Qb_&index=3
00:30:10
Barnabas:new bugs seems to be added
00:30:14
Barnabas:very little bugs are fusaka related
00:30:47
spencer-tb:Yeh they all seem very regression related (at the EL ones)
00:31:09
Barnabas:This is also why we have client diversity :D
00:31:15
Ben Adams:Nethermind already hotfixed 12hrs later
00:31:55
Barnabas:Most expensive setups will most likely be highly redundant
00:33:03
Barnabas:just in comparison, this has been the smoothest testnet fork since the merge.
I don’t think any of our previous forks were 100% flawless
00:33:21
Enrico Del Fante (tbenr):As it should be for proper business 🙂
00:33:29
parithosh:Smoother than the merge 😛
We had all testnets fail for the merge
00:34:01
parithosh:But yes please test your releases!
00:34:02
Barnabas:I’d be alarmed if we didn’t have any bugs :D
00:34:03
parithosh:We will be too
00:35:48
Łukasz Rozmej:Do we have stats for mempool post peerdas?
00:37:08
Barnabas:we see no issues with 21 blobs
00:38:15
kasey:How are we feeling about:
Dealing with private blobs?
Re-org heavy situations?
00:40:46
parithosh:https://github.com/flashbots/mev-boost/pull/855
00:42:44
kasey:Could we drive this based on the pm issue agenda or lack thereof?
00:42:50
Barnabas:keep 👍
cancel 👎
00:44:37
Justin Traglia:Bye everyone, see ya!
00:44:40
kasey:Bye!
00:44:41
jochem-brouwer:Bye!!
Summary
16 highlights
· 2 decisions · 8 action itemsExperimental
Summary
16 highlights · 2 decisions · 8 action itemsExperimentalfork status and schedule
critical infrastructure
client updates
testing progress
Decisions
Action Items
- Mario: Follow up with Reth on release and blog update00:05:59
- Enrico (Teku): Post about premature RPC requests in Telegram/Discord00:08:25
- Barnabas: Follow up with Nimbus on devnet-3 halting issue00:09:32
- Spencer and Felipe: Release BAL v1.4.0 patch and fix Hive configs00:21:19
- All client teams: Implement debug logging for block-level access lists00:23:36
- ePBS spec team: Release ePBS spec with execution payment and Randao00:27:05
- Barnabas and DevOps: Conduct non-finality testing after Nimbus bug resolved00:41:24
- Prysm team: Release stable Prysm mainnet version00:14:21