AllCoreDevs - Testing #057

|2025-10-13| Agenda: #1756| canonical JSON

Transcript

Mario Vega:Welcome, everybody. This is ACDT number 57, today is October 13th.

Mario Vega:And we have a light agenda today, so basically we'll go over, some Fusaka, status, then we'll pass on to discuss the gas limit.

Mario Vega:updates, and then, Glamsterdam.

Mario Vega:And we also have another topic that we want to discuss in this meeting about the RPC testing failures.

Mario Vega:So yeah, let's just get started. Alright, on Fusaka, regarding the BPO fork updates, do we have any…

Mario Vega:Incidents, updates, or status that we want to bring up?

Barnabas:Not really. We have the next BPO scheduled for tonight.

Barnabas:I think it's in a couple of hours,

Barnabas:Yeah, and we have Sapoya tomorrow.

Barnabas:Yeah, it's in 7 hours, for the next BPO on Holsky.

Barnabas:be duty, Bit of a drop in participation, but it's minor at this point.

Barnabas:We expect a smooth sailing for tomorrow's park, and

Barnabas:And the fork is on Sepolia is in 17 and a half hours.

Barnabas:Last week, we did a Sapolia shadow fork, and everything was okay there.

Mario Vega:Alright, yeah, sounds like we're moving along. Any…

Mario Vega:Insights from the client teams.

Mario Vega:Any concerns regarding either… sorry, regarding either Holeshki or Sepolia?

Mario Vega:Happening tomorrow?

Mario Vega:Alright, seems like not.

Bharath:Yeah, I have one.

Bharath:Sorry, sorry, I missed you.

Bharath:I… one… one thing I want to, like, inform all the client teams is, we have a new release of MevBoost, V1.10 Alpha 6. The difference is that, at this point.

Bharath:we've chosen to add, support to fall back to the V1 Blinded Blocks API if the V2 Blinded Blocks API is not implemented by the relay. We initially chose not to do that because we wanted MevBoost just to be a proxy.

Bharath:But I think on further discussions and thought, we just saw this as a much better idea. So, yeah, so MepBoost now supports, like, falling back to the V1 API, if the V2 API is not supported, but for…

Bharath:For whatever reason by the relays. We've been checking with the major relays, they have added it, they should be ready. For Sepolia, the flashbots ideally should be ready with the V2 API.

Bharath:So, yeah, this has been tested on DevNet 3, we've, like, tested by disabling, like, the V2 API on DevNet 3, and I generally think seem fine with the relays, with the clients.

Bharath:So, yeah, just wanted to, like, inform all the client teams about it, and all the support… please, like, do update all the Sepolia validators to use this release of MavBoost. I have put that in the Sepolia Validators Telegram channel, too, with all the information about the release.

Bharath:So yeah.

Bharath:Yeah, I just wanted to mention that.

Mario Vega:Perfect, thank you.

Mario Vega:Just one question out of curiosity, how do you detect that B2 is not supported?

Bharath:Yeah, for now, we've just, like, have a situation where if the return status code is greater than just, like, 4XX,

Bharath:like, we fall back to the V1 API, right? So, usually, if the API is not there, we get a HTTP status, like, 4.4 not found, and we fall back to the V1, right? But just for safety, we… what we just did was, if the HTTP status code is, like, greater than 4XX, we fall back to, like, the V1 API.

Bharath:So… but, like, the relay itself is, like, booked, like, then, like, even the V1 will fail, and there'll be a miss slot, so that's… that we can't do anything about.

Mario Vega:Alright. Yeah, thanks, thanks for the context. Anybody,

Mario Vega:Any opinion on this, from client teams? Are we okay?

Bharath:Yeah, so one thing is, like, client teams can choose… so, I know some client teams

Bharath:have fall… like, Lighthouse have supported, like, fallback, and some client teams are calling the V2 API from Fulu. Generally, like, client teams can, with this new change, client teams can ideally just stick to however, like, they're calling the V2 API. Like, ideally, like, MevBoost will handle all the fallback, like.

Bharath:So, yeah.

Mario Vega:Excellent. Thanks.

Mario Vega:Alright, and also, there was one comment from Prism, they have a new release out, 6.1.2,

Mario Vega:And the blog has been updated. But also, Barnabas shared a list of the versions that we have in the blog post, so please take a look. If the version that you have here is outdated, please reach out so we can update it.

Mario Vega:And one another comment is, Lighthouse, be so organizing, do we expect a stable release before Hoodie?

Mario Vega:Anybody from Lighthouse, Vesu, or Grandin in the call?

Ameziane Hamlat:Yeah, so, as I said, for, for Bessru,

Ameziane Hamlat:We are going to have a new release for, for Hodi and, and mainnet related to, to 60 million gas.

Ameziane Hamlat:Yeah, so, the release is expected to be, out this week.

Ameziane Hamlat:And we'll share, we'll share it as soon as we have it.

Mario Vega:Got it. Thank you. Anyone from Lighthouse?

Mario Vega:Anyone from Grandin who wants to make a comment?

Mario Vega:Are we do… are we good there?

Mario Vega:Alright, I think we can follow async, if not.

Mario Vega:Okay, any other updates regarding, Holeshki or Sepolia that we want to bring up?

Mario Vega:If not, I think we can move to the next topic.

Mario Vega:Yeah, so the next topic is regarding…

Mario Vega:this message on Discord via Barnabas.

Mario Vega:And basically, it's the idea that we want to remove the named forks from the BPO schedule.

Mario Vega:If the values are not affected. But yeah, Barnabas, do you want to keep more context to this, and just begin the discussion?

Barnabas:Yes, so the idea is that we will be dropping the named folks from the blob schedule in the genesys.json file.

Barnabas:We have already done this for Bezu and Nethermind, and

Barnabas:Breath and Eragon also seems to be fond of it, and right now the only question is the Nimbus EL and, Geth.

Barnabas:So the… the few PRs that are open are to remove Osaka, and also in the future, we just don't plan to include Amsterdam in the block schedule either.

Mario Vega:And just for clarification, this is only if the name fork does not update the values, right?

Mario Vega:If it doesn't…

Barnabas:The name, the name fork cannot update the values anymore.

Bumblefudge:on the CR side.

Mario Vega:I sucked.

Barnabas:conclude.

Mario Vega:I see.

Barnabas:And that's why I'm proposing to remove this, because this is a redundant tag at this point.

Mario Vega:Makes sense.

Mario Vega:Any comments on this? Should we move on with this? I think it makes sense. On the east side, I think we have to make a new release to make sure that we are not including the name of Forks, but I think it should be an easy change, and we can have a release in the following days.

Mario Vega:On the client side, is there any concerns regarding this update?

Mario Vega:And one comment from Barnabas Marius. Is this okay to you? I think that you're only the one, missing to comment on this.

Marius van der Wijden:Yes, we have not implemented it in guest yet, but, yeah.

Marius van der Wijden:This is fine.

Mario Vega:Alright. Then I think what we'll do on the east side is also we're gonna make a new release, just to make sure that we can run the clients with static tests, and then we can move on to…

Mario Vega:Blessing granted. Yeah, and we can move on to updating the notes on the dev notes, I guess.

Mario Vega:Alright.

Mario Vega:Barnavas, any other comments, or is this okay?

Barnabas:Yeah, that's okay.

Mario Vega:Thanks.

Bumblefudge:Alright, I think that's…

Mario Vega:That was everything for Fusaka.

Mario Vega:And I guess we can move on to the next, topic, which is the gas limit testing that we've been doing. I think Camille is in the call, or also anyone else who wants to chime in, but Camille, do you want to begin the discussion?

Mario Vega:On the updates on this topic.

Kamil Chodoła:Sure. So, as you heard already from Bezo and other clients, everyone is kind of ready for 60 million, for mainnet releases.

Kamil Chodoła:And in the meantime, we are focusing on stageful testing, which is the biggest, maybe not unknown, but the hardest thing to test right now.

Kamil Chodoła:But I think that, actually today I have a good milestone met, is that I'm running free EL clients on top of BloatNet in a reproducible way, so in the same way as gas benchmarks are doing.

Kamil Chodoła:So, for now, we're running Nethermine, Bezu, and GEF.

Kamil Chodoła:having still issues with, Aragon and Ref, so waiting for some help here on State Blood Research Channel.

Kamil Chodoła:So I can make all 5 major EL clients being tested this way.

Kamil Chodoła:And Joel had prepared some scenarios, actually forks and scenarios, and for now, we do not see any huge bottleneck.

Kamil Chodoła:And those scenarios are not for sure the most performant ones out there, and for now, Netarmine is having the biggest struggles, but not, like, consistently. So, just to give you an example, once some of the extent scenarios are performing at more or less 250 megabits per second.

Kamil Chodoła:And sometimes they are dropping significantly to something like 35.

Kamil Chodoła:Which is still relatively good, and not putting any risk on 60 medium gas increase for now. But I believe that we need to still spend much more time investigating that, because it doesn't align really well with what we've observed on Mainnet.

Kamil Chodoła:So, there's still some work to be done out there. But anyways, dashboard is set, 3 clients are running right now, I'm just doing some final fine-tunes, and we'll… the only thing which is left is work on more stateful scenarios on top of mainnet.

Kamil Chodoła:So, at least the entire mechanism there is running.

Mario Vega:Thank you. Yeah, great work on this,

Mario Vega:Any comments from clients on this?

Mario Vega:Do you have, like, a theory on why are we not seeing, like, the same…

Mario Vega:benchmarks that we are… that we're seeing on mainnet.

Kamil Chodoła:So, I have a few theories for now. One will be that we are preparing some sense preparation blocks, then trying to spam with different contracts to fill the caches, and later on we are doing the actual attack.

Kamil Chodoła:And one client might cache more while the other is caching less, and for some reason, the actual send path is warmer than expected, and maybe that's one of the things, but it's pretty hard to say, and trying plenty of things, like including restarts, right, without the attack.

Kamil Chodoła:Or, doing…

Kamil Chodoła:few thousand of the spamming blocks, which are doing completely unrelated work, and then attacking XAM, but it's pretty time-consuming, and taking some time to… to narrow it down together with Johan.

Kamil Chodoła:plus the hardware. I mean, we are using hardware which is below the recommendations, but I need to compare with the notes which were reporting pretty slow times on Xen to see if this is somewhat in common.

Kamil Chodoła:So, suspecting that maybe my hardware is a little bit better than from the notes which were reported to quite slow times.

Kamil Chodoła:And maybe we still need to spend more time to see if we can do something worse using some contract.

Mario Vega:Got it, thank you. Makes a lot of sense.

Mario Vega:Any comments from other teams on this?

Mario Vega:Amazon?

Mario Vega:Yep.

Ameziane Hamlat:Yeah, I have a question. Are those con… are those blocks, like, exam blocks, part of the…

Ameziane Hamlat:performance DevNet 2, like, they are really important as canonical blocks on that DevNet.

Kamil Chodoła:The ones which I'm using for testing.

Ameziane Hamlat:Yes.

Kamil Chodoła:No, these are crafted by us.

Kamil Chodoła:So, we use the same contract, like the real one. The baseline is BloatNet, not mainnet, so we use 2x8, but we craft all of the preparation blocks, and then the actual attack block.

Ameziane Hamlat:Okay. Yeah, I was asking just to, to see if we can help, you know, in, in profiling those blocks.

Ameziane Hamlat:From our side as well, to see if there is anything to… to find from the profiling.

Kamil Chodoła:I can work with you to maybe hook up some profiling tools to see how… how…

Kamil Chodoła:what we can get from there. For example, trying right now to see if I can hook profiling tools from Nethermind to see how it will be doing. Because it's not that easy just to take that block from mainnet and just recall it once more.

Kamil Chodoła:Because actual send needs a few blocks to prep… to prepare, some hacking around timestamps of the blocks, because some of the functions from SEND are being able to be called after, right, 24 hours.

Kamil Chodoła:So, once some block will be called on mainnet, you can just recall some functions, and the result will be the same. We just need to do quite a few of the preparation steps to call it.

Kamil Chodoła:More details are from Jochen, probably, here.

Ameziane Hamlat:I see. Yeah, we can… we can do it, I think.

Ameziane Hamlat:Thanks.

Mario Vega:Thank you. Any other comments?

Mario Vega:There was one comment in the chat that the…

Mario Vega:Same content is already bigger in Bloodnet, so, I think…

Mario Vega:We are on a good path to trying to reproduce this thing.

Mario Vega:The same as mainnet.

Kamil Chodoła:Very important note for me to EL clients will be, like, if you see any interesting stateful scenarios which we can implement right now, because actually what is left right now is implementation, so you can just reach out to us and tell what bottlenecks you see in your clients, and what interesting things we can do to

Kamil Chodoła:to attack some specific clients in a worse way, which might not be that bad for the other clients to try to stress that as much as possible. And also, the warming up part for me is a little bit more tricky than on normal compute scenarios, because here.

Kamil Chodoła:what gets warmed is not only EVM, but then we have also loading of the data from disk to memory, and then also there is some kind of hardware warm-up, which affects the results pretty much.

Kamil Chodoła:So… yeah, there are quite a few things which need to be considered comparing two compute scenarios where just simple

Kamil Chodoła:block with invalid state would give you improper warm-up, like…

Kamil Chodoła:Which would… which would be good enough.

Kamil Chodoła:So here we have a few levels of issues, which we need to solve, so if you see any…

Kamil Chodoła:Possible solutions, how to tackle that, just let me know.

Mario Vega:Thank you. Yes, so please reach out to Camille, I'm guessing R&D Discord is the best way to approach. Yeah.

Mario Vega:We'll keep an eye on the… on the scenarios that we can implement.

Mario Vega:Alright.

Mario Vega:If nothing else, we can move on to Glansterdam.

Mario Vega:Oh, sorry, Marcin, there was, please, go ahead.

Marcin Sobczak:Hi, I have one more thing which has sense to discuss before going to Glamsterdam. We are doing analysis of Modex usages.

Marcin Sobczak:Like, as a peak introduction, we are… we are doing traces of blocks under…

Marcin Sobczak:PECTRA consensus rules and Fusaka Consensus rules.

Marcin Sobczak:We are artificially increasing gas limit of transactions before purchasing to… because, like, modex gas prices increased, and,

Marcin Sobczak:We think that everyone will take it into account before sending transactions… transactions post

Marcin Sobczak:hard fork. So, with these assumptions, we are looking for differences, basing on, debug traces.

Marcin Sobczak:And we found a few hundred transactions which are failing.

Marcin Sobczak:after Fusaka, but it's, all of them are, have one pattern, the same pattern, and the same, error message.

Marcin Sobczak:It's, approximately one transaction per hour, which is failing because of that. And I pasted that, error message is from account abstractions.

Marcin Sobczak:And all these transactions are interacting with the same contract. It's entry point 0.6.0. I pasted one transaction as an example.

Marcin Sobczak:And

Marcin Sobczak:We suspect that it is because we are increasing the general gas limit of transaction, but inside of account abstraction transactions, there are user operations.

Marcin Sobczak:which we suspect that after hard fork, we have new estimates with higher gas limits, and then it shouldn't fail, but it's, like, we suspect this. So, we would like to confirm that this is the issue.

Marcin Sobczak:And generally, share the progress, from this, area. So,

Marcin Sobczak:Yeah, it is our finding, and we need to confirm that it will be fine after Fusaka. We suspect that it will be, but we are not sure at this point about it.

Mario Vega:Okay, do you know, Marcin, if there's already a reference test for this?

Mario Vega:Scenario, or similar?

Mario Vega:If not, we can investigate that for you, but if you know…

Marcin Sobczak:I don't think so, I don't know.

Mario Vega:Alright. Yeah, I think… I think we need to investigate whether this… there's a reference, test implementation for this.

Mario Vega:Or something similar.

Mario Vega:Anyway… Any comments on this?

Mario Vega:Now that we can move to Amsterdam.

Mario Vega:Alright, thanks, Dr. Martin, for the, for the comment. We can start with, block-level access list,

Mario Vega:I see… Who's in the call? Felipe? Whilst…

Mario Vega:And Rahul, do you guys want to kick off the discussion?

felipe:Yeah, I can mention,

felipe:Over the weekend, the… there were some of the test vectors from our release last week.

felipe:That were flagged for a couple changes needed.

felipe:But those have already been fixed and merged as of this morning.

felipe:So, I'm gonna triple check… If there's anything missing.

felipe:And try to get a release out early this week.

felipe:For… for block-level access lists.

Mario Vega:Thank you. And also, Raul shared the results of the tests that are running and passing on each of the clients.

Mario Vega:On the DevNet side of this, Barnabas, do we want to wait for 100% passing of the tests, and then schedule

Mario Vega:The block level access list definite.

Mario Vega:Or what would be the next steps?

Barnabas:The BL testnets are taking over by Stefan.

Barnabas:Maybe Stefan can comment?

Stefan Starflinger:They're still working on, like, revamping their implementation. There seem to be some improvements that they want to make.

Stefan Starflinger:So I think we should have at least 3 clients working pretty well before we start the definite.

Stefan Starflinger:And I think, to your question, I think it would be good to have as many tests passing as possible before we start the DevNet, of course.

Mario Vega:Absolutely. Alright, so I think the next steps on this is trying to get this…

Mario Vega:The number of dust passing to at least

Mario Vega:I would say 100%, before continuing. Do we know the nature of the tests that are failing, Rahul? Like, are these consensus tests, or are these, like.

Mario Vega:Other kind of tests.

Mario Vega:That are failing.

Mario Vega:Raul or Felipe?

felipe:the… for the DevNet tests?

Mario Vega:For the spec tests.

felipe:For the spec tests, the ones that were flagged were, out of gas issues that…

felipe:On the spec side, we had made some changes that needed to… Be returned.

felipe:The specs got updated over the weekend, and the tests got updated over the weekend. So they were generating, some factors that were…

felipe:Not correct. And then we weren't… Properly validating on the, test side?

felipe:And so, I made sure that we could… we were using the right markers there, so that we could catch it.

Mario Vega:Yeah, and on the client side, do you know what's the… the reason why they're failing?

Mario Vega:I mean, rather than…

felipe:The vectors that were being generated were not correct.

Mario Vega:Oh, I see, I see.

felipe:But for the tests that I looked at, I'm not sure if this is the same as the ones being run on the DevNet.

Mario Vega:Thanks.

Mario Vega:Rahul, do we have, like, a link where we can see the results for each client on this?

Mario Vega:I think I got it here.

Mario Vega:Thank you. I already shared. Great. Thanks. I think, please take a look, clients, on this.

Mario Vega:To see if there's any outstanding tasks that are failing on your side that we need to… that you guys need to fix.

Mario Vega:And please reach out to Felipe and Rahul also, if there's…

Mario Vega:Any unknowns on the tests that are currently running, so we can get this to 100%, and then we can schedule the…

Mario Vega:Local access to this Devna.

Mario Vega:All right, any other comments or topics on block-level access list?

Mario Vega:If not, we can move on to, EPBS side of things.

Mario Vega:Alright, who's, the person.

Justin Traglia:I can try.

Justin Traglia:Alright. I mean, not much of an update, just the client teams are still working on DevNet Zero implementations.

Justin Traglia:It appears that DevNet Zero by the end of October is unlikely, just in my opinion.

Justin Traglia:Progress has slowed a bit by, like, the off… off-protocol payments discussion debate that's going on. I'm hoping that we can make a decision on this at the upcoming ACDC call.

Justin Traglia:Small stuff, not a whole lot to talk about here.

Justin Traglia:That's pretty much it.

Mario Vega:Alright, thank you.

Mario Vega:Okay, yeah, thanks for the updates on this.

Mario Vega:Anything else on Glaston that is not related to Blackload Access List or EPBS that we need to discuss today?

Mario Vega:Alright, if nothing else, we can jump onto the last topic for today, which is the RPC test failures. There's a blog post in the EATH research site.

Mario Vega:That was raised by, S.C. Burkel.

Mario Vega:Regarding failures on the RPC test suite that we have in Hive.

Mario Vega:I see Sebastian on the call right now, do you want to kick us off with this conversation?

Sebastian Bürgel [Gnosis/HOPR]:Sure, yeah. Thanks for supplement for having me here, Mario. I'm Sebastian, from Gnosis slash Hopper.

Sebastian Bürgel [Gnosis/HOPR]:And, yeah, I think we're all aware that Ethereum is trying to onboard a huge number of individual users.

Sebastian Bürgel [Gnosis/HOPR]:And, yeah, that's… in that regard, we're quite aligned with what we do at Gnosis to build products that actually use an Ethereum chain. Currently, Gnosis L1.

Sebastian Bürgel [Gnosis/HOPR]:And, in doing that, we kind of see, and that's why I'm here, to kind of sound the alarm bells, kind of a…

Sebastian Bürgel [Gnosis/HOPR]:increasing number of issues around, specifically, the Ethereum RPC layer.

Sebastian Bürgel [Gnosis/HOPR]:And, I mean, just to take for a second a step back, right? How do we get, actually, many people to use Ethereum? Well, by Ethereum-based apps, right? And these app developers, unfortunately, they don't touch the network layer, they don't touch the consensus layer.

Sebastian Bürgel [Gnosis/HOPR]:And as a networking guy, I kind of hate to say that, but that is the reality. They only touch the Ethereum RPC layer. So if that one is kind of critically broken, then Ethereum is in a pretty rough position, and that is, as I outlined in more details in the post, the situation right now.

Sebastian Bürgel [Gnosis/HOPR]:So, I think we made pretty cool progress to get us into a better, space, so shout out, obviously, to ETH PandaOps for the Hive test suite and so on.

Sebastian Bürgel [Gnosis/HOPR]:But we need more than that, right? So if we look at the IF test right now, there's around 190

Sebastian Bürgel [Gnosis/HOPR]:only on the entire Ethereum RPC layer, which is basically spinning up kind of a fake blockchain of only a few blocks length, and then do, like.

Sebastian Bürgel [Gnosis/HOPR]:Like, 4 or 5 tests on them.

Sebastian Bürgel [Gnosis/HOPR]:for ETHAT logs, for example, specifically, which is the example that I picked here, but I really only mean it to serve as an example of just how dire the situation around the RPC layer is.

Sebastian Bürgel [Gnosis/HOPR]:And… so specifically.

Sebastian Bürgel [Gnosis/HOPR]:We see clients where, logs just get swallowed and omitted. So, you know, you as a user don't even know that you're missing state, which is a kind of catastrophic failure case for an Ethereum-based application that doesn't have any additional information.

Sebastian Bürgel [Gnosis/HOPR]:And, yeah, on the one hand, you could say, well, okay, that's a client bug, just let clients sort that out. But I find it more interesting, and that's why I posted this blog here.

Sebastian Bürgel [Gnosis/HOPR]:Because in the process of trying to get to the bottom of that, a few things were sticking out. So the first one is, this was not caught by any hive tests.

Sebastian Bürgel [Gnosis/HOPR]:Because they kind of test for very basic things. Again, which is helpful, but we need to grow by many orders of magnitude beyond that.

Sebastian Bürgel [Gnosis/HOPR]:And the second is, Hive tests are failing. I think it is fair to say that client teams, in many ways, don't care too much if they fail.

Sebastian Bürgel [Gnosis/HOPR]:Because otherwise, they wouldn't have been over 50% red for several months at this point.

Sebastian Bürgel [Gnosis/HOPR]:And the third thing is, when engaging, like, I talked to several client teams, and the interesting thing was, there isn't even consensus. We do not have consensus on how Ethereum is exactly supposed to work.

Sebastian Bürgel [Gnosis/HOPR]:Meaning, we're missing standardization, and yes, I'm aware that we have the ETH execution specs, which is… has been helpful in finding, kind of, tight mismatches and so on.

Sebastian Bürgel [Gnosis/HOPR]:But what I mean when I say that we don't have specs how Ethereum is exactly supposed to work, again, just as an example.

Sebastian Bürgel [Gnosis/HOPR]:from the ETHGetLogs front, it is currently unclear, and clients have in the past made individual decisions on

Sebastian Bürgel [Gnosis/HOPR]:what happens if somebody tries to, obtain locks, let's say ETH get locks, on the wrapped ETH contract from Genesis until tip of the chain, right? That's obviously labor-intensive, I guess, computationally expensive, and the response size might be huge.

Sebastian Bürgel [Gnosis/HOPR]:So individual clients made individual implementation choices, such as cropping the results, erroring, just taking a whole long time.

Sebastian Bürgel [Gnosis/HOPR]:or building kind of bespoke indexing solutions on top of it, which has not undergone any VIP, and it's kind of individual innovation, which happens in parallel for every client team.

Sebastian Bürgel [Gnosis/HOPR]:Which is clearly kind of suboptimal.

Sebastian Bürgel [Gnosis/HOPR]:So…

Sebastian Bürgel [Gnosis/HOPR]:Yeah, basically, what I'm saying here is, from my perspective, looking at this, we are lacking two things. One is, in orders of magnitude, growth of what Hive is today.

Sebastian Bürgel [Gnosis/HOPR]:to kind of bring it up to… in parallel, I see it somewhat similar to how the web platform tests work. So the browsers that many of you are using right now has undergone over 2 million tests, right?

Sebastian Bürgel [Gnosis/HOPR]:So Ethereum is certainly not much less complex, so we need

Sebastian Bürgel [Gnosis/HOPR]:Multiple orders of magnitude growth in these test cases.

Sebastian Bürgel [Gnosis/HOPR]:we need focus on the basic stuff, so if I look at what we discussed in this call.

Sebastian Bürgel [Gnosis/HOPR]:And if I look at the Hive test results, the cutting-edge stuff is kind of okay, right? So the Fusaka tests are kind of all okay.

Sebastian Bürgel [Gnosis/HOPR]:it is the basic, kind of, for example, RPC layer that is not okay.

Sebastian Bürgel [Gnosis/HOPR]:So, that is focus, and then I think we need additional focus on, well, getting to consensus of how Ethereum is actually supposed to work.

Sebastian Bürgel [Gnosis/HOPR]:Because when I do talk to client teams, I also hear, well, you know, Hive tests were basically just the test suite, so of course, NetherMind or, you know, any of the others is not one-to-one following that, and we have our reasons for that.

Sebastian Bürgel [Gnosis/HOPR]:Which is fine, right? I think many of these reasons are fine, but it just shows me we need to go through a consensus-finding exercise of how Ethereum is actually supposed to work on that layer.

Sebastian Bürgel [Gnosis/HOPR]:And yeah, that's, that's what I see here right now. Yeah.

Mario Vega:Thank you.

Sebastian Bürgel [Gnosis/HOPR]:Happy to take it forward.

Mario Vega:Thank you, thank you. So yeah, a lot of hands up, we can start with Marius.

Marius van der Wijden:Sure, yeah, I think this is a… this is a problem that we know about for a really long time, and, we…

Marius van der Wijden:We at the Ethereum Foundation have also started hiring, hiring someone for exactly this purpose.

Marius van der Wijden:But there have already also been a few proposals, from, for example, Jules.

Marius van der Wijden:On how to, how to, create a new…

Marius van der Wijden:REST-based API, in order to,

Marius van der Wijden:kind of, yeah, bring the RPC into the…

Marius van der Wijden:21st century, I would say. So basically.

Marius van der Wijden:when the RPC was created,

Marius van der Wijden:the… it was… it was basically created for something like Mist.

Marius van der Wijden:And, the problem is that it…

Marius van der Wijden:It's very hard to evolve it, because it impacts so many people in the ecosystem. And,

Marius van der Wijden:Yes, and also the standardization process is very…

Marius van der Wijden:difficult, because you kind of… like, RPC is just a second, smaller cousin, to the normal consensus, and so client teams are not prioritizing it as much as the normal consensus.

Marius van der Wijden:But as I said, I think… I personally think it's a big problem, and

Marius van der Wijden:I know people have been, kind of.

Marius van der Wijden:A lot of people have tried to make this a priority, and tried to make the standards, and have been burnt out doing that.

Marius van der Wijden:for example, like, I don't know, I think ETH Simulate is one of the… one of the examples where it worked, where we found,

Marius van der Wijden:found a way to find consensus between clients and create a good spec, but a lot of the other things are not.

Marius van der Wijden:I also want to go, like, a step further in this.

Marius van der Wijden:I think it would be good, because, like, as we all know.

Marius van der Wijden:nodes are not the same thing as they were 10 years ago. A lot of people are not running full nodes, or, like, archive nodes anymore. A lot of people are running with, with, history expiry.

Marius van der Wijden:and in the future, what a node…

Marius van der Wijden:Stores is going to be,

Marius van der Wijden:very different, so we will see more specialized nodes. And I think the RPC

Marius van der Wijden:Needs to reflect that, in order to make it clear that these are calls that happen on the head of the chain.

Marius van der Wijden:These are calls that require, access to the history, and so on. And the last point that I wanted to make regarding the log indexing.

Marius van der Wijden:So old has, also written an EAP,

Marius van der Wijden:About that, in order to get the…

Marius van der Wijden:log indexing that we built internally in Geth, to get it into the protocol. The problem here, again, is prioritization of, of that,

Marius van der Wijden:against other features, and I think that is, like, dev time is very finite, and, so, yeah. It's definitely an issue that is on top of mind for me, and, I hope to bring some resources from the Ethereum Foundation, in order to

Marius van der Wijden:In order to get that going, but in the end, it's clients who, who kind of need to implement it, and, yeah.

Marius van der Wijden:I think client teams are stretched already.

Mario Vega:Thank you, Maurice. Lukash?

Łukasz Rozmej:Yeah, I have quite a few comments, so…

Łukasz Rozmej:One, I joined the, JSON RPC call, actually, recently, and the temperature check I got there that the… they're working on a very, very small and basic RPC spec, that won't handle everything mentioned here, most of the things mentioned here.

Łukasz Rozmej:And actually, compared… comparing that to the correct node-deep limitations, nodes would be mostly already in that spec. Maybe there would be a very small,

Łukasz Rozmej:changes in the nodes, and the bigger problems they are actually focusing on is the, like, RPC providers are, like, general EVM compatibility of a train. So, for example, some L2s that have very custom and very,

Łukasz Rozmej:Not standard things doing. So that's one thing.

Łukasz Rozmej:So, to make this extremely well standardized on each client level.

Łukasz Rozmej:Yeah, that's probably what is the problem, to actually build the… the…

Łukasz Rozmej:money and motivation, because it's very boring work to build the whole spec, and then execute on it, and create tests for everything.

Łukasz Rozmej:Moreover, some things I disagree that should be the specs. So, for example, Marius mentioned the AT GetLogs index they built.

Łukasz Rozmej:We built a different one, we will be releasing it soon, but our is not provable, and… but our is faster, right? So, it has some upsides and downsides compared to other things, and I don't think everything should be standardized.

Łukasz Rozmej:I think some figs would be better to…

Łukasz Rozmej:Be left to implementation details, because, different nodes can then, have different advantages and disadvantages.

Łukasz Rozmej:Another thing, a bit of problem with the Hive test is, for us, is that,

Łukasz Rozmej:they are basically generated through GEF, so GEF is the spec, and while we agree that something between 95% to 99% of what GEF does is the best, and even if we don't agree, we most often align with what GEF does.

Łukasz Rozmej:There are very rare exceptions that it would be good to actually define the spec and potentially change something would get us to be better.

Łukasz Rozmej:And this is especially sometimes problematic for us if we return… the last thing is about the tests themselves being very basic. So, for example, in some rare cases, we want to return additional fields, because, for example, some of our users or customers requested it.

Łukasz Rozmej:But the test doesn't allow us to fail the test at that point, because the test says this should be the list of fields, and you cannot return anything else, which in JSON RPC, it would be fine to return something optional or something extra, but the test fails.

Łukasz Rozmej:So we have this kind of false, negative in terms of testing.

Łukasz Rozmej:And… okay, I had one more thing.

Łukasz Rozmej:But I think I lost it. Yeah, I know. So we have a list of failing Hive tests, and we actually are trying to, kick down some of them, and I'm seeing it, decreasing every week by a few.

Łukasz Rozmej:instances, so… this is something that Netherbyte is working on and is prioritizing, but…

Łukasz Rozmej:It is not as high a priority as, for example, all the consensus level.

Łukasz Rozmej:changes. So, yeah, it is lacking, and it will be lacking, compared to the core protocol of Ethereum.

Łukasz Rozmej:And yeah, one last thing. ETH Simulate was actually pretty well tested, and while still we failed some tests there, we are working on it. Maybe there will be some changes to the test here and there, if we decide on that. I don't think we found

Łukasz Rozmej:that examples, we had one or two questions about if the chest should be changed, but we decided to go with what GEF did, because both versions were fine, but, yeah, this is, again, circling back that GEF is the specs at the moment, which is…

Łukasz Rozmej:Sometimes not the best thing to have, and that's it for me.

Mario Vega:Thank you, Lukash. Felix?

Felix (Geth):Yeah, I mean, I strong… I want to respond, first of all, by strongly disagreeing that Gethis is back here, so the… the,

Felix (Geth):The official specs are the…

Felix (Geth):YAML files, which are contained in the execution APIs repository. And when it comes to the tests, it's very critical to say that, yes, we generate the test vectors using Geth, but the tests are not the spec. The spec is the spec.

Felix (Geth):And the SPAC is independent from… from the client. We… we use GATH because

Felix (Geth):For us, it was the easiest to set it up like this, and also because for most calls.

Felix (Geth):the, the, so the testing methodology that is used for RPC tests is that the, the,

Felix (Geth):Some RPC requests are performed, and the interactions between the test and the client are recorded. And then, we replay these interactions against another client, and compare the result.

Felix (Geth):the comparison of the results has some special cases. For example, for the errors, we do not match on the error message, but we do expect the error code to be the same. And I feel like this is kind of critical, because for the users, the message does not matter as much as the code.

Felix (Geth):So, but even most of the differences in the clients is in cases where, the…

Felix (Geth):requests submitted by the test is containing something invalid. So basically, we are testing the, like.

Felix (Geth):the, like, limits of the spec in the… in as few of the tests we have. So all of the tests where you are just, like, requesting

Felix (Geth):something in the normal way, they are passing. It's mostly about… like, with some exceptions, of course. But then, for the… for the… for example, with NetherMite, most of the differences are in cases where, for example, the request performs… has an invalid argument value.

Felix (Geth):And then, we expect the client to react with a certain error code, or we expect the client to detect this invalid argument, and instead, it either accepts it because the parsing is too lenient, or it returns a different error code.

Felix (Geth):And I feel like…

Felix (Geth):These failures, in some way, are less important than if it returned, like, a totally wrong value.

Felix (Geth):But, obviously, it would be good to align.

Felix (Geth):And for these things, I feel it's just a matter of agreeing on the correct error code. It's not so much, like, there isn't gonna be a deep discussion. There are some other differences, specifically between Geth and Nethermind, when it comes to the optional fields. For example, things like the total difficulty, we discussed it also recently.

Felix (Geth):And I have to say that

Felix (Geth):We have to… we still have to find a solution for that that works in the testing. I am not against, return… never mind returning the total difficulty, it's just that it's kind of hard to make it work with our current testing method, because the testing method is just to compare, the outputs, literally.

Felix (Geth):For some calls, we do not compare the outputs. Literally, we have another method that is about checking, basically type-checking the response only, and for some responses, this can work. For example, for estimateGas.

Felix (Geth):We do not require all clients to agree on the algorithm for estimating the gas usage of a transaction. The clients can actually each have their own method for that, so we only check that the result is a correctly encoded number, as is required by the spec.

Felix (Geth):So, It's,

Felix (Geth):I feel like the discussion is quite nuanced. Like, I personally am the author of most of the existing RPC Compet tests, and I try to choose them very carefully to exercise the behavior of the clients as

Felix (Geth):at the same time, I didn't want to introduce, like, hundreds of tests that would all fail, because when we got started, to be more serious about specifying the RPC with the execution APIs, and then also with the test suit.

Felix (Geth):It became clear that

Felix (Geth):is gonna be very demotivating for clients if they face, like, 300 failing tests, and they will just never get fixed. So we have been very careful about slowly expanding the range of behavior that we are testing. We started very small, tried to make a test suit that just contains, like.

Felix (Geth):50 or 60 tests that really, you know, are pretty much guaranteed to pass, and now we are at this point where, like, every so often, I will just add a couple more tests, just to see what's the behavioral difference in the client.

Felix (Geth):And then we can, you know, try to get them fixed. I did report over time also many of the failures to the clients, but

Felix (Geth):It just…

Felix (Geth):at some point, also, I gave up on that, because it just seemed like it's not exactly a priority, and some of the things just turned out to be very hard to resolve, or something. That said, there are some constructive things we could do, so we could try to…

Felix (Geth):Improve the testing methodology by allowing, for example, certain optional fields for certain responses, or things like that.

Felix (Geth):Or we could simply have multiple test vectors, which are allowable responses. The difference then is that we kind of have to manually check

Felix (Geth):The responses to see if they are actually correct.

Felix (Geth):And, this is not easy. So…

Felix (Geth):I don't yet know what's gonna be the answer, definitely we need to make some improvements, but yeah, it's been an uphill battle, and I'm actually pretty happy that, for now, for the basic, calls, we have,

Felix (Geth):quite some good coverage, also, of the edge cases. Forget logs, it would actually be good to add… introduce more tests, so Sebastian, if you have suggestions for that.

Felix (Geth):we have the repository where the tests are generated. It is very easy to actually add tests for this, and we could then very easily also verify the behavior.

Felix (Geth):Maybe as a final note, some of the clients are currently not even…

Felix (Geth):I guess for the RPC, all of the clients are somewhat able to load the test chain, but I see that, especially, for example, for RETH, we have a lot of failing tests, and…

Felix (Geth):I don't actually really know…

Felix (Geth):Yeah, I mean, we would really have to see, like, the client that's failing the most right now is Nimbus, actually, and I don't really fully know why, for example. Like, there are… some of these differences are, like, kind of hard to debug, so…

Felix (Geth):I guess we would have to… Yeah.

Felix (Geth):Again, I can only say that, like, the clients should really attend the RPC standardization calls, and then we can see, what kind of…

Felix (Geth):What kind of progress can be made on that?

Mario Vega:Thank you. Justin?

Justin Florentine (Besu):The next RPC call is in 6 minutes. So… get excited.

Mario Vega:Thank you. Alright, I see who had the…

Mario Vega:Lukash, do you want to go? We only have 5 minutes, though, but we can keep discussing this on the RPC call, so I think it's a great…

Łukasz Rozmej:Great thing that we…

Mario Vega:discussing this year.

Łukasz Rozmej:Fine, I added some things into the chat.

Mario Vega:Perfect, thanks.

Mario Vega:Felix, that was the hand up from the last time, or do you have something else to comment?

Felix (Geth):Sorry, yeah, I didn't lower, I'm sorry.

Mario Vega:Alright, thanks.

Mario Vega:Yeah, I think it would be nice if someone can share the link to the next call. It's gonna be happening in 5 minutes, we can keep the discussion running in there.

Mario Vega:So I don't think we have reached an agreement at the moment, because it's a very nuanced discussion, but…

Mario Vega:Justin suggested that we bring this up into ACDE, which I don't think is a bad idea, and also.

Mario Vega:Yes, just to close…

Mario Vega:for now, I think everyone that is interested in this discussion, please join the next call.

Mario Vega:And also, please, join ACD, and we can, also follow up the discussion in there.

Mario Vega:Anything else from this scythe?

Mario Vega:I think also from the specs perspective, what I can say is that, we have, this moment… I think at this point.

Mario Vega:we are reaching a point where we need a formal, a better specification model, perhaps. So, it might be worth to see if we can

Mario Vega:bring the ideas that we have put into yields on the consensus side of the specs into the RPC.

Mario Vega:Perhaps to make the, testing…

Mario Vega:language more expressive, and we can, even generate better tests with this. So, yeah, I think that discussion is still open. We can see if that would help in the long run.

Mario Vega:But yes, there's a ton of things to do. I don't think it's gonna be, like, very easy to get this going, but it's doable, definitely.

Mario Vega:All right, yeah, thanks, Justin, for the Zoom link to the next call.

Mario Vega:Alright. Anything else?

Mario Vega:That can be discussed in this call.

Mario Vega:Yeah, and thanks, thanks, Sebastian, for bringing up the issue and starting the discussion. I think this is a really important topic, thanks for bringing it up, and also thanks for the very well-written post in ED research. It's been a great, resource to read, and yeah.

Mario Vega:Yeah. Thanks, everyone, for joining the call. If there's nothing else, we can close the discussion here.

Marius van der Wijden:Alright, thank you.

felipe:Thanks, everyone. Bye.

Łukasz Rozmej:Bye.

Chat Logs

Louis:Replying to "gm" Gm

James He:We have a new release out so if people can upgrade 😅

James He:Prysm v6.1.2

Barnabas:Replying to "Prysm v6.1.2" blog is updated

Barnabas:Any other team’s got an update that we are missing?

James He:Replying to "Prysm v6.1.2" 👍

Barnabas:I guess LH rc 1?

Barnabas:Lh/besu/grandine do we expect a stable release before hoodi?

Ameziane Hamlat:Replying to "Screenshot 2025-10-13 at 16.06.16.png" For besu, we’ll have a new release this week to handle 60 mgas for Hoodi and mainnet

Barnabas:🦗

Mario Vega:https://discord.com/channels/595666850260713488/688075293562503241/1425748011438575719

Barnabas:https://github.com/eth-clients/hoodi/pull/23 https://github.com/eth-clients/sepolia/pull/119 https://github.com/eth-clients/holesky/pull/136

Barnabas:@Marius van der Wijden can I get your blessing?

Marius van der Wijden:Blessing granted

CPerezz:Nice work Kamil!

CPerezz:XEN is also bigger in Bloatnet. Though depth is expected to be the same TBH.. As avg depth of XEN is already 8-9. So we would need to bloat it insanely to make a diff

jochem-brouwer:Yes any worst case scenario for any client should be part of the benchmark test suite (so we can keep track of this performance over time). So if a client notices a specific scenario, please contact us, we will implement it in the benchmark suite :)

Louis:Replying to "Yes any worst case s..." Anyone who have an interesting case could also drop a comment in this issue! https://github.com/ethereum/execution-specs/issues/1577

Marcin Sobczak:AA40 over verificationGasLimit

Marcin Sobczak:https://etherscan.io/tx/0x1010c2cbd31e3c46e512670bfc41f995a4811fc1bbceb62363d4a712dabc1b10

raxhvl:Latest test results for release v1.2.0: 1. 🏆 Besu: 80/90 (89%) 2. Reth: 67/90 (74%) 3. Geth: 64/90 (71%) 4. Nethermind: 63/90 (70%)

Barnabas:stefan your mic is super echoy

draganrakita:All tests passing is good step before devnet

Raúl Kripalani:i think it's a different raul?

Jared Wasinger:Rahul are you running those tests with bal-devnet-0 branch of geth? From my side, 2 tests fail locally with the master branch as of 12 hours ago. One of these is one that I flagged as being an incorrect test.

Mario Vega:Replying to "i think it's a diffe..." Sorry yes I meant Rahul

Stefan Starflinger:Replying to "stefan your mic is s..." sorry forgot to connect my airpods in time

raxhvl:https://pokebal.raxhvl.com/

Mario Vega:https://pokebal.raxhvl.com/

marek:Replying to "https://pokebal.raxh..." I like the site name

felipe:Replying to "Rahul are you runn..." Can you confirm which test this is that you flagged

Mario Vega:https://ethresear.ch/t/ethereum-needs-standards-punk/23151

felipe:I think I see it, the SSTORE case?

Bumblefudge:I also think that eth community often thinks of solidity as the "business logic" layer and browser/wallets/clients/dapps as the "client layer", and that entire disjointed pile of chaos is entirely interacting just via RPC...

Bumblefudge:or maybe the clients care about hive, but if they are confident the test is wrong and incentives aren't there to fix the test for everyone...

Łukasz Rozmej:We have consensus on Ethereum as a protocol, we don't have consensus for RPC calls

Jared Wasinger:Replying to "Rahul are you runnin..." Yes

Jared Wasinger:Replying to "Rahul are you runnin..." it's the bal_sstore_and_oog with the oog_at_sstore case

jochem-brouwer:Antwoord verzenden naar "We have consensus ..." But RPC standardization is somewhat in the works right? RPC Standards call in 23 mins

Łukasz Rozmej:Replying to "We have consensus on..." very basic one

felipe:Replying to "Rahul are you runn..." Yep, just catching up there. Thanks.

felipe:We will try to get this in on a patch release early this week

Mercy Boma Naps-Nkari:https://github.com/ethereum/execution-apis/pull/650/files https://github.com/simsonraj/execution-apis/blob/527bb9af49ba12bbed37cc36d4cfe42048cd114e/src/extensions/README.md dropping the links for proposal on error codes standardisation, we need feedback from client teams and community in general in respect to this.

Bumblefudge:Replying to "We have consensus on..." that call isn't on the calendar, is it open to public?

spencer-tb:Can we make it an EIP? If the community care enough it should get shipped right?

Justin Florentine (Besu):there is a channel in the discord, and recurring calls to discuss as a working group

Mercy Boma Naps-Nkari:Replying to "We have consensus on..." https://github.com/ethereum/pm/issues/1758

jochem-brouwer:Antwoord verzenden naar "We have consensus ..." https://github.com/ethereum/pm/issues/1758 eth/pm issue for the call :)

Justin Florentine (Besu):Mercy is doing a great job running the calls, and it is an excellent entry point for Ethereum contributors.

Barnabas:Replying to "Can we make it an EI..." This !

Barnabas:Replying to "Can we make it an EI..." But it probably should be rest based at this point

Justin Florentine (Besu):Besu fleet mode is a very "near head" scenario

FLCL:During those few RPC related calls there was feeling we were not in sync enough with API consumers, who are not on such calls

Ansgar Dietrichs:also somewhat connected to what Marius said: As we scale more, especially with zkEVM soon, RPC nodes will need to specialize a lot to remain performant. So any next generation standards should be designed with awareness of that future

FLCL:Another thing that might be interesting to explore imho is to write hive tests by devs

Felix (Geth):@FLCL can you explain?

Felix (Geth):the tests are written by devs?

keri:We need client devs to attend the RPC meetings so we can have these conversations and get consensus 🙂

FLCL:Replying to "the tests are writte..." Me as dev don't do so 😅

FLCL:Like those devs who add features to a EL and mostly contribute to specific EL tests

Łukasz Rozmej:Tests are generated from geth, all error codes are in those files?

Justin Florentine (Besu):we should talk about it on ACDE - the problem is bigger than devs not being on the rpc calls - even when they are, the path from consensus on a call to a MERGED pr into a spec remains unclear

Barnabas:Replying to "we should talk about..." it needs to be an EIP, it needs to have a champion, and it needs someone that can’t burn out

Barnabas:Replying to "we should talk about..." its a thankless job, but can be done

Justin Florentine (Besu):Replying to "we should talk about..." you need to find someone who fits all that and also wouldn't rather work on something more interesting

Barnabas:Replying to "we should talk about..." The only way I can see us moving to an actual rpc standard if we block devnets on this EIP

keri:Replying to "Can we make it an EI..." I’d vote for simple changes first - consensus on RPC methods rather than a big change to rest

Łukasz Rozmej:Replying to "Tests are generated ..." So I assume not, so Geth is the spec ;) @Felix (Geth)

Barnabas:Replying to "Can we make it an EI..." sounds like 2 step pain vs 1 step pain

Sebastian Bürgel [Gnosis/HOPR]:Replying to "we should talk about..." It seems we're trying to fit a standards process into an existing EIP process. At least to some people who worked on standards, that doesn't seem to be the recommendable way to go: https://learningproof.xyz/lifecycle-of-a-blockchain-standard/

Sebastian Bürgel [Gnosis/HOPR]:Replying to "we should talk about..." (I know this is painfully meta)

Łukasz Rozmej:getLogs are problematic as they are very stateful

Sebastian Bürgel [Gnosis/HOPR]:It's impossible to test these flaky logs for large ranges. We don't have a public test repo for that.

Felix (Geth):it's not impossible!

Felix (Geth):we are in control of the test chain, so we can just generate a longer test chain

Sebastian Bürgel [Gnosis/HOPR]:Can someone share the link for the next call?

Justin Florentine (Besu):https://github.com/ethereum/pm/issues/1758

Mercy Boma Naps-Nkari:https://github.com/ethereum/pm/issues/1758

Łukasz Rozmej:Replying to "It's impossible to t..." we want to create a testing pipeline to sync multiple clients and do fuzz testing on eth_getLogs, but having hard time proritizing it, but should be done in not too much time

Justin Florentine (Besu):it's also listed on the Eth calendar, and in the json-rpc-api channel on the eth R&D discord

Łukasz Rozmej:Replying to "It's impossible to t..." compare eth_getLogs responses between clients

Justin Florentine (Besu):zoom link: https://www.google.com/url?q=https://ethereumfoundation.zoom.us/j/87943719720?pwd%3Dvq7rARW8AjizhE2bj1ds3oQM0OWDyQ.1&sa=D&source=calendar&ust=1760798516199411&usg=AOvVaw3jSKlyiBS7m4sp4nX76uFK

Felix (Geth):Replying to "It's impossible to..." Geth has a fuzzer for this, but we are also not running it against multiple clients

Łukasz Rozmej:Replying to "It's impossible to t..." but hardfork season is making resources stretched

Łukasz Rozmej:Replying to "It's impossible to t..." @Felix (Geth) we should do it then!

Łukasz Rozmej:Replying to "It's impossible to t..." Can you point us to it?

Felix (Geth):Replying to "It's impossible to..." I'm searching for link

Felix (Geth):Replying to "It's impossible to..." Will get back to you about this offline

Summary

11 highlights · 1 decisions · 3 action itemsExperimental

fork status and schedule

BPO fork on Holesky tonight, Sepolia tomorrow; smooth progress expected00:07:09
MEVBoost v1.10 Alpha 6 released with V1 API fallback support00:08:38
Named forks (Osaka, Amsterdam) removed from BPO genesis.json - CL side doesn't change00:13:25

testing progress

Stateful gas testing: 3 EL clients running on BloatNet; Nethermind shows variability00:16:49
Performance doesn't match mainnet observations; investigating caching and hardware factors00:17:36
Block-level access list tests: Besu 89%, Reth 74%, Geth 71%, Nethermind 70%00:29:18
EPBS DevNet Zero delayed past October; off-protocol payments discussion blocking progress00:33:41

rpc standardization discussion

RPC layer has only 190 tests; clients swallow logs causing critical failures00:35:03
No consensus on RPC behavior (e.g., getLogs limits); clients made individual choices00:39:56
RPC specs exist in YAML but Geth generates test vectors; clients often fail tests00:43:38
Client teams need to attend RPC calls; path from consensus to merged spec unclear00:49:08

Decisions

Remove named forks (Osaka, Amsterdam) from BPO genesis schedule when CL values unchanged00:16:02

Action Items

All validator operators: Update Sepolia validators to MEVBoost v1.10 Alpha 600:09:04
Client teams: Report worst-case RPC scenarios to execution-specs team for benchmark suite00:29:07
Client teams: Attend RPC standardization calls to reach consensus on behavior00:56:07