danceratopz:Perfect, thanks. Hello, today's the 15th of September 2025, and welcome to ACDT number 53.
Transcript
danceratopz:Mario is out of office today, so he's asked me to step up.
danceratopz:And, facilitate the call. I just dropped the agenda in the chat. And first off is, happy Mergeiversary. 3 years today.
danceratopz:We have the merge, so, everyone was watching the Panda ASCII out in their terminals happily.
danceratopz:Crazy to think it's 3 years ago, and 3 upgrades ago as well. So…
danceratopz:2 for Saka. First rule, we're gonna have some Fusaka DevNet updates.
danceratopz:And then we're going to talk about the new simulator that's landed on the HiveView page.
danceratopz:And, have a look at the EL consensus tests and what the state is there.
danceratopz:So first off, Fisaka DevNet updates. Barnabas, do you want to take the lead?
Barnabas:Yeah, sure. So, physical.NET 5 has started last week, Wednesday, and we've been going through a fork a day.
Barnabas:We did, for example, 256 epox later. There was no significant problem there, and then the first BPO, increased the blood, maximum blood, count from 9 to 12. Then…
Barnabas:The 12th?
Barnabas:Just one sec.
Barnabas:No, to 15, and then we went to 21, and now we are at, 13th Street, and tomorrow we're gonna go to 48, and then, on Wednesday to 72.
Barnabas:We currently see quite some orphaned blocks. This can possibly be caused by a few different things, that are being currently discussed in the interrupt channel.
danceratopz:I could go into.
Barnabas:off…
danceratopz:Is there anything else you want to add for DevNet5?
danceratopz:Omnibus.
Barnabas:Yeah, so one of the main issues that we had is the MEV machine, had the wrong rate limiting, so none of the MEV blocks,
Barnabas:got included in time, and that was a misconfiguration from our side. So hopefully, we just fixed this issue, like, maybe half an hour ago, so hopefully we're gonna be able to see less orphan blocks, built by the, Navy builder.
Barnabas:Another thing, Nimbus, reported some issues, possibly with the DKSCG library. Maybe, someone from, the Nimbus team can give us an update regarding that.
Dustin:Sure, yeah. So, we're, we're not… I want to say totally…
Dustin:sure that… that… where the, issue… issue lies here. But, this… oh, let me say the symptom is that
Dustin:If you look at the… the data column,
Dustin:the verify cells, proofs, function, something named like this, is… is in, inconsistently, apparently, returning, sort of a… a success or not success.
Dustin:response, in different clients, and Nimbus is, at least at times, not, validating a lot of these columns, but we'll…
Dustin:See, that… that's… that's…
Dustin:what exactly is happening there, and part of the issue is that there's, as is already being discussed in some things, different people are running, and different clients are running different versions of the CKCG library, and there have been changes about how strict that is over time, so that's one possibility. There are others. Anyway, yeah.
danceratopz:Okay, thanks, Dustin.
danceratopz:But is this the… I kind of don't quite have a context on this. Is this the issue that you've seen with Nevermind, that Nevermind are working on, or is this a different issue?
Dustin:No. No, those are… those… Sorry, yeah, that's a completely separate issue. That's their V-Org issue. So they say. I'll take their word for that, and… and in any case, no, it's a different issue.
danceratopz:Okay, thanks for clarifying.
Barnabas:I think for the Nethermind bug, maybe someone from Nethermind can give us an update.
Alexey:Yeah, so… Oh.
Alexey:Recently, we started getting errors related to, reorgs.
Alexey:It happened on Lighthouse and Dimbus nodes, so most likely the issue is on our side.
Alexey:Well, it's… when the…
Alexey:Consensus, client, sent us new payloads.
Alexey:But, on the same level, or the same block number.
Alexey:But, from different forks, sometimes, we, use, incorrect parent from different fork, for, the block,
Alexey:We are retrieving, for synchronization.
Alexey:And, it's fixable. We are testing the fix.
Alexey:And, it happens, due to reorgs, and probably,
Alexey:Like, mixed with, restarts, or something like that.
Alexey:Which leads, to having, Run parent for a block.
Alexey:will… we did not, see it happening a lot before. Maybe we just broke master.
Alexey:In some, By some other contribution.
Alexey:Not sure, but, Fixu seems to,
Alexey:be helpful, and I hope we will soon have it in master.
danceratopz:Okay, perfect. Thanks for the update, Alexi. I see there's some chat about,
danceratopz:the original bug that Dustin mentioned,
danceratopz:I think Justin mentioned that, it's not handling, point infinity commitments properly.
danceratopz:And… so, basically, Justin's already on it, as I understand.
danceratopz:And Dustin mentions that the problem is that Nimbus can't run the other six reference tests.
danceratopz:Okay, just a theory, says Justin.
Barnabas:There's one more thing. So, Teku had a bug just merged in yesterday, and they were also updating to 2.1.2.
Barnabas:I'm… that Bob was caught by our, think trainer. I'm not sure if that's related or not.
Enrico Del Fante (tbenr):Yeah, we were seeing the same problem, so… we actually, reverted
Enrico Del Fante (tbenr):the CQCG, that was our… our Java wrapper that was pointing to the very latest CKZG.
Enrico Del Fante (tbenr):Lib, and we reverted that.
Enrico Del Fante (tbenr):And because the latest version was actually able… enabling us to pass all the tests for the Alpha 6, we then reverted the…
Enrico Del Fante (tbenr):The enablement of those tests back to disable, so we can actually be,
Enrico Del Fante (tbenr):on the network, but not passing reference tests yet. That's the current… Situation.
Justin Traglia:Okay, I'll fix this in CKZG and publish a new release as soon as possible.
danceratopz:Awesome, thank you. Good news.
danceratopz:Okay, the other things I just wanted to mention about DevNet 5 is, we… well, it was launched, obviously, last Wednesday. We've had 3 BPOs so far, and the next BPO is scheduled.
danceratopz:For tomorrow morning at 6.32 UTC.
danceratopz:And I just wanted to add that we have some tooling now on the steel side.
danceratopz:And, Felix, managed to, confirm
danceratopz:that the clients were correctly configured via, Dano's Eveconfig, endpoint that he championed.
danceratopz:Using the new execute EAFConfig command. And, if you want to find out a little bit more, I just dropped a link in the chat to, a blog post on eveconfig that Felix added this morning.
danceratopz:To the documentation log.
danceratopz:Execute.
danceratopz:and thanks, Jay, for sharing the BPO schedule in the chat. That also states when the… Next VPO
danceratopz:Upgrade will go live.
danceratopz:Okay,
danceratopz:just in case you missed it, Watchtower's not ran for DevNet 5, so if you need your image to be updated for your client, please do reach out… reach out to someone, it won't be updated manually.
danceratopz:And, I saw some other chatter.
danceratopz:About some mystery node on the network called,
danceratopz:TYSM, and this is a diagnostic node from PRISM, so it's a modified… sorry, it's a diagnostic node from each PanDrops.
danceratopz:This is a modified PRISM node with some additional monitoring that, if Pandrop's added, so if you see those on the network, you know what you're looking at.
danceratopz:Is there anything else for DevNet 5?
danceratopz:Before we move on.
danceratopz:Okay, so, vanvas, did you want to mention anything about DevNet 3? Seems like it finalized, but…
danceratopz:It's now not finalizing again, as of about 4 hours, as far as I could see.
danceratopz:Is, worth going into that now, or…
Barnabas:Oh, I didn't know. We can take a look.
Barnabas:I guess, we're doing another non-finity test.
Barnabas:This… yeah, we… we don't plan to do this, so I will have to take a look at what's happening.
danceratopz:Okay, yeah, no problem. Devnet 3, Pavan. I was just mentioning DevNet 3. DevNet 5 is fine.
danceratopz:Okay, Paris says DevNet 3 was unplanned.
danceratopz:Okay, so we don't need to worry about definite 3.
danceratopz:I thought we were…
Barnabas:No, it was unplanned.
Barnabas:Not planned.
danceratopz:Okay.
Barnabas:We should worry about that industry, of course.
danceratopz:Yes.
Matthew Keil:Yes, please, because we are actively using that as a testbed, and I think some of the other teams are as well.
Parithosh Jayanthi:Yeah, it wouldn't be… we were, I think, at 70-ish percent, so it could just be that there was, some client push, and Watchtower just deployed an update that led to a restart loop. We can have a look and make sure that that isn't the case.
danceratopz:Yeah, so it was at, 78% as of 4 hours ago, and then it dropped off to about… just under, so, like, 63… 62, 63%.
danceratopz:But yeah, it's quite new, just 4 hours ago.
danceratopz:But I guess the main focus at the moment is DevNet 5 for everyone.
danceratopz:The last definite topic I'd have for Fusaka would be if there's any updates on Shadow Fork testing, or whether…
danceratopz:That's not in pro… process… progress? Not on the go at the moment, no.
danceratopz:Okay, if there's no shadow fork testing update, then…
danceratopz:I'd go to the next agenda item, which is the fact that we've got a new simulator on Highview since last Wednesday.
danceratopz:Called ConsumeSync.
danceratopz:I think Felipe would like to… Briefly introduce what it does.
felipe:Sure, yeah.
felipe:Yeah, I can give a brief update.
felipe:So, I posted something in the chat that… that if you're familiar with Consume already.
felipe:This basically builds on top of it, and what we wanted was a way to, verify client sync after certain tests.
felipe:And right now, it only exists for,
felipe:the block RLP limit, for Osaka.
felipe:And, so basically, we parametrize whatever clients are set up in the HiVML file, twice.
felipe:And so… For each client under test, you'll end up syncing to every other client, including itself.
felipe:After these payloads are sent, via…
felipe:the engine API. And so, just like in Zoom Sync, we send payloads over the engine API, we start up the sync client.
felipe:And we use the Hive boot node and, admin at peer to… Establish syncing.
felipe:And we make sure…
felipe:Syncing is kept alive, and we just verify at the end of each of these tests.
felipe:And so this is kind of a nice way for us to sanity check, syncing and, the state route after… after some… some test payloads.
danceratopz:Awesome.
danceratopz:Thanks a lot for the update.
danceratopz:So this was mainly introduced to test the, RLP block limit, right? Or the block RLP limit.
felipe:Yeah, for now. So this is a… this is a new marker on the… on the east side, and so we can add this…
felipe:Either locally, if we're testing Hive for sanity checks, or we can also just… this will run on the Hive dashboard as well for the few tests that we turn it on for.
danceratopz:Awesome, thank you.
danceratopz:So, the last Fusaka point is a quick…
danceratopz:overview of the execution layer consensus tests. I think Spencer would like to…
danceratopz:Give a quick overview here, what's the state?
spencer-tb:Yeah, sounds good. I'm gonna share my screen just briefly, if I can.
spencer-tb:You guys see… see what's going on?
danceratopz:Yep, we can see it, thank you.
spencer-tb:So, yeah, I basically just wanted to say that,
spencer-tb:we're now running the Ethereum test, general state test, so these are all the older tests
spencer-tb:Not the tests that we wrote, but we have refilled them all for a sec- for a second now.
spencer-tb:And I mentioned in the last call that we added them to the latest release, but we're now running them in Hive, and it looks like all the clients are doing a really good job.
spencer-tb:All these fails are kind of non-critical fails, and can be tweaked, quite easily.
spencer-tb:On the east side, on our side, and also on,
spencer-tb:Yale side, so yeah, to be honest, I just wanted to say that, it feels… Feels good,
spencer-tb:And yeah, also thanks, like, to all the EL clients for being really responsive, over the past, couple weeks.
spencer-tb:Yeah, and…
spencer-tb:On our side as well, we'd like to finalize our EIP checklist for DEL tests. We're nearly there.
spencer-tb:Personally, I'd like to add some more BPO tests.
spencer-tb:And maybe a few more EIP7918 tests, yeah, on the spec test side, on the EL side.
spencer-tb:Just wanted to give that brief update. And, yeah, ConsumeSync here is added to these…
spencer-tb:are failing right now, I think mostly for… from timeouts. It's a new simulator, so…
spencer-tb:Hopefully a quick fix, but yeah, that's it.
danceratopz:Great, thanks a lot for the update.
danceratopz:So the, the Ethereum general state tests that are now included in the release, these tests have been moved from Ethereum tests to execution spec tests.
danceratopz:And they're being filled by the new framework, no longer by retestees, so…
danceratopz:These tests have sort of been migrated to the
danceratopz:to the new system. So there's some unification there, but the… they haven't been ported.
danceratopz:the JSON and the YAML, test format has stayed the same, but we read them and we fill them using the execution spec test framework.
danceratopz:Okay, so I think that's it for Fusaka, unless anyone else has something to add.
danceratopz:Okay, moving on to scale DI1, gas limit testing. I can just very briefly mention on our side,
danceratopz:As in from steel and execution spec tests. We have some ongoing work on a new test wrapper format, which basically will just improve test developer experience, because we'll, like, abstract out the EIP7825 transaction gas limit.
danceratopz:So you don't need to worry about that when you're… when you're trying to fill up your… your block. The framework will handle that for you.
danceratopz:I think this work is quite close to completion.
danceratopz:And… Secondly, some work is ongoing, regarding, sort of validating or verifying a bloatnet testing approach using
danceratopz:The east execute command.
danceratopz:Apart from that, Camille is out of office at the moment.
danceratopz:If anyone else from Nevermind would like to give an update, please… Do step in.
danceratopz:failing that, Jochen… I think would like to briefly mention the…
danceratopz:benchmark tests that he's been writing based on the ZEN contract, and using mainnet state.
jochem-brouwer:Yes, yeah, I want to quickly say that we are currently setting up a test network.
jochem-brouwer:to, well, we have seen that, mainly we have seen some slow blocks, and these are all related to Xen, so Xen is, like, a super big state contract.
jochem-brouwer:And what we are trying to do is we are researching what happens if we raise the limit to 60 million.
jochem-brouwer:And then try to get, like, okay, what's the worst case behavior if we now do the same scenario, but now not for the 45 million guest limit?
jochem-brouwer:For the 60 million estimate.
jochem-brouwer:And that's still… well, it's… the tests are being written, and the network is, well, we are setting this up.
jochem-brouwer:And we hope to produce, results On this, like, this week.
danceratopz:Thank you. So, I think, Joachim's mainly interested in creating tests that touch a lot of state to stress test that scenario. So, if anyone has any ideas on other scenarios to add for that, then please
danceratopz:Feel free to add them, reach out to Jocham.
danceratopz:I think that would be it for the gas limit testing update. Moving on to Glamsterdam.
danceratopz:block-level access lists. So, they had breakout number 2 last Wednesday, which is live on YouTube.
danceratopz:The agenda is linked in the notes.
danceratopz:I'll just add quickly that, there was a new, test vector release last week.
danceratopz:With… which fixed some, issues found in both the specs and the tests.
danceratopz:So I'm not sure if Tony's here and would like to give a brief update on the…
danceratopz:State block lists outfit, access level lists.
danceratopz:I'm not sure if he's in the call.
Toni Wahrstätter:Yes, I'm on the call.
danceratopz:Perfect.
Toni Wahrstätter:Yeah, there are not many updates. This week, next breaker call will happen Wednesday next week.
Toni Wahrstätter:We're currently still planning to do the definite end of month, and we are right now waiting for the first CL implementation, such that we
Toni Wahrstätter:Can test it locally.
Toni Wahrstätter:And I think Terrence already said that you will do that.
danceratopz:Excellent. Sounds good.
danceratopz:From EPBS, I don't have much to say. I think, as far as I could tell, the last breakout was on the 29th of August.
danceratopz:Not sure if POTUS is in the… in the room, wants to share something, or someone else, about EPBS.
danceratopz:Okay.
danceratopz:It would seem that there's no EPVS updates this week.
danceratopz:The only other thing that I would… Mentioned briefly is that…
danceratopz:Reth had a mainnet stall, was it last week, or the week before? I've lost track.
danceratopz:But there's an excellent post-mortem for it, and…
danceratopz:I can chat in the chat really quick.
danceratopz:I'm sure everyone's seen it already. And, of course,
danceratopz:We, on the execution spec test side, we would like to be able to catch this kind of failure, obviously, and we don't at the moment.
danceratopz:I would just add that Mario's already added a test for this specific case.
danceratopz:But triggering it via a reorg scenario in the engine simulator is still a work in progress. So we don't really run through reorg scenarios so far, we're just validating payloads. So this needs a little bit of work, but we also hope
danceratopz:That, that this can be covered via the spec tests.
danceratopz:I saw that, Ref already,
danceratopz:Added end-to-end testing to prevent this in the future on their side. But of course, it'd be nice to have it.
danceratopz:Somewhere where all clients can profit from.
danceratopz:Did anyone want to… did anyone from the ref side, or did anyone else want to mention something about that?
danceratopz:Otherwise,
danceratopz:Yeah, yeah, it's very detailed. Thank you, Roman. It should be… should be in the post-mortem. Yeah, thank you.
danceratopz:Yeah, it's an excellent document. Okay, if…
danceratopz:If anyone else has any other topics, then please shout up, or if I've forgotten something, please let me know.
danceratopz:I think we've gone through the agenda.
danceratopz:PK.
pk910:Yeah, I got one thing to talk about. It's, Fusaka-related again. Eth calls,
pk910:We have the new, gas limit for, per transaction in a block, but, we agreed to not apply that limit on ETH calls, as that's a read-only
pk910:method, that's used by UIs a lot.
pk910:And the situation right now is that get, arrogant, and Red, ensure or enforce a limit there.
pk910:Which is, yeah, with just a new, gas limit at 16 million.
pk910:Even worse is that they're,
pk910:even if no limit is set in the RPC call, they are somehow taking a limit of 50 million, and then failing further down the line, because it's higher than the new EIP
pk910:Yeah, gas limit. And that's a regression from what we had before FULU, and should really be fixed before any release, because it's heavily, heavily used by UIs.
pk910:And yeah, it affects, Geth, Aragon, and Rath right now. I've posted a message in, execution Devs.
danceratopz:Amazing, PK, thank you very much. I just dropped a link to the Discord chat.
danceratopz:Okay, she wanted to go there directly?
danceratopz:Anything else anyone has to mention?
danceratopz:Okay, perfect. Then, thanks everyone for coming.
danceratopz:And, have a good week.
danceratopz:Happy hacking.
jochem-brouwer:Thank you.
Toni Wahrstätter:Bye-bye. Thank you. Bye-bye.
felipe:Thanks, Steph, bye.
Louis:Thank you.
Chat Logs
00:05:08
danceratopz:Agenda: https://github.com/ethereum/pm/issues/1719
00:05:27
Justin Florentine (Besu):oh snap!
00:05:37
Trent:Happy merge day ❤️
00:06:54
J Sunnyside Labs:This might be useful
00:08:09
Justin Traglia:I just started looking it btw.
00:09:19
Justin Traglia:I think it’s not handling point-at-infinity commitments properly. Something in ckzg changed here in the most recent release. Should be easy to check.
00:10:55
Dustin:jtraglia: the issue here is Nimbus can't run alpha.6 reference tests at all (or at least all of them) without 2.1.2
00:11:08
Dustin:so we can't practically revert
00:11:46
Justin Traglia:Just a theory
00:11:50
Enrico Del Fante (tbenr):We reverted ckzg lib and redisabled latest kzg ref tests
00:12:10
Justin Traglia:Replying to "We reverted ckzg lib..."
Oh, because of this issue?
00:12:15
Dustin:Enrico: in response to this, or just in general?
00:13:14
Barnabas:commit that broke teku: https://github.com/Consensys/teku/commit/fe5753d0659339463f969c18c812119ecededbbd#diff-ccf0742627eacc56b79b1664cc9b996a4d916268aa4d429ab286ffe5e0df86f9R39
00:13:41
danceratopz:@felix314159 verified correct EL client configurations pre-launch via eth_config using a new execution-spec-tests command, more info:
Blog: https://steel.ethereum.foundation/blog/blog_posts/2025-09-15_eth-config/
Docs: https://eest.ethereum.org/main/running_tests/execute/eth_config/
00:13:52
Enrico Del Fante (tbenr):Replying to "Enrico: in response ..."
That’s general I think. If you use ckzg lib
00:15:53
pawan:On devote 3 or 5?
00:16:00
Parithosh Jayanthi:Devnet3 was unplanned non-finality
00:19:48
Justin Florentine (Besu):seems straightforward and useful, thanks for adding this to Hive!
00:20:18
spencer-tb:https://hive.ethpandaops.io/#/group/fusaka
00:24:10
Louis:EIP7825 x Bencharking PR: https://github.com/ethereum/execution-spec-tests/pull/1945
00:27:21
Justin Traglia:Oh sorry distracted. I’ll share the recap here in just a minute.
00:27:29
danceratopz:https://laced-king-de5.notion.site/Incident-Post-Mortem-Reth-Mainnet-State-Root-Mismatch-26732f2c348480dea8b8c2a8753696dc
00:28:02
Justin Traglia:Replying to "Oh sorry distracted...."
ePBS breakout call recap: https://x.com/JustinTraglia/status/1966554374018396235
00:28:42
Roman:everything should be in post mortem
00:28:46
Roman:no additional comments
00:30:26
danceratopz:https://discord.com/channels/595666850260713488/688075293562503241/1416131247251521556
00:30:58
Christine Kim:thank you!
00:30:59
Alexey:ty!
Summary
14 highlights
· 1 decisions · 3 action itemsExperimental
Summary
14 highlights · 1 decisions · 3 action itemsExperimentalfork status and schedule
critical infrastructure
client updates
- Nimbus: verify_cells_proofs inconsistent across clients; related to ckzg library versions00:07:43
- Nethermind: reorg bug with incorrect parent selection from different forks; fix testing00:09:41
- Nimbus can't run alpha.6 tests without ckzg 2.1.2; Teku reverted and disabled tests00:11:08
- Justin Traglia: Point-at-infinity commitments not handled properly in latest ckzg release00:13:03
testing progress
- Felix verified EL configs pre-launch using new eth_config command in execution-spec-tests00:13:41
- ConsumeSync simulator live on Hive; tests EL sync for EIP-7934 RLP limit00:18:02
- Fusaka dashboard: ~20K tests for PragueToOsaka/Osaka/BPO transitions; mainly non-critical fails00:20:54
- Ethereum general state tests migrated to execution-spec-tests framework and running in Hive00:21:22
Decisions
- EIP-7825 transaction gas limit not applied to eth_call read-only operations00:21:53