Barnabas:Thank you.
Transcript
Barnabas:Welcome, everybody, to all core devs, testing call number 52. Fahil, could you please mute?
Barnabas:Thank you.
Barnabas:We're going to go into, physical right away. Physical.net status updates,
Barnabas:Definitely has a pretty high participation rate right now. We are sitting at about 80… 82%, which is not…
Barnabas:Great, but not horrible.
Barnabas:re… seem to have a few issues with a few specific clients, especially Nimbus and Aragon.
Barnabas:And, we have a couple of issues with, decorath and Nimbus EL and grandine.
Barnabas:Ideally, we want to see some fixes from Nimbus and Ergon before we…
Barnabas:going to the next unfinity period, we want to launch that, later today, or latest, by tomorrow.
Barnabas:So, first of all, we would like to get some information from these two clients, what are the issues, and if there's any fixes coming in.
Barnabas:Is anyone from Nimbus on the call?
Barnabas:Or is anyone from Eragon?
Andrew Ashikhmin:Yes, I'm here, so… We don't have a…
Andrew Ashikhmin:fix yet, and probably the issue will take some time to investigate. I think what's happening is that there is a flaw in Aragon's logic on long unwinds, and we sometimes deem a block invalid while it's
Andrew Ashikhmin:still valid, but we need more time to investigate and debug the issue. Probably will first add some more logging, things like that, but it's a tricky issue to debug, so don't expect a fix today, unfortunately.
Barnabas:Okay.
Andrew Ashikhmin:On the upside, I've made a patch to fix the last remaining,
Andrew Ashikhmin:Test failure on the Fusaka DevNet 5, Hive dashboard.
Andrew Ashikhmin:There is at least that.
Barnabas:Yeah, that's… that's great. We also want to make sure that Hive tests are 100% covered by every single EL.
Barnabas:I guess, nobody from Nimbus is around to discuss their box. Anyone from TECU could comment on, the red box?
Enrico Del Fante (tbenr):Oh, so harrowing it.
Enrico Del Fante (tbenr):So, we are still, working on…
Enrico Del Fante (tbenr):One sinking issue, one of the errors that has been reported in the… in one of the issues that is part of that document summary.
Enrico Del Fante (tbenr):So we lately, found a couple of things to improve and bug fixes around the way we were throttling requests from RPC that was causing a lot of issues with this.
Enrico Del Fante (tbenr):peer that was asking for 512 columns. We also fixed the validation of that, because by spec, that thing should have been rejected.
Enrico Del Fante (tbenr):We have already merged the PR for that. So, at the moment, the TECO boxes, even if they are… there are a couple of PR needs to go in to improve a situation, they're actually looking good, except for one that is…
Enrico Del Fante (tbenr):Seems like he's having memory issues, and this is REF2.
Enrico Del Fante (tbenr):It was… It's been in sync for 4 days, and waiting for us to keep up.
Enrico Del Fante (tbenr):Apparently, Ref never… Keep up, and
Enrico Del Fante (tbenr):Suddenly, while waiting, we had some additional memory issues. It seems, like, unrelated to the previous memory issues that we had, so I'm investigating.
Enrico Del Fante (tbenr):At the moment, dumping and downloading and checking.
Enrico Del Fante (tbenr):Yeah, that's it.
Barnabas:Thank you. Anyone from REST?
Barnabas:Could you give us an update?
Roman:On the document itself, The document, mentions the…
Roman:main net sinking bug, and we're gonna have the release for it, pretty soon.
Roman:The only thing that I can say is that it is reorg-related.
Roman:It never occurred on… on DevNets.
Roman:So, hopefully, it should not be a blocker, but it would be better to include the new police version
Roman:In DevNet 5.
Roman:And, regarding the… rough.
Roman:a cool client perp, Barnabas plugged it to me, I'm also gonna take a look today.
Barnabas:Thanks. And, everything is already fixed in, the main branch, in the trunk branch of graph, right?
Roman:Yeah.
Barnabas:If they go, I have the DevNet5 with the trunk branch, then that should be all good. We don't need to wait for a release.
Roman:Definitely.
Barnabas:Alright, I think that's all the known bugs, at this point.
Barnabas:What we need to discuss next is, possible timelines for .NET 5.
Barnabas:We wanted to do another non-finity on .NET 3, and see how quickly we can recover from that first, and then we were gonna go into, .NET 5 planning.
Barnabas:But I think, we're gonna need to start parallelizing these two, so ideally, I would like to get a thumbs up from every single client if they think that their trunk branch is ready for, DevNet 5.
Barnabas:Gundine?
Barnabas:Is anyone from Grandin? Grand?
Saulius Grigaitis | Grandine:Yeah, yeah, so we think that we fixed all the things, all the issues we had with syncing, and everything is merged to develop a branch, so I think we are ready now.
Saulius Grigaitis | Grandine:And, I think all the nodlings now in the DevNet 3 is sync them.
Barnabas:Thank you. Prism?
Manu:Yeah, so, the only… big missing feature right now in Develop is the backfill.
Manu:A part of that, we are still continuing to merge some performance improvement into the develop branch, but yeah, we are ready for DevNet 5 in our develop branch.
Barnabas:Thank you. Taeku?
Enrico Del Fante (tbenr):I think we're fine within the current state. We can have… so the bug that we were talking about before, I think it is not a blocker.
Enrico Del Fante (tbenr):Running it.
Barnabas:Thank you. Lighthouse?
pawan:We should be good, too. We currently have, like, the custody backfill thing as a PR, and we'll be reviewing it this week, and hopefully merging it soon, but like you said, that shouldn't be a blocker.
pawan:We also have another peer with a different sync strategy, and that is under review right now, but yeah, that one should also not be a blocker. We basically want to run our unstable branch and the other sync strategy
pawan:branch at this, like, half of our… half of the lighthouse notes with each. When you do the non-finality event, we want to observe how, like, unstable
pawan:Catch us back, and…
pawan:up to head, and the other sync strategy also catches to head, so we don't want to merge that PR yet until we see how it works out.
Barnabas:Yeah, that sounds good. We can organize, that… probably later this week.
Barnabas:Thank you, Doctor.
Matthew Keil:Oh, we identified a memory leak in some of the DB work that we did last week. We've got it figured out, we're in the process of testing it, hoping to get that merged tonight. Shouldn't stop us from getting onto the,
Matthew Keil:DevNet, though, with our trunk branch. We do have a pretty substantial refactor, to clean up some technical debt, that we're also trying to get merged, tonight.
Matthew Keil:So we can be ready for DevNet 5, but again, that also is not a blocker for… for DevNet 5. We can run locally, with some of our nodes if,
Matthew Keil:you know, push comes to shove, but we'll try to get it, merged ASAP.
Barnabas:Sounds good.
Barnabas:Anyone from Nimbus, managed to join?
Agnish Ghosh:Yeah. So… We have a blocking issue, that's happening…
Agnish Ghosh:because of, the way GetBlobs works in our codebase, so… It's… it's sort of,
Agnish Ghosh:A bug in the block validation pipeline?
Agnish Ghosh:Happens only for super nodes for now.
Agnish Ghosh:We are trying to resolve that issue by tonight. I have actually pushed
Agnish Ghosh:some of the fixes, so… in a couple of hours, I think the Watchtower should be picking it up.
Agnish Ghosh:Apart from that, we are yet to ship our final version for the sinker.
Agnish Ghosh:However, I think we should be fine.
Agnish Ghosh:If we are planning to do another non-finality test, in DevNet 3.
Agnish Ghosh:about DevNet 5,
Agnish Ghosh:we are ready, but I… I'm not super sure about the blob count. Where do we tip over? So yeah, like, unless and until that happens,
Agnish Ghosh:We… it's kind of a blind spot for us.
Agnish Ghosh:Right now.
Barnabas:Is there any update regarding your trunk branch readiness? Is everything still in HR branch?
Agnish Ghosh:Oh, right. So, for the trunk branch readiness, the update that I pushed
Agnish Ghosh:couple of, minutes ago, it should be picked up by Watchtower, and soon after, we would be merging that into our trunk branch. Once that is done, our trunk branch must be ready, so let's say by tonight.
Agnish Ghosh:Or maybe tomorrow early morning, the trunk branch should be ready.
Barnabas:Sounds good.
Barnabas:Anyone from Gaff around?
lightclient:I… yeah, I think we're good for definite 5.
lightclient:The main thing we're working on at this point is just handling blob conversion around the migration, but that's not really a blocking issue.
Barnabas:Sounds good. Breath?
Roman:Yeah.
Roman:Wave received ready from the mainfront.
Barnabas:Thank you. Never mind.
Łukasz Rozmej:I think we should be fine.
Barnabas:Thank you. Aragon.
Andrew Ashikhmin:Well, as I mentioned in the beginning, we still have this bug that, we, like, probably mishandle along reorgs, but it'll take us some time to investigate it. Other than that, I think our main branch is ready.
Barnabas:Thank you. Nazu?
Justin Florentine (Besu):Yep, we're on Main.
Justin Florentine (Besu):Thank you.
Barnabas:Okay, so it sounds like, we should be good to go with the non-finals event,
Barnabas:Maybe by end of today, and then we're gonna try to do it for at least one day, and try to recover, and then we can probably schedule Debnet 5 on the second half of this week. As soon as we have,
Barnabas:The time set, then we're gonna announce it.
Barnabas:it's gonna be a very big network in order to DevNet 4, and hopefully things will go smoothly there.
Barnabas:Next topic will be shadow fork testing. So, parallel to doing .NET 5, we are planning to look into doing Shadow Forks for
Barnabas:all the different testnets. Currently, we have two main blockers from gas and ergon, due to the fact that,
Barnabas:then they don't have a way to override the blob schedule values. Therefore, scheduling a shadow fork is currently not possible.
Barnabas:on Eragon and Geth.
Barnabas:I know that, Matt is working on the graphics,
Barnabas:Is there gonna be a fix for this in Ergon as well?
Andrew Ashikhmin:Well, no one is working on it yet, but…
Andrew Ashikhmin:Probably be a quick change. Okay, so when do you need it?
Barnabas:Probably by the second half of this week, ideally.
Andrew Ashikhmin:Yeah, okay, yeah, we'll try to do it, yeah.
Barnabas:Sounds good.
Barnabas:And regarding, releases, do client teams feel like they would be ready to do,
Barnabas:releases for Hosky, or technically, we don't need an actual release for Hosky, but it would be nice if
Barnabas:We could have, like, a pre-release… Maybe by next week, Monday?
Barnabas:Because, Hosky wasn't part of the…
Barnabas:all testing schedule, that's why I'm pushing for such accelerated timeline.
Barnabas:I don't think we have decided on timestamp yet.
Barnabas:I think, we can discuss timestamps async, and maybe make a decision on the call on Thursday.
Andrew Ashikhmin:For Aragon, it's problematic, because we are trying to wrap up our 3.1 release, and we'll need to cut another release, 3.2 for Holesky, because 3.1 doesn't have all the Fusaka changes, and it's basically,
Andrew Ashikhmin:we… we're constrained on the testing side, and so on. So, yeah, a week from now is too soon.
Barnabas:Okay, what would be the preferred date?
Andrew Ashikhmin:In two weeks would be better.
Barnabas:Correct.
Barnabas:What about other, like, how do CL teams feel about it?
Barnabas:Yeah, so we're gonna be trying to accelerate DevNet5 as soon as possible to be able to determine the BPO values.
Barnabas:Hopefully by next week, then we're gonna have that, and then a week after that, we should…
Barnabas:be able to make, current releases, that should fit also the schedule of Aragon.
Barnabas:So I think if we can aim for, plant releases by the 22nd of September, then,
Barnabas:We should have all the BPO values determined by them, and, should be all set.
Barnabas:Again, and if we don't have, like, a full-blown release, I don't think that's gonna be such a big deal, because Holzky has a pre-known validator set, so we can accelerate the deployment and have custom deployments as well.
Barnabas:We can just build a… tagged, locker image as well.
Barnabas:Alright, so I think we discussed, demo 5, we discussed the non-finalty.
Barnabas:timeline, maybe we can move on to gas limit testing. Anyone from Netherland could give us an update? How is that going?
Łukasz Rozmej:I don't see Camille on the call. Martin, do you have anything to update?
Marcin Sobczak:Yeah, so we are running EST test, and right now, we have a tooling to run it,
Marcin Sobczak:stateful, we did some experiments in Hudi, and, right now, I'm trying to, to run it on, on MyNet, and, one of my,
Marcin Sobczak:next, next steps is to, run, Zen, test, prepared by, Johim, and, try it on, on, top of MyNetDB.
Barnabas:Okay, thank you.
Barnabas:Regarding, Glamstone testing updates, is anyone, Tony is around, maybe? Or, look, level access?
Barnabas:Or anyone else?
Barnabas:Could you give us an update, how's that going? How's the specs coming along, and if we could be ready to have the first,
Barnabas:Well, 7.0 launch by the end of the month, maybe?
Jared Wasinger:Yeah, so I can't really comment on that, but I can say that regarding testing, I know that the testing team, recently made an initial release of blockchain tests for block-level access lists, and…
Jared Wasinger:In testing them against Geth, we found…
Jared Wasinger:a few ish… well, I think two issues with the… EALS implementation of the spec.
Jared Wasinger:So I guess those are being… Currently being addressed.
Jared Wasinger:But, like, regarding, A timeline for DevNet.
Jared Wasinger:I… from the… from the… from my…
Jared Wasinger:from my point of view, writing this for Geth, I think we could be ready,
Jared Wasinger:potent, very soon, but I don't know…
Jared Wasinger:If other people are on the same page.
Barnabas:Any other ELs have any updates regarding… Dr. Alexis List?
Łukasz Rozmej:So, Nevermind has a prototype of block-level access list. I'm not sure what the testing status, Mark, Harvey Hill is handling it. I don't… also don't see him in this call, but I know, we have,
Łukasz Rozmej:Something that he said it was working.
felipe:Yeah, I… I can give an update from the testing side.
felipe:Yeah, we did release a pre-release for block-level access lists, and as, Jared mentioned.
felipe:The specs, need some updates.
felipe:And, we are… we're getting a proper review on the spec side right now, and I… I assume once it's merged.
felipe:We can figure out those issues, and then we'll get another release out this week as well.
Barnabas:That sounds good. Have any other EL teams managed to take a look at… Implementation?
spencer-tb:Hey, this is not, block-level access list related, but,
spencer-tb:Just wanted to jump in from the east side, just to say that we've… Just released our…
spencer-tb:V5 full release. It's been a long time coming for this one, and
spencer-tb:I think I've already messaged all the EL teams specifically.
spencer-tb:but it's worth taking a look at the release notes.
spencer-tb:and… We essentially are announcing here that we're planning to merge execution spec tests with eels, execution specs.
spencer-tb:What that means for test releases, so in the future, when we do finally merge the two repos, test releases will still…
spencer-tb:come from East?
spencer-tb:So that will still be the main place to get tests, but the actual Python tests…
spencer-tb:And the East Framework will move to eels.
spencer-tb:For now, if you're working on East, PR should still go there, but we'll,
spencer-tb:communicate accordingly for when people need to start moving the PRs to ELS.
spencer-tb:I guess I can also mention a bit about the test fixtures, so we have, 3 tarballs here, so we've got
spencer-tb:The latest benchmark fixtures?
spencer-tb:We also have fixtures stable and fixtures developed, so fixtures stable, contain all the tests, filled until Prague.
spencer-tb:And Fixtures Develop contains all the tests, filled into Osaka.
spencer-tb:And within these two tarballs now, we have refilled general state tests from Ethereum tests.
spencer-tb:So… Yeah, I recommend running the fixtures Develop Powerball, because they have…
spencer-tb:the general state tests refilled for Osaka, and obviously we had the… Transaction gas limit cap, so…
spencer-tb:Just to make sure you guys are passing those tests, too, because we changed, we changed a lot of the tests,
spencer-tb:Or the transaction gas limit cap.
spencer-tb:A quick note on the actual general state tests, that they're only filled… From Cancun.
spencer-tb:So, if you want to still run
spencer-tb:the general state tests from Ethereum tests for forks before Cancun.
spencer-tb:like Shanghai, backwards, you'll want to still get these from Ethereum tests. But if you're only concerned with
spencer-tb:Prague and Osaka, for the general state test from Ethereum tests.
spencer-tb:then, the… tarballs in this release should be perfect for you.
spencer-tb:Yeah, more information's in the release notes, and if there are any questions, please…
spencer-tb:Fire them to us. Yeah.
Barnabas:Thank you, Spencer.
Barnabas:So if nobody else has any updates regarding whatever access list, then maybe we can jump into EPBS.
Barnabas:Anyone on the call? That's…
Barnabas:the most up-to-date with the PBS?
Barnabas:Any photos, or…
Barnabas:Terrence?
Barnabas:I guess the instant suspects are still under review, and I'm not sure how the…
Barnabas:Testing of it is everywhere, but…
Barnabas:Yeah, I think we can circle back to that next week, or week after that.
Barnabas:That was the agenda for today. Anyone else has anything else that they would like to discuss?
Barnabas:Sounds like everybody's happy, so maybe you can end the call a bit sooner.
Barnabas:Thanks for coming, and see you all on Thursday.
pawan:Okay.
Agnish Ghosh:Right.
Pooja Ranjan:Yeah.
Antoine James:Thank you. Thanks. Bye-bye.
Andrew Ashikhmin:Okay, bye-bye.
Dmitrii Shmatko:Aye.
Chat Logs
00:02:37
Christine Kim:ooo barnabas hosting today?
00:03:09
Barnabas:https://dora.fusaka-devnet-3.ethpandaops.io
00:03:10
Phil Ngo:Replying to "ooo barnabas hosting..."
Bringing out the whip
00:04:07
Justin Florentine (Besu):how long did it take us to finally re-finalize?
00:04:38
Barnabas:We have a new validator summary page view on dora: https://dora.fusaka-devnet-3.ethpandaops.io/validators/summary
00:04:58
Barnabas:Replying to "how long did it take..."
6 days
00:06:13
Barnabas:Summary of the bugs: https://notes.ethereum.org/@ethpandaops/fusaka-syncing-bugs
00:07:29
Roman:i’m also taking a look
00:07:32
Roman:at that pair
00:10:10
Ben Adams:Nethermind: no reported issues
00:10:17
Barnabas:lack of backfill shouldn’t be a blocker.
00:10:24
James He:We are also missing some api endpoints that aren’t mandatory
00:10:37
James He:And web3signer changes*
00:10:57
Manu:(W3S is not tested in the devnet)
00:15:42
lightclient:also still looking into this regression in the syncing bugs doc, related to our state history feature and not fusaka though
00:18:02
Łukasz Rozmej:We could do Holesky release
00:18:06
Roman:have we decided on the timestamp?
00:18:20
lightclient:plausible, but would need to discuss w team
00:18:32
Roman:so, Thu -> Mon for release?
00:18:44
Barnabas:too soon?
00:19:22
pawan:We can do a rc release I think
00:19:26
Phil Ngo:Possible. Will confirm
00:19:32
Roman:i don’t mind, that would just be close to our current release, but this shouldn’t impact the decision to proceed quicker with the holesky fork
00:19:33
pawan:But need to check with the rest of the team as well
00:19:50
Manu:Have to check with the rest of the team as well
00:20:36
Agnish Ghosh:need to check with team
00:20:53
Enrico Del Fante (tbenr):Early next week we can, maybe not Mon but Wed should be good
00:21:32
Barnabas:Lets aim for 22nd for pre-releases that could include holesky timestamp and bpos
00:21:45
Enrico Del Fante (tbenr):Replying to "Lets aim for 22nd fo..."
Good for us
00:22:06
Łukasz Rozmej:Kamil is on vacation AFAIK
00:24:43
Mario Vega:https://github.com/ethereum/execution-spec-tests/releases/tag/bal%40v1.0.0
00:25:20
Justin Florentine (Besu):Besus is in progress, not ready for testing yet
00:25:28
spencer-tb:New EEST full release v5: https://github.com/ethereum/execution-spec-tests/releases/tag/v5.0.0
00:25:37
Łukasz Rozmej:Feel free to ping me with updated BAL specs ,will be happy to review
00:28:04
felipe:Replying to "Feel free to ping ..."
Will do. We've been updating in the block-access-lists channel in Eth R&D as well.
00:29:35
felipe:Thanks Barnabas!
00:29:43
Trent:Thanks all!
00:29:45
Jen:Thank you !