Josh Davis:Alright, your life.
Transcript
stokes:Okay, thanks.
stokes:Hi everyone, welcome to ACDC164. I'll put the agenda here in the chat. It's issue 1700 on the PM repo.
stokes:And, yeah, so the agenda's pretty light today, because we're all quite busy with Fusaka and making progress there on the road to mainnet. So, yeah, let's get into it.
stokes:First up, really, let's check in with, DevNet 3. There's been a number of issues, that people have been dealing with.
stokes:So, yeah, I don't know if there's anything to discuss there.
stokes:I know there's been rounds of finality on the test, or DevNet, and then rounds of non-finality.
stokes:Barnabas put together this doc here with a bunch of open issues, so yeah, anything anyone wants to touch on?
stokes:I'm not sure if Barnabas or maybe Perry are here.
stokes:They could speak to that a bit more.
stokes:Okay, I might be out today.
stokes:But yeah, essentially just, like, a number of different bugs, across a mix of clients.
stokes:So, yeah, I don't know if there's anything else worth touching on it right now.
stokes:Okay, Guinea says, Lighthouse bugs should be fixed now and unstable. That's good to know.
stokes:Okay, prison bug fixed, cool.
stokes:Yeah, no issues for another Mindarketh here.
stokes:So, yeah, I mean, again, I think people are generally busy with these bugs, and or fixing them.
stokes:And the plan would be to roll out updates to DevNet 3 as soon as we're there.
stokes:Oh, no worries, Barnabas is here. Yeah, Barnabas, we're talking about DevNet 3 and any syncing issues.
stokes:Along with any other general issues, I don't know if you have another update beyond that you'd like to give?
Barnabas:So I… I combined a list of documents for,
Barnabas:I collected all the feedback from the different client teams, and I posted it in the PM repo.
Barnabas:We got the zoom for it.
stokes:Oh yeah, I put it in the chat area. Maybe I'll repaste it. You might not be able to see it.
stokes:So, yeah, I don't know if there's anything… there's a number of things in here, but yeah, some of the clients are saying in the chat that some of the bugs are fixed,
stokes:Teku says they're still working on some RPC things, but… yeah, looks like… Everyone's making progress.
Barnabas:Yeah, so we want to confirm that, everybody has their trunk branches, updated with all the fixes, and, once that's done, then we're gonna…
Barnabas:try to, get DevNet3 up to very close to 100% participation, and then mimic another non-fanuity event for a day or two, and then we're gonna try to recover again and see if all the fixes have actually worked.
Barnabas:After that, we can, think about launching DevNet5 with 1,500 nodes.
stokes:Mirror?
stokes:Sounds good.
stokes:So, yeah, I think that's pretty much where we're at, unless anyone has anything else to add, that's kind of just the status with soccer right now.
stokes:Squashing bugs, getting ready to move on to the next step. DevNet 5 will be one of these larger DevNets, like DevNet4, and from there, we'll have some data to know how to proceed to testnets, and then ultimately mainnet, but…
stokes:We're not quite there yet. Yeah, with all the open issues we have.
stokes:So, that does tee us up into, DevNet5, and again, I'm not sure there's much to touch on, just given where we're at.
stokes:But yeah, any questions or comments right now?
stokes:Okay.
stokes:Then, there's really just one other thing we wanted to touch on today, so… last ACDE, last week.
stokes:We touched on the protocol upgrade process doc. I'll grab a link here in case you want it.
stokes:And this document kind of came in response to Hectra, where we had a number of issues across some testnets that were pretty gnarly to fix.
stokes:This process document lays out kind of a timeline that both, you know, ourselves as core developers, but also the community can use to coordinate around these upgrades.
stokes:And one thing that came up was talking about timelines for testnets.
stokes:Just simply because, yeah, especially with Fusaka, in the event that we want to try to move as quickly as we can.
stokes:Getting Haleschi, Sepulia, and Hoodie all together, following the process doc implies one set of timelines. And the question came up, you know, can we…
stokes:Essentially move faster, and more generally, like, how… what is… what is safe?
stokes:What's the right trade-off between speed and safety here?
stokes:And in response to that, so Tim couldn't make it today, but he went back and looked at, the calls where we discussed this previously. We had said that…
stokes:Essentially, we had a given scheduling for timelines in the dock, and…
stokes:That was the most contentious point, back when we first discussed this, so if anything, now is the time to revisit this.
stokes:In response to that, Tim made a PR.
stokes:to this document that suggests some changes. He also reached out to a number of different, L2s, LSTs, other infrastructure providers.
stokes:And generally, people are okay with, these changes.
stokes:And… let me just grab a link to the changes here.
stokes:So, let me summarize them.
stokes:Essentially, don't couple testnet and mainnet forks and client releases, which we kind of already had that, but just making it clear.
stokes:It's okay to have multiple test networks in one release.
stokes:Then, I think the first concrete change here is saying a minimum of 14 days rather than 30 days between client releases and the first testnet.
stokes:Then, his proposal here in the PR is minimum 10 days between each testnet, but closer to 2 weeks is, you know, ideal.
stokes:Then we keep this invariant of having 30 days between client releases and mainnet.
stokes:So… He wanted to discuss this today, and there's some… actually quite a little bit of feedback,
stokes:on the PR,
stokes:And there's also some comments here on the agenda itself, so I don't know if people want to, respond to any of this.
stokes:Otherwise, I think the ask would be to go ahead and either approve the PR, or yeah, surface any, any feedback there.
stokes:Generally, I think this is a nice refinement. It gives us, you know, a little flexibility around how we want to think about pipelining Tesla upgrades, just, again, so that we can not have all these
stokes:Big chunks of time between these different stages.
stokes:But yeah, even just looking across this, there's a lot of different opinions on this, so…
stokes:I will open the floor now for comments on that.
stokes:Or any questions if it wasn't clear what I was talking about.
stokes:Have people had time to look at this?
stokes:Otherwise, it's not clear if silence is… I'm okay with it, or… I just have no opinion, because I haven't looked at it.
Justin Florentine (Besu):Basie team has not had, enough time to look at this.
stokes:Okay.
stokes:We do have a comment in the chat here from Lighthouse, generally in favor of the changes.
stokes:I guess one thing to note here is…
stokes:We, you know, obviously this is in the context of Fusaka timelines.
stokes:And if we kind of follow the letter of the document, I would put a lot of time, given that we essentially have 3 testnets to go through with Haleshky this go-around. That being said, Heleshky is being deprecated, so it seems fine to have a bit more of a compressed timeline there.
stokes:Where…
stokes:you know, if we kind of consider Sepolia to be our first, quote, real testnet, that would still follow this, like, 30 days, buffer.
stokes:That was in the original document.
stokes:And then separately, this PR does kind of set us up to move faster in general.
stokes:Around testnet timing.
stokes:Okay, Phil from Lodestar says they'll take a look. It'd be good to get approval from non-core devs on the PR2, yeah. So, I know Tim did reach out to a number of different community members, and generally they're all on board. I think…
stokes:Probably the most… Chris's feedback was, from Lido here on the PR. Let me find it.
stokes:They're kind of arguing for, yeah,
stokes:a bit more time, I think. Especially around the first SNF fork.
stokes:So… Yeah, given that people haven't really had time to digest this that much.
stokes:Yeah, Prism just said this as well. Oh yeah, Justin?
Justin Florentine (Besu):So, one of the things that was a bit of an issue back in the Pector days, and that my understanding is this process was to address, was kind of the mixed-use nature of the testnets, where we want them to be testnets that we break, or may break, obviously we don't want to, but they are breakable.
Justin Florentine (Besu):And… L2s, app developers, staking providers, etc. are kind of using those as staging environments.
Justin Florentine (Besu):Before they go to production.
Justin Florentine (Besu):I don't really see anything in the ongoing discourse or the proposed changes to any of these processes that kind of addresses that. So, from my perspective, this is still kind of an outstanding issue that we haven't really addressed yet, and I'd like to see a little bit more of a concrete perspective on that.
Justin Florentine (Besu):So if we have one, I'd love to hear that shared, or to hear feedback from L2s, app devs, staking providers, etc. And if we don't have that yet, I'd like to see that more before we modify the existing process. Thanks.
stokes:Okay.
stokes:Yeah, I know Tim reached out kind of directly to different people, but yeah, he did encourage them to chime in publicly, but yeah, definitely something we can make more…
stokes:more transparent. So yeah, I can reach out to different people and see what I can do there.
stokes:Okay, Barnabas says something here about Shadow Forks. So yeah, we'll have them on all test nets going forward.
stokes:Which is just an additional, place to catch any issues.
stokes:Okay, I mean, it kind of sounds like there's not much more to add here.
stokes:So, in that case, yeah, please take a closer look, both as client teams and anyone listening, take your feedback to the PR, I think is probably the best place for it right now.
stokes:And yeah, I'll do what I can on my end to, get clear, sort of, public statements from these different community players, and yeah, we'll move forward there.
stokes:So, that was really all we had on the agenda. I wanted to keep it pretty light today, because again, everyone's just…
stokes:Focused on Fusaka. Yeah, any other…
stokes:things for discussion. Otherwise, this will be a very short call today.
stokes:Okay, then good luck, with Fusaka.
stokes:Okay, Alexi, here's asking… I think you mean about the Fisaka timelines, and yeah, I think, given where we're at, it's a little premature to, like, talk about them concretely.
stokes:You know, the next steps here would be getting DevNet3 together, like Barnabas was saying.
stokes:If we think everyone has fixed all the bugs, then we do one more finality, non-finality test there.
stokes:If that looks good, then we can move back to MNET5, like we had discussed.
stokes:And from there, you know, we can kind of stick to the rest of the schedule.
stokes:But, yeah, it's, I'm not sure, until we…
stokes:Get through those first couple milestones, how we're going to…
stokes:Discuss timelines and have them really make too much sense.
stokes:I did post something on Discord on the AllCoreDevs channel, on the Ether R&D server, that's probably my latest view of what makes sense.
stokes:But, yeah, otherwise, again, until we have DevNet3,
stokes:Sorted out, and 5 running, it's a little premature.
stokes:Marius is asking releases on Monday.
stokes:If you feel like you're ready to make a release, you could, but again, I don't think, we'll have even…
stokes:Yeah, I don't think we'll even have more than a few clients who'll be ready.
stokes:Okay, Barnabas is asking trunk branches merged by Monday. Yeah, that seems like the next milestone for everyone, and I feel like that was kind of…
stokes:The current item for everyone, but…
stokes:If we need to make that clear. Is there more to say there, one of us?
Barnabas:Not from my side. I'm just hoping that we can get everyone to fixing all their…
Barnabas:known box by Monday.
stokes:Bye, man.
stokes:I mean, are there clients who don't have
stokes:most of a chunk branch ready. I thought it was more that we kinda had good progress there, and it was more now just fixing bugs.
stokes:Back them up.
Barnabas:I think some clients are still, working on merging all the peer desk features into their trunk branches.
stokes:Okay.
stokes:So yeah, in that case, getting everything into Trunk Branch by Monday would be great. And yeah, otherwise…
stokes:Squashing bugs.
stokes:Mario has an asked to check Hive here, if there are any outstanding issues. Yeah, Mario, I'd like to say something?
Mario Vega:Yeah, yeah, just basically that. Please check the link. We're going over the failures, but if you guys can check them too, please, yeah, please do so. It will be very nice, yeah.
stokes:Okay, I think it should be clear what the next steps are for everyone, and…
stokes:Otherwise, I would propose we close the call, so people can do that, and we can keep the Saka moving.
stokes:I guess just to touch on Glamsterdam, just as a sort of reminder, we had said that we would have,
stokes:the next step there for EIPs for Amsterdam would be with your stock and maintenance releases.
stokes:Again, it's hard to say exactly what calendar date that is, given the progress of the FSACA, but just as an FYI.
stokes:Keep that moving in parallel.
stokes:Okay
stokes:Sounds like we've gotten what we're going to get out of this call, so let's go ahead and wrap up.
stokes:Oh, one last hand. Yeah, Marcin?
Marcin Sobczak:Hi, I would like just to highlight that, when, testing, EIP with Modex gas.
Marcin Sobczak:cost increase.
Marcin Sobczak:We found that one transaction is failing, like, one contract seems to be broken. We already know that it's from Align Liar, and it's being discussed right now on Gustnie's testing channel on Telegram.
Marcin Sobczak:So, I'm just highlighting that there is potential, issue where, the CIP will break the contract, but, yes, it's being… it's being discussed right now on other,
Marcin Sobczak:On Telegram.
stokes:Okay, and these were changes with Fusaka, right? The modXP changes made in Fusaka?
Marcin Sobczak:Yes, yes, it's about 2 sucker.
stokes:Okay.
stokes:Yeah, anything anyone here needs to know there, beyond… Check out this Telegram channel.
stokes:If it's relevant.
stokes:Okay.
stokes:Then, yeah, let's go ahead and wrap up.
stokes:I'll see you all next time, and good luck with Osaka.
lightclient:Bye.
Mario Vega:Goodbye.
Marius van der Wijden:Oh, yep.
Chat Logs
00:03:18
stokes:https://github.com/ethereum/pm/issues/1700
00:03:58
stokes:https://notes.ethereum.org/@ethpandaops/fusaka-syncing-bugs
00:04:31
kingy_sigp:lighthouse bugs should be fixed now in unstable
00:04:42
Marius van der Wijden:Replying to "https://notes.ethere..."
Nethermind: no reported issues
Geth: no reported issues
00:04:45
Manu:Prysm bug fixed
00:04:47
Marius van der Wijden:Replying to "https://notes.ethere..."
Kings 🤴
00:04:57
saulius:Grandine bug fixed too
00:04:58
Enrico Del Fante (tbenr):Teku is still working on some bugs on RPC side
00:05:03
Barnabas:soz for joining late
00:05:38
stokes:https://notes.ethereum.org/@ethpandaops/fusaka-syncing-bugs
00:07:50
stokes:https://github.com/ethereum/pm/blob/master/processes/protocol-upgrade.md
00:09:26
stokes:https://github.com/ethereum/pm/pull/1715
00:11:18
kingy_sigp:lighthouse generally in favour of the changes
00:12:24
Phil Ngo:Will look at this, but would be great to get approval from non-core devs on this PR too and spread the world
00:12:58
lightclient:+++
00:13:04
James He:Have not looked as a team, sharing internally
00:14:41
Alexey:shadowfork tesnet once?
00:14:55
Barnabas:We will be having shadowforks on all testnets going forward
00:15:01
Barnabas:Its an easy win
00:15:04
Marius van der Wijden:Wdyt about long running shadowforks of the testnets that we can fork
00:15:13
Marius van der Wijden:Lol
00:15:22
Justin Florentine (Besu):i think sfs should run till they fail
00:15:50
Justin Florentine (Besu):maybe 'decay' is a better term than 'fail'
00:16:03
Vub:I think it's good for testnets that have apps on them to occasionally break. Apps too need the practice in handling exceptional situations :P
00:16:32
nixo:woah
00:16:38
nixo:Replying to "woah"
record setting?
00:16:41
Alexey:No release schedule?
00:16:51
Marius van der Wijden:Short call maxi
00:16:52
Josh Davis:Replying to "woah"
Think so…
00:16:53
Alexey:testnet ready releases
00:17:03
Justin Florentine (Besu):devnet5 is still tbd
00:17:05
Marius van der Wijden:So no releases on monday?
00:18:00
Barnabas:Can we aim trunk branches merged by monday?
00:19:08
Mario Vega:Please check https://hive.ethpandaops.io/#/group/fusaka-devnet-5 for any outstanding issues, testing team is currently going over these but clients checking them too would be nice
00:19:13
Manu:Prysm: We can use trunk branch
00:19:27
Barnabas:Replying to "Prysm: We can use tr..."
can i roll on devnet 3?
00:19:47
Manu:Replying to "Prysm: We can use tr..."
Depending if you want to use evil options like —withhold-data-columns
00:19:56
Manu:Replying to "Prysm: We can use tr..."
These kind of options won’t be merged in develop
00:20:13
Manu:Replying to "Prysm: We can use tr..."
(These options are the only diff regarding BN between peerDAS and develop )