Mario Vega:All right, so welcome, everyone, to ACDT number 78. Today is April 20th.
AllCoreDevs - Testing #078 — BAL Breakout
AllCoreDevs - Testing #078 — ePBS Breakout
Transcript
Mario Vega:Mainly we'll focus on discussing DevNets today.
Mario Vega:So, yeah, let's just get started. And the first topic is blob DevNets.
Mario Vega:Blood Delta Serum, and I think Barnabas, that's,
Mario Vega:Your domain? Do you have an update on this?
Barnabas:There's no, currently, any updates. I think most of the CL teams are focusing on, shipping ePBS. We currently have still just Prysm and Lighthouse deployed on them. Last week, we have enabled partial cells on every single node, and we're experimenting with that.
Mario Vega:Alright, thanks.
Mario Vega:Any points to be raised by any of the client teams? We're gonna need a lot of minutes.
Barnabas:It would be good to start onboarding other clients here as well.
Mario Vega:Which are the ones that are currently, joined and not, and who is missing?
Barnabas:It's Prysm and Lighthouse only.
Barnabas:Everyone else is missing.
Mario Vega:Alright, anyone from Nimbus?
Mario Vega:Lodestar, who want to chime in?
Nico Flaig:Yeah, for us, there's still a few pieces missing we need to get done, probably this week.
Mario Vega:Alright, thank you. Anyone from Nimbus?
Mario Vega:This is blob.net server, by the way.
Nico Flaig:Oh, never mind my comment. I was thinking ePBS, Definite.
Mario Vega:Oh, I see.
Mario Vega:Got it.
Mario Vega:Alright, I think… I don't think there's anyone from Nimbus.
Mario Vega:Should we jump into the next topic, then?
Mario Vega:Or, Barnabas, do you want to… .
Barnabas:Yeah, because my wife.
Mario Vega:PowerPoint Alright, thanks.
Mario Vega:Yeah, let's move on straight to, blog level access list, DevNets.
Mario Vega:There was a comment in the agenda today by Marius that, he suggested that we should jump directly into Glamsterdam.devNet 0 instead of going into, Block Level Access Lists.devnet 4. So Marius, would you like to, explain on the topic?
Marius Van Der Wijden (M):Oh, give me a second, I'm in the room, and my fiance is also in the call, so…
Marius Van Der Wijden (M):Okay, so…
Marius Van Der Wijden (M):Basically, what we want to do is we want to move, to a world that has all of the changes for Glamsterdam as soon as possible.
Marius Van Der Wijden (M):And there's two ways to do… of doing it. We can either go to VelDefNet 4, or we can go to Limited DevNet 0. What we would gain by going to BAL DevNet 4 is that there is a…
Marius Van Der Wijden (M):That, it's, like, regardless of whether ePBS is happening on… on the consensus layer, we can, we can continue testing stuff on the, on the execution layer. But what we can instead also do is just go to glimpse at M, definite0,
Marius Van Der Wijden (M):And then… Get an old version of the consensus layer, to create a network.
Marius Van Der Wijden (M):Based off of that, if we feel like the contents layer is not ready.
Marius Van Der Wijden (M):for, for Glamsterdam yet.
Marius Van Der Wijden (M):Yeah.
Mario Vega:So, Barnabas, you have, the other, suggestion, right?
Barnabas:Yeah, so basically, we want to make sure that the PBS branch is as stable as possible, so we're not going to be doing any kind of pausing yet in the early days of Amsterdam DevNet Zero, for sure.
Barnabas:So, if you want to do fuzzing and benchmarking and stuff like that, I think that should be done on Ball-specific DevNets, like BAL DevNet 4.
Barnabas:And ideally, Glamstreet and DevNet3 should be, based on top of, ball DevNet3 plus, the PR770 from Execution APIs.
Mario Vega:Yes, produce?
Potuz:I want to support what Barnabas is saying. I think,
Potuz:the Glamsterdam DevNet Zero should be thought of as an ePBS DevNet. It's not going to be stable.
Potuz:And it doesn't matter for us that it's not so stable, and it doesn't matter if
Potuz:Block Level Access Lists wants to be testing, like, new features on that DevNet, because it's very unlikely that we're gonna be hit with problems that are gonna be…
Potuz:more complicated for us to debug, but I suspect that for EL devs, debugging on an unstable consensus chain is going to be much more complicated. So, my advice for EL devs would be to just have a second,
Potuz:just purely EL, just purely bald DevNet, while we test consensus on the Glamsterdam at DevNet0.
Mario Vega:Okay, Johnny?
Toni Wahrstätter:Yeah, or I can also just echo that… what Potos just said, and also what Barnabas said, just with one small caveat, because Barnabas, you said, we should then
Toni Wahrstätter:build, glow as definite.
Toni Wahrstätter:or Amsterdam Definite 0 on top of BAL DevNet 3, but we are currently working on BAL DevNet 4, so I guess you mean we build on top of BAL DevNet 4, right?
Barnabas:No, because I don't want any 8037 HAVOC on Glamsterdam, that net zero. That's why I.
Toni Wahrstätter:But we have it on Definite Free already.
Barnabas:No, those are using static values. Yeah, but those are using the static values, not the dynamic ones, and as far as I know, the dynamic values are the ones, that, like, require a huge, rebate.
Marius Van Der Wijden (M):you know.
Barnabas:the execution of the airplanes.
Marius Van Der Wijden (M):No, no, no. No, that is just changing one value. I'm doing a rebase because, because we've merged some stuff, into…
Marius Van Der Wijden (M):into… into master and… and rework some of the stuff. That's why we have to do rebase. It's not because of the dynamic value for the… for AT37. Basically, like.
Marius Van Der Wijden (M):whether it's a static or a dynamic value shouldn't make a big difference. The calculation is pretty easy.
Marius Van Der Wijden (M):The… the bigger point is… is,
Marius Van Der Wijden (M):whether, is whether 8037 is correctly implemented, and that is…
Marius Van Der Wijden (M):Testable regardless of what the cost per state byte is.
Barnabas:Alright, but then, could maybe Dragon mention what he means under disruptive changes?
Dragan Rakita:Maybe that's true, forget, but for our EVM, we still need to implement those dynamic changes, and there is additional, like, 4 or 5 additional changes that needed to land on the state gas.
Dragan Rakita:So yeah, there is… Yeah, there's multiple trains that need to be done.
Marius Van Der Wijden (M):Yes. Like, there are changes that need to be done, also for the GAF side for Definite 4, but…
Marius Van Der Wijden (M):Like, those are not super traumatic.
Marius Van Der Wijden (M):God, I would say.
Marius Van Der Wijden (M):Anyway, like, it seems like everyone is, is, yeah, for making definite for…
Marius Van Der Wijden (M):the next, thing for… well, definite 4, and… but I, I think we should not make Glamsterdam DevNet.
Marius Van Der Wijden (M):0 based on, BAL DevNet 3. That would be a pretty huge mess for us.
Mario Vega:Alright, let's do Maria, D'Antonio, and Spencer.
Maria Silva:Yeah, I think I was just going to… to add that the complication on the dynamic cost was mostly around the test, so the Steel team had to rework a lot of the previous tests to,
Maria Silva:to be able to take these dynamic costs into consideration, and my understanding is that most of the work there is being done. Then, I would agree with Dragan, so we do have some spec changes between, BAL DevNet 3 and Bold DevNet 4.
Maria Silva:But I feel… I don't know, it's still up to the clients to decide, but I think it would be easier to manage to just have Paul DevNet 4, and then have Glamsterdam feed from… from that development branch, because implementations will be slightly different, and so it's… it's a bit weird to…
Maria Silva:be using, like, old implementations of AT37 in the…
Maria Silva:interop, DevNet, and then, on the side have, like, continued to work on the new, specs. But yeah.
Maria Silva:Happy to also hear your thoughts there.
Mario Vega:Tony?
Toni Wahrstätter:Yeah, I agree with that, what Marius and Maria just said. We should do DevNet 4, and then have the first Glamstadom DevNet also use DevNet 4 instead of doing Definite 4, and then have the first Glamsterdamdam Defnet, basically.
Toni Wahrstätter:Go a step back, and… be a DevNet-free implementation.
Mario Vega:That's true.
spencer:Yeah, I guess from my side, I think it depends how… like, how quickly do we want to test a glance at RamdevNet? If we just want to quickly test the integration between the EL and the CL, then using Belle DevNet 3 feels good, but if we want to use BAL DevNet 4, then it means the first glance at MDevNet's probably going to be delayed.
spencer:And… Probably towards the end of next.
spencer:next Friday.
Parithosh Jayanthi:I think at least timeline-wise, we should definitely be aiming for
Parithosh Jayanthi:Late this week, or early next week, because…
Parithosh Jayanthi:If we have stuff in person, then it's going to be most productive if we're not waiting till the end of the week to have it done.
Barnabas:Exactly, and that's why I proposed that we would do, bold.net 3, instead of Bold.net 4.
Barnabas:Like, we ideally want to start the Glamsterdam DevNet.
Marius Van Der Wijden (M):Barnabis, like, we… like, we have most of the changes for BAL DevNet 4 already, and, like, it takes us another day or two to get a version of BAL DevNet.
Marius Van Der Wijden (M):for done, for all clients. And then it makes no sense to… to test a version of…
Marius Van Der Wijden (M):of AT37 that is, like… I don't know.
Parithosh Jayanthi:But then where does Spencer's timeline of end of next week come from, if the changes are 1-2 days?
Marius Van Der Wijden (M):I don't know.
Marius Van Der Wijden (M):Like, definitely for us, it's not… it's not one to two days. It is… sorry, it is one to two, two, two days. I need to do a rebase of,
Marius Van Der Wijden (M):Of the stuff, and then I need to implement the seven changes that we discussed on top.
Marius Van Der Wijden (M):And that should be done in a day.
Marius Van Der Wijden (M):Maybe two.
spencer:I guess I'd add the reason I said, it'd be a bit later is just…
spencer:each individual new spec change, so I think there's 7 total to 1837, has, like.
spencer:added already a lot of… a lot of extra testing that has taken… taken a while, and I think…
spencer:I mean, maybe we'll be very slick in person, and
spencer:It'll be very quick for us to…
spencer:Like, iron out all the bugs in person.
Marius Van Der Wijden (M):like, do you have… do you have tests for BAL DevNet?
Marius Van Der Wijden (M):4… already?
spencer:No, no, not yet. I was hoping to have it released for this call,
spencer:But yeah, we're… we're still ironing out, some… some of the AT37 changes.
spencer:But yeah, well, I think we're just filling the tests now, and trying to fix…
spencer:And the remaining changes.
spencer:Yeah, and I think most of the fixes are… Due to the dynamic…
spencer:Cost per state byte, addition.
spencer:But we're close, we're very close.
Mario Vega:Yes, Stephan?
Stefan Starflinger:Yeah, I would also propose, to have… to launch, BAL DevNet 4. We can have it
Stefan Starflinger:very short, but then to give clients to iterate a little bit, and then quickly afterwards launched Glamster Dam.
Stefan Starflinger:Definite zero, that would be my suggestion. It doesn't have to be very long-lived, but just…
Stefan Starflinger:To get some fixes in, and then focus on GameStream DevNet Zero, or we can have both running then at the same time as well.
Barnabas:My main question would be, is there any other EL teams that feel, sufficiently ready, basically, to have 10.4 changes?
Barnabas:In a feature branch?
Toni Wahrstätter:I guess the question is by end of week, right? Or what's the timeline you're now thinking of, Parabas?
Barnabas:Well, I'm trying to do local group assistance tests before we launch a .NET, so that usually requires at least another day.
Barnabas:ideally by Wednesday, if we would have… like, Mario said that, they should be ready by Wednesday, so that's why it would be good to have at least another client or so by Wednesday.
spencer:We'll have tests, tests out by the end of today, at the latest, early tomorrow.
spencer:Yeah.
Mario Vega:So, to add up to that, I think, we can reduce the number of tests by reducing the coverage, if something is not… does not go quite smoothly.
Mario Vega:But we'll try to have something. Bye tomorrow.
Mario Vega:Or Demophore. Ball metaphor.
Marius Van Der Wijden (M):Instead of… instead of reducing the number of tests, can you just undo the, the dynamic state cost thing?
Marius Van Der Wijden (M):Because, like, that would be, for us, very easy to test. We would just undo that change.
Marius Van Der Wijden (M):We would… we would run all of the tests, all of them would be passing, and then we would just…
Marius Van Der Wijden (M):Redo the gas cost change, and then…
Marius Van Der Wijden (M):Like, if this is the… if this is the issue with the… why the tests are not… are not published yet, then, I think that should just be undone.
Marius Van Der Wijden (M):Because, as I said, I don't see many… Issues happening with that.
Marius Van Der Wijden (M):Well, the, like, the behavior changes that we are doing to 8037 are more complex and would require, like, specific test cases.
spencer:I think… I think we'll be able to get them in. Like, we're… we're… we're basically there, yeah.
spencer:M.
Mario Vega:Just to be clear, that's a 4 spec,
Mario Vega:Are we all on the same page?
Mario Vega:All teams?
Mario Vega:Alright, so, just to reiterate, we're aiming, for spec.
Mario Vega:For Block Level Access Lists, DevNet 4, which will become Glamsterdam DevNet0.
Mario Vega:Is that… is that correct?
Mario Vega:Barnabas, or anyone else?
Barnabas:Yeah, I'm still waiting for at least one more client to speak up about it, because last week we were still hearing a bunch of people that were doubting that they could be even done, by end of next week, which means end of this week.
Mario Vega:Can other clients comment on this? How about Nethermind Besu? Yeah, Besu, please.
Daniel Lehrner (Besu):Yeah, I think as long as we get the tests, we could be ready. So, I mean, we have most changes prepared in draft PRs, basically. I'm a bit afraid of the 80-47 changes without tests, because those are really easy to get wrong.
Daniel Lehrner (Besu):But I think if we get those, we should be ready, yeah.
Barnabas:Yeah, in that case, we can definitely do DevNet4, with the new
Barnabas:two new EIPs and the dynamic,
Barnabas:changes plus, yeah, plus the PR from 770.
Barnabas:Which we should discuss next, I think.
Mario Vega:Alright, before we jump into that, there are, any other ideal clients, with objections to…
Mario Vega:The current, spec, that we are aiming for.
Mario Vega:All right. If not, I think, just to reiterate, we're aiming DevNet4, so full 83.7 for DevNet 4.
Mario Vega:And this will become Gamson Devidence Zero.
Mario Vega:And the last top… the last, PR to discuss would be the Execution AP's, PR, so let's… let's discuss that. I just linked that in the comment, so, for everyone to see.
Mario Vega:Who is in charge of this, PR? Who can give us an overview?
Mikhail Kalinin:I can give an overview.
Mikhail Kalinin:Can you hear me?
Mario Vega:Yes, perfect.
Mikhail Kalinin:Alright, so… this PR, basically.
Mikhail Kalinin:Changes the, Oh.
Mikhail Kalinin:the behavior of EL when CL is, like, kind of in sync and way behind the head that EL is aware of.
Mikhail Kalinin:Previously, if, FCL, Sends the head block.
Mikhail Kalinin:that EL already has validated many, many, much time, a lot of time ago.
Mikhail Kalinin:In the case when this book is from the canonical chain, it's ancestor of the head that the EL wears of.
Mikhail Kalinin:Basically, this back was…
Mikhail Kalinin:Saying that. In this case, the folk choice Update.
Mikhail Kalinin:Can be skipped.
Mikhail Kalinin:And in the case when it's skipped, there is no payload build process triggered.
Mikhail Kalinin:And, yeah, that was before Gloss, and worked before Gloss.
Mikhail Kalinin:But now, since gloss, we need to…
Mikhail Kalinin:build payloads, and we need to rework out payloads, canonical, like, from the canonical chain, from the point of view that EL thinks that it's canonical, because it was previously, pointed out as the head. So we need to rework back to, like, ancestor of the head of canonical chain, and be able to… to build
Mikhail Kalinin:payloads on top of ancestor of…
Mikhail Kalinin:Of the head of canonical chain.
Mikhail Kalinin:Yeah, that's… that's what this PR is about, and also it introduces some limitation on how deep this reorog could be.
Mikhail Kalinin:Which is 32 blocks, basically, and… Yeah, but this limitation only…
Mikhail Kalinin:matters for the case where we do rework from the head of canonical chain to one of its ancestors, but not to any other kind of reworks.
Mario Vega:Marius?
Marius Van Der Wijden (M):Yeah, so, I, I left this PR, I,
Marius Van Der Wijden (M):It's, it's, it's a 5-minute thing for… for every client to, to, to change. Basically, the way we, we, we have discussed it, in the gas team is,
Marius Van Der Wijden (M):So the way where we had this rule not to reopen the head block was, in order to get, yeah, to filter out faulty or late messages from the…
Marius Van Der Wijden (M):from the CL, that would, yeah, basically, show a weird state to the user, by going one block back.
Marius Van Der Wijden (M):what we're doing here now is just to track more closely the fork choice of the CL, and we hope and we trust that they're not
Marius Van Der Wijden (M):doing weird things with it, so, yeah, but this is anyway an assumption that we have. So, yeah, this…
Marius Van Der Wijden (M):should be really a no-brainer. I've… I've implemented it twice today.
Marius Van Der Wijden (M):And in… in… within 10 minutes, so…
Marius Van Der Wijden (M):I think every, every client should be able to do this.
Marius Van Der Wijden (M):Yep, that's it.
Mario Vega:Maputos?
Potuz:Yeah, so I think the phrasing on the PR of, like, the block… the limits being 32 blocks for this reorg, it's off.
Potuz:ideally, what EL clients should support is a reorg up to the finalized hash.
Potuz:And if clients cannot deal with very deep reorgs, that's fine. The spec says that you should be able to support up to the finalized hash, and if you can't, then just put in your implementation some limits that are reasonable for the implementation themselves. But I think it's…
Potuz:not correct to have the specification, allowing for this sort of reorgs up to a, like, fixed… any fixed number of blocks.
Mikhail Kalinin:I think it's not, like, fixed. I mean, yeah, it could be… any number.
Mikhail Kalinin:I think we need some sane limit.
Mikhail Kalinin:And finalized, realled back to finalized. Yeah, I'm just, you know, it's up to EL implementation, so what do they think about it? And, for instance, what happens, then the question, what happens if…
Mikhail Kalinin:If EL does not support a reorg of, you know, of 10,000 block depth, What should it…
Mikhail Kalinin:Return, in this case.
Mikhail Kalinin:Probably it should be an error.
Potuz:I don't…
Mikhail Kalinin:Or something like that.
Potuz:It's fine, because it's only on the proposer side. If you're proposing locally, you're self-building and your EL cannot support this.
Potuz:then you will fail just proposing your local block, but you can just still take bits, and you can still propose a block. And for anyone else in the chain, it doesn't matter, because if there is such a long reorg, anyone else that actually gets that payload will see a non-S reorg, not one of these going back to head.
Potuz:So it only affects the proposer side and in the self-building path.
Potuz:So I just don't see any reason why we shouldn't just put there that you're… you should be able to reorg up to the finalized hash.
Mikhail Kalinin:Then what will happen? I don't know.
Potuz:Then you get a blog.
Mikhail Kalinin:Yay.
Potuz:And then you have a branch, and that's just the usual case in which the EL should be able to support the reorg up to the finalized hash. That happens today. Right. The EL today should be able to reorg up to the finalized hash.
Mikhail Kalinin:Yeah, just imagine the case where we have a long period of non-finality.
Mikhail Kalinin:And then…
Potuz:Happens today. This is today!
Mikhail Kalinin:Right.
Potuz:specification.
Mikhail Kalinin:But, yeah, so we had this… In this fact, so not…
Mikhail Kalinin:not do a reorg if the CL, like, syncs with, like, the chain, right, with the network.
Mikhail Kalinin:And it tries to catch up with the head, and it sends already known payloads and payload hashes, and yeah, and FogChoice updated as well, pointing out already known and validated in the past blocks by EL.
Mikhail Kalinin:So, and in this case, if there is a long period of non-finality, in this case, EL will have to do a reorg to a finalized block regardless. So, yeah.
Mikhail Kalinin:I tried… actually, this PR just tries to do something in between, still allows for this optimization.
Mikhail Kalinin:Let's name it this way.
Mikhail Kalinin:And, allow for… For reorgs to, you know, if it's, like, quite close to the head, to…
Mikhail Kalinin:Yeah, to address the case for a PDS.
Mario Vega:Tony.
Toni Wahrstätter:Yeah, my question was, just answered by Marius, so I was curious, like, because Podos mentioned it's only for the proposal, but I was interested in, yeah, how would this work for a test desk? Would they have the state on the EL site available? But Marius just,
Toni Wahrstätter:replied in the chat that EL clients would basically need to resync in such a situation. So I guess it's not super smooth if we ever had such a long reorg, because this would mean
Toni Wahrstätter:The nodes could not attest, timely, because they would need to re-sync in order to get the… state.
Toni Wahrstätter:Of that pivot block.
Potuz:That cannot really happen.
Potuz:If clients cannot support a 90,000 block reorg, then that chain will not have the attestation weight to actually cause the reorg.
Potuz:So that's… I think it… we're just discussing over nothing.
Marius Van Der Wijden (M):Yeah, so, I… I like… I like the limitation to this, one, because what we have seen in the past was faulty CLs just feeding us old blocks and… and causing… and then, like, we ignored them, but this would now cause us to reorg back to old blocks.
Marius Van Der Wijden (M):So I think it makes sense to me, to have a limitation there, how deep these…
Marius Van Der Wijden (M):Like, these reorgs within the same branch can take.
Marius Van Der Wijden (M):And yeah, as… as Mikhail already said, this doesn't…
Marius Van Der Wijden (M):This doesn't, impact how reorgs to a different branch, take.
Marius Van Der Wijden (M):Right.
Potuz:But the guy thinks… you guys are dismissing the fact that we need to change FordChoice to put this kind of limitation, because otherwise, we actually might get into sort of, like, a deadlock. If FortChoice is saying, you should reorg up to, like, say, 64 blocks.
Potuz:and the EL cannot handle this, for whatever reason, then…
Marius Van Der Wijden (M):Why…
Potuz:CL, not advanced, because that's…
Marius Van Der Wijden (M):Why would…
Potuz:We should be building on top of.
Marius Van Der Wijden (M):But why would the fog choice ever need to reorg 64 blocks on a…
Marius Van Der Wijden (M):A chain without any new block.
Potuz:No, no, so 4Choice would need to reorg on whatever… that's the assurances that we have. We can always reorg back up to our finalized checkpoint. It may not happen, it may be very difficult to happen, but this is what 4Choice is. That's what… that's what FFG guarantees to us.
Mikhail Kalinin:You mean, like, you… you need… you are talking about a reorg on the same chain, back to finalized block?
Mikhail Kalinin:Without, like, you know, introducing Without, like, introducing a competing block, right?
Potuz:Correct.
Mikhail Kalinin:Why would… why would, this kind of rework need it?
Potuz:Well, because this is what fortune is. We don't have a proof of fortjoice, we don't have a proof that this is impossible to do.
Mikhail Kalinin:Yeah, I'm just trying to find… find a way, you know?
Potuz:It might be impossible, that's… that's the whole point, and… and it's…
Potuz:It's fine if the spec says we should be able to reorg up to finalize, and implementations cannot handle this situation. Today, the spec says you should be able to sync any block on top of an unfinalized block, and that's what the spec says. Actually, the spec doesn't even say that it needs to be, well, it doesn't matter.
Potuz:In principle, clients should be able to sync any block.
Potuz:On top of a block that is not finalized.
Potuz:And we just can't handle this. We get dust, and therefore, we ignore some blocks. That's just fine.
Potuz:So, I don't see why we would put an artificial limitation where clients can just deal with those limitations. As Mario's saying, the clients already are not following the spec, and they cannot handle a reorg that is deeper than 90,000 blocks.
Toni Wahrstätter:I think they can handle it, they just need to re-sync. That's the… that's the thing.
Potuz:No, that means not.
Marius Van Der Wijden (M):No, no.
Potuz:Because if I don't get a… if I don't get an accepted, or if I don't get a valid there, then we just can't, we can't be in sync.
Marius Van Der Wijden (M):Yeah, but the… like, that's… that's the thing, we always had this… we always had this guarantee that the folk choice will never…
Marius Van Der Wijden (M):Flip as long as there is no other chain, no other block.
Marius Van Der Wijden (M):Where, where, like, it will just not go…
Potuz:Like, what happens if we're not finalizing for 90,001 blocks, and I just produce a beacon block on top of that one, like, 90,000 old block?
Potuz:You're telling me that you cannot handle this?
Marius Van Der Wijden (M):I'm telling you that we cannot handle this, but this is a different… this is a different scenario from what we've been talking about.
Marius Van Der Wijden (M):is, is, reorgs within the same chain.
Potuz:I understand, but the point is that there's already a situation in which the spec says something, which is what ideally clients should do, and what actual clients can do is true.
Marius Van Der Wijden (M):Yeah, yeah, I agree, I agree, and the thing is, we can handle 90,000… guests can handle 90,000 blocks. Good luck trying that with other clients. I think they can handle, at most…
Marius Van Der Wijden (M):maybe a thousand blocks, if you're lucky. So…
Marius Van Der Wijden (M):And even for us, the 90,000 block reorg takes several seconds.
Marius Van Der Wijden (M):Probably minutes, by this point.
Mario Vega:you know, Mark.
marek:Yeah, so what I wanted to say is that, on one point, we brainstormed the idea that basically, in case of long reorg, because it's really problematic for us to handle such long reorgs.
marek:But on the other hand, we can just switch to, let's say, archive node in this case. And basically, we will use more
marek:storage.
marek:But that's it. We will handle this anyway, so this is not fully implemented in Netherlands, we can do it like this, and I think I agree with Potus that this is what we should do.
Mikhail Kalinin:So it also kind of implies that the spec since Paris, was incorrect about what we are talking now. I mean, like, with respect to what we are talking about now.
Mikhail Kalinin:So, it should not have this optimization at all.
Mikhail Kalinin:And what worries me is that, for some reason, we just remove it now, and yeah, we're required to get reoralk to final… back to finalized.
Mikhail Kalinin:Which is probably… Fine, if finalized block is not too far from the head.
Mikhail Kalinin:But then what happens if, you know, if this starts to… if this… Is the case on some…
Mikhail Kalinin:testnet, or whatever.
Mikhail Kalinin:That we've seen in the past.
Mikhail Kalinin:So just… I'm just trying to think about potential implications of that.
Marius Van Der Wijden (M):Yeah, the bigger problem is that we don't have this finalized state until, until… basically until we get to the head. Like, we don't know when… when is the final…
Marius Van Der Wijden (M):what block is final until we get to the head? And so there's… there's, like, this… there's many cases where we don't even know
Marius Van Der Wijden (M):Where the final… where the final block is, and what… where the finalization point is.
Potuz:relevant in this case, right? Because this is only for a proposer that should be already in sync. And for what it's worth, unless we are under an attack.
Marius Van Der Wijden (M):Okay.
Potuz:I think this limit could be just 2.
Potuz:Like, we would never need to reorg more than just to the previous block.
Marius Van Der Wijden (M):Yeah, I agree. The problem is, if we, if we, if we don't put a limit there.
Marius Van Der Wijden (M):Then… and the CL tells us weird things, then we might just not be able to sync.
Marius Van Der Wijden (M):Because we keep… we keep getting, like, just imagine having, having tools.
Potuz:It's fine, that's an EL that is already a faulty EL, and you shouldn't be syncing in the case that the EL is faulty. And if the EL is not faulty, then we are under a situation in which we should be reorging.
Potuz:So, it'll cause a deadlock either way.
Marius Van Der Wijden (M):Just, like, just imagine, like, right now you can run an EL off of two CLs.
Marius Van Der Wijden (M):like, you have two CLs that feed you different points, you have one CL that is at the head, you have one CL that is at, not at the head, that is currently syncing up, and it works, because we specifically just ignore all the updates from
Marius Van Der Wijden (M):from the one CL that is out of sync. If we don't have this limit, what will happen is that this one CL will go forward, the other CL will tell us, hey, you actually need to go 5,000 blocks back.
Marius Van Der Wijden (M):We will happily go 5,000 blocks back, then the other CR tells us, no, you have to go 5,000 blocks forward, we will go 5,000 blocks forward.
Marius Van Der Wijden (M):So we will not ignore the, ignore the blocks from the out-of-state CL anymore.
Marius Van Der Wijden (M):This is… this is why this… this limit is, is there.
Mario Vega:It sounds… Barnabas, it sounds like there's… needs to be a change in this… in the PR.
Mario Vega:At least to me. Is that correct?
Marius Van Der Wijden (M):I don't think so.
Mikhail Kalinin:Yeah, my… my…
Mikhail Kalinin:Yeah, my opinion is, I'm just… I'm fine with, like, introducing an error, which will say that we will not reorog, you know.
Mikhail Kalinin:Beyond some.
Mikhail Kalinin:And yeah, yeah, and beyond some, you know, some depth, MDL will just… each EL implementation will pick up
Mikhail Kalinin:Own, limitation on that.
Mikhail Kalinin:We're gonna say that It must be at least till the finalized block, which is some…
Mikhail Kalinin:Yeah, which is kind of like…
Mikhail Kalinin:for EL finalized means that it actually doesn't know where a finalized block is, right? So it's, like, arbitrary,
Mikhail Kalinin:Depth.
Mikhail Kalinin:So we can just have some kind of 90,000 block limit on that, or whatever.
Mikhail Kalinin:So…
Mikhail Kalinin:Yeah, but that's, like, up to EL client developers to decide on what should be spec'd out there. Again, this PR tries to find a middle ground between the optimization that we had since berries, and the new requirement that Gloss actually has.
Mikhail Kalinin:That's all. If that's gonna be an error, whatever, yeah, let's just do it.
Mario Vega:Alright, can we, wrap this discussion and continue on the PR?
Mario Vega:Does anybody else have any questions about this?
Mario Vega:Alright, the follow-up question would be, is this going to be introduced in Devna 4?
Mario Vega:Is this a requirement? .
Barnabas:Yes.
Mario Vega:I feel like we're leaning towards… yes, okay.
Marius Van Der Wijden (M):Well, no, it's not a… it's not a requirement for… for BAL DevNet 4, because it's not used by BAL DevNet 4, but if we ever want to do, ePBS testing on top of BAL DevNet 4, then…
Marius Van Der Wijden (M):then we would need it, so I would also say
Marius Van Der Wijden (M):that we should do it, but it's not a… it's not a requirement for BAL DevNet 4.
Parithosh Jayanthi:I'd also say we're probably not stressing this exact path, right? We need quite deep reorgs, and right now we're still doing relatively happy case testing anyway.
Marius Van Der Wijden (M):No, ePBS, triggers this path quite.
Marius Van Der Wijden (M):Easily.
Parithosh Jayanthi:Cool.
Barnabas:Yeah, we had this issue already on .NET1, or .NET 0.
Mario Vega:Alright, regarding the requirements, is it okay,
Mario Vega:Barnabas, that we create, the…
Mario Vega:the note with the description above them before, without this PR, and then on Glamster then zero, we add this PR as a requirement for EO clients to be able to join.
Mario Vega:Can't… Okay.
Parithosh Jayanthi:And is this implementable by all clients by next week?
Barnabas:This week, end of this week.
Marius Van Der Wijden (M):Yeah, this should be implementable by all clients by the end of the call, and we have 20 minutes left, so get to it.
Mario Vega:Alright. Great, so we have agreement. Good. Do we have,
Mario Vega:do we need to break into… into break… into the breakout rooms for, discussions in specifics to block global access and ePBS?
Mario Vega:Yes, Potus?
Potuz:Before any break, I would like to talk about syncing, again, just 5 minutes.
Mario Vega:Yep, okay.
Potuz:Oh, can I do it now?
Mario Vega:Yes, yes, please do.
Potuz:Alright, so it's the same thing that I raised on ACD, and I would try to raise it again. I want to… and no one has returned on us with this. I really want to have some eyes on, potentially removing completely the payload envelope syncing… range syncing on the CL. There is…
Potuz:So the question is to analyze the security, or lack of thereof.
Potuz:from range syncing on the CL side without even requesting the payload envelopes.
Potuz:the CL could perform all of the state transition on the CL side without ever taking any payload, without even knowing if the payloads actually existed.
Potuz:And then, when it catches the head, as soon as it sends a new payload, the EL should be able to tell if it's invalid or not.
Potuz:So what I'm proposing is to change the notion of optimistic sync in this setting. So the CL would not even check the block hash of the blocks, because it wouldn't even take the payload envelopes. Another
Potuz:Problem with this would be that, the CL…
Potuz:would not verify the signature of the payload envelope, because it's not even requesting those payload envelopes. On backfield, if the CL
Potuz:wants to backfill and get those envelopes, then we could do a batch verification of those signatures.
Potuz:So I'd like to understand what is the trade-off space here, because CPBS allows us to sync without ever requesting a payload.
Mario Vega:Alright, are you looking for input from the ELs, CLs, or what's the, approach here?
Potuz:From anyone, but mostly from…
Potuz:the EL, because Optimistic Sync is something that is quite complicated on clients, and I believe that most of the edge cases had to do with the merge itself.
Potuz:So, I believe we can just remove and simplify quite a bit of optimistic thinking.
Potuz:We can just sync without a payload, and when we're on gossip, validate payload just on Gossip.
Marius Van Der Wijden (M):Hmm, that's it.
Marius Van Der Wijden (M):Does it matter for the ELs?
Marius Van Der Wijden (M):Like, as long as you, at some point, give us a payload, we will be doing SnapSync anyway.
Potuz:Well, that's… so if we trigger SnapSync, that would be a big problem. So the security assumption that we do not want to lose is that
Potuz:an EL that had the state at some point, and we're just, like, restarting or whatever it is, we do not want to trick an EL into following a finalized but invalid chain.
Potuz:If this can happen, and if this can happen because of this change, that would be a problem.
Marius Van Der Wijden (M):Okay, so what you're saying is that the year is… is… is at some point?
Marius Van Der Wijden (M):you shut it… you shut it down, you… you restart the CL, the CL does some… some sync to the head.
Marius Van Der Wijden (M):Without giving us any payloads, and then it tells us this is the block that you're syncing to.
Marius Van Der Wijden (M):And then, in that case.
Marius Van Der Wijden (M):What happens is the ER will go and… and fetch those… this data from the network anyway.
Potuz:Exactly. And today, for example, my house, like, fetches it twice.
Marius Van Der Wijden (M):Yeah, that's… Sounds unnecessary.
Potuz:Yeah, so what I would want to do is not even fetch it, so not even check the block hash, not do anything on the CL on this, and just give you, later, when I catch up, give you a block, and make sure that if this block is a descendant of an invalid block, even if it was already finalized on the CL,
Potuz:I want to make sure that the EL would just tell me this is invalid.
Marius Van Der Wijden (M):Yeah, I think that is already working.
marek:Yeah.
Marius Van Der Wijden (M):for us.
marek:But is it, like, the problem with some concrete EL, or you just want to make sure that
marek:That's how ELs work nowadays.
Potuz:Yeah, so I want to make sure, and I want to, like, to have it written somewhere so that we can actually get into agreement. It sounds to me
Potuz:that there is no reason for the CL to keep payload envelopes, and there's no reason for the CL to even serve by range payload envelopes.
marek:So, my only comment is that it makes sense to me, it's how Nethermind works nowadays, but the only thing is that it's quite a tricky scenario that you cannot easily trigger, so we need to make sure that it is really well covered by tests, and there were… some of the tests were,
marek:Flaky a little bit for us.
marek:because there are many testing hives that are checking it, that you… you will return correct, invalid hash, and so on. So, I guess it should be fine, generally, what you are talking about.
Potuz:Yeah, so what I would want… so this is the concrete request. The concrete request would be to EL devs to analyze the situation, analyze what are the security, the problems that this could cause.
Potuz:For example, one that would be dramatic would be if such a situation can trigger all ELs into snap syncing.
Potuz:Because if that happens, then my understanding is that, that could force… that could trick the EL into treating a finalized but invalid block as valid.
Potuz:And this would be an exception.
Potuz:acceptable.
marek:So, sound…
Mikhail Kalinin:Please repeat that, sir.
marek:Yeah, go ahead.
Mikhail Kalinin:Just, photos, could you please… could you please do a recap on why C… for CL with, like, in a PBS case, it's… it becomes a problem?
Mikhail Kalinin:Don't exactly…
Potuz:It's the other way around. ePBS allows… today, the CL cannot sync.
Potuz:Without getting the payload, without downloading the payload, because the payload is within… is with the block.
Potuz:So the way we sync when this EL is not catched up is we verify the block hash, either we do it, or the EL does it for us, and then we continue syncing. But we download the full thing.
Potuz:ePBS allows us to sync without ever downloading the payload.
Potuz:to which we would fail… we would just no longer verify the block hash, and moreover, we wouldn't verify the signature on the CL side that the payload envelope carries.
Potuz:But I believe that this is safe, anyways, because of what EL devs are telling me of the way they sync.
Mario Vega:Alright,
Mario Vega:I feel we have to park this conversation. Bottus, can you, coordinate to do a,
Mario Vega:write-up on this topic to see if we can keep discussing it.
Potuz:already written…
Potuz:most of the scenario up, and I've already posted it on ACD. I'll ask to, again, bring this up on ACD next time. I'm hoping to get someone to, I mean, ELDFs to look into this and actually produce some written statement.
Mario Vega:Yeah, sorry about the rush, it's that just we have to go into breakouts, because we have barely enough time. Is that okay?
Mario Vega:Alright.
Mario Vega:Yeah, let's try to jump quickly into the breakout rooms. One for ePBS, one for Block Level Access Lists.
Mario Vega:And, probably if people have more time, we can stick a little bit longer. I know these… I know these have already 15 minutes, but let's just try to…
Mario Vega:Let's try to finish the discussion topics that we have today.
Mario Vega:Alright.
Mario Vega:Who's helping today with, breakouts?
Parithosh Jayanthi:I think it's Tony and Justin.
Justin Traglia:Yep.
Mario Vega:Right, but I mean, admin contracts. Yeah.
Mario Vega:The admin controls on the… on the call.
Parithosh Jayanthi:Voloim and, Dionysus.
Mario Vega:Alright, can you guys create the breakout rooms? I don't have admin controls on this.
Parithosh Jayanthi:Oh, one sec, let me…
Parithosh Jayanthi:They should be on.
Marius Van Der Wijden (M):Do you really… do we really need one… need them?
Potuz:I don't think we underneath PBS.
Marius Van Der Wijden (M):Yeah, then just do the ballot stuff here.
Justin Traglia:Beautiful.
Toni Wahrstätter:My stuff is also very short.
Toni Wahrstätter:It depends, like, we need, like, 5 minutes, I guess.
Barnabas:People are already left at work.
Mario Vega:Yeah, they're already on the… Okay.
Toni Wahrstätter:Let's try.
Mario Vega:Let's go… let's just go there, yeah.
Mario Vega:All right.
Justin Traglia:I'm also gonna join the APPS breakout, yep.
Transcript
Transcript pending
Transcripts for breakouts are not yet processed
Transcript
Transcript pending
Transcripts for breakouts are not yet processed
Chat Logs
00:04:56
Potuz:I’d like to talk about syncing both here and ACD eventually, a few minutes at the end of the joint section
00:06:52
Kevaundray Wedderburn:This was brought up in ACDC last week too I think
00:07:04
Toni Wahrstätter:How stable is (1) ePBS devnet and (2) bal-devnet-3?
00:07:12
Barnabas:I’d like to avoid what Marius is proposing. I’d suggest we stick to having both.
I think what we could do is launch both, and use glamsterdam-devnet-0 branches for both network.
00:07:40
Toni Wahrstätter:Replying to "I’d like to avoid wh..."
agree
00:08:35
Stefan Starflinger:Replying to "I’d like to avoid wh..."
also agree
00:08:36
Dragan Rakita:Next devnet introduces few disruptive changes to EIP-8037 so it makes sense to have it separate at bal-devnet4
00:09:00
Mario Vega:Barnabas could you link the execution-apis PR pls?
00:09:01
spencer:Using bal-denvet-3 as the first glamsterdam devnet feels good to me. bal-devnet-4 might take a while to get stable, the 8037 changes are taking us a while to get right
00:09:10
Parithosh Jayanthi:Replying to "I’d like to avoid wh..."
+1
00:09:20
Maria Silva:If glam-devnet-0 is using bal-devnet-3, then I think we need bal-devnet-4 in order to complete development for 8037 and data EIPs
00:09:41
Marius Van Der Wijden (M):Okay you all convinced me..
00:11:00
Barnabas:Replying to "Next devnet introduc..."
what are these changes then?
00:11:06
Stefan Starflinger:do we need #770 for epbs + bal
00:11:10
Stefan Starflinger:execution apis
00:11:18
Stefan Starflinger:https://github.com/ethereum/execution-apis/pull/770
00:11:57
Barnabas:we should have this in bal-devnet-4 pls ^
00:12:04
Barnabas:that way we could use same images
00:12:10
Potuz:Replying to "https://github.com/e..."
Yes we do
00:12:37
Stefan Starflinger:Replying to "https://github.com/e..."
okay so we can't have glamsterdam devnet without it
00:13:16
Marius Van Der Wijden (M):Lol thats what I wanted to propose @Barnabas you just call it bal devnet 4, i would call it glamsterdam devnet 0
00:13:40
Barnabas:option 1: bal-devnet-3 + 770 = glam-devnet-0
option 2: bal-devnet-4 + 770 = glam-devnet-0
00:13:56
Dragan Rakita:Replying to "Next devnet introduc..."
Here is list: https://github.com/misilva73/evm-gas-repricings/blob/main/reports/eip-8037/spec_review_state_gas_accounting.md
00:15:57
Kevaundray Wedderburn:Replying to "option 1: bal-devnet..."
Isn’t the issue with (1) is that if there is a bug in bal-devnet-3 that arises with epbs, then it’s a context switch since clients will be working on bal-devnet-4 changes?
00:16:52
Barnabas:as long as I can have 2 ELs done with devnet 4 + 770 by wed I’m happy to use devnet 4
00:17:06
Barnabas:Replying to "as long as I can hav..."
can any other EL team volunteer?
00:20:11
Stefan Starflinger:so are we removing the variable state cost?
00:20:15
Stefan Starflinger:or not
00:20:19
Barnabas:Replying to "so are we removing t..."
pls no
00:20:27
spencer:full 8037!
00:20:30
Maria Silva:We are keeping the variable cost
00:20:43
FLCL:end of this week sounds more real
00:22:13
Mario Vega:https://github.com/ethereum/execution-apis/pull/770
00:25:56
jochem-brouwer:Sorry people I have to drop early unexpected. Benchmark update for BALs: first runs are in https://benchmarkoor.core.ethpandaops.io/suites/bc398819d1ebc628. Only Geth and Besu have full + no batch io + sequential run modes, so request for other clients to add these modes ASAP.
00:29:09
Marius Van Der Wijden (M):We only support reorgs 90k blocks deep
00:29:18
Marius Van Der Wijden (M):Regardless of finalization
00:29:43
Potuz:Replying to "We only support reor..."
That’s absolutely fine, it’s fine to have implementations not supporting the full spec, every client in consensus cannot sync all blocks
00:29:44
Toni Wahrstätter:Replying to "We only support reor..."
would you need to resync for state in such a scenario?
00:30:01
Toni Wahrstätter:Replying to "We only support reor..."
or would you keep state diffs for that many blocks in case there's no finalization
00:30:02
Marius Van Der Wijden (M):Replying to "We only support reor..."
Yes
00:30:12
Marius Van Der Wijden (M):Replying to "We only support reor..."
We need to resync
00:30:26
Ameziane Hamlat:How do the runs are scheduled ? I see a sawtooth pattern on mgas/s on besu side which doesn’t seem to be the case for Erigon .
00:31:59
Karim T. (matkt):Replying to "Sorry people I have ..."
I think you should ask on Discord Jochem is not here anymore
00:33:14
Barnabas:where is Francesco when we need him?
[Full message cannot be displayed on this version]
00:35:25
Dragan Rakita:Should EL just return error (and we put this in spec), if they can’t reorg?
00:35:49
Potuz:Yes that’s what should happen
00:35:52
Potuz:And that’s it,
00:36:02
marek:https://github.com/NethermindEth/nethermind/issues/9149
00:36:04
Potuz:But the spec should say that you’re supposed to handle up to finalized
00:39:46
Barnabas:is it me, or it sounds like everyone is just agreeing? 😄
00:40:36
Parithosh Jayanthi:Is the proposed change to specify till finalized (caveats on implementation) vs the current specific static limit?
00:41:11
Marius Van Der Wijden (M):We can just make the spec say that clients need to handle 2 block reorgs in the canonical chain
00:43:31
Parithosh Jayanthi:whip.jpg
00:43:33
Dragan Rakita:whip
00:43:37
Barnabas:ship
00:43:39
Barnabas:whip
00:43:41
Barnabas:same thing
00:44:31
Mikhail Kalinin:CL that is syncing from scratch may have a finalized block way behind the EL has, this is why I am saying that if we introduce this then EL needs to be able to reorg back up to arbitrary depth
00:47:37
Toni Wahrstätter:I replied here:
https://discord.com/channels/595666850260713488/1493309049708744756/1493563229392338994
00:48:20
Barnabas:would geth ever fall back to snap sync if it has an initialized db?
00:48:37
Marius Van Der Wijden (M):No shouldn't do it
00:48:39
Toni Wahrstätter:Replying to "would geth ever fall..."
I think only if the state got corrupted but usually not
00:51:52
Marius Van Der Wijden (M):Yeah I don't think its unsafe
00:51:52
Barnabas:so everyone agrees?
00:51:57
Barnabas:lets move on?
00:51:59
marek:yes
00:51:59
Potuz:Replying to "CL that is syncing f..."
The finalized from the EL would be the one that can’t be reforged, so it’s fine anyway I think
00:52:05
Mikhail Kalinin:Need to think more into it
00:52:36
Toni Wahrstätter:Replying to "I replied here:
http..."
that's the discussion thread
00:52:56
Potuz:Do we need one @Barnabas ?
00:53:25
Potuz:Do we need one @Justin Traglia
00:53:26
Potuz:?
00:54:36
Potuz:Where are the breakouts?
00:54:44
Potuz:I never get to see one
00:54:53
Potuz:@Parithosh Jayanthi can you add me?
00:55:07
Maria Silva:I am not seeing the breakout as well..
00:55:22
Stefan Bratanov:Same here, not seeing it this time
00:55:26
Kevaundray Wedderburn:Can click “More” then “breakout rooms”
BAL Chat
10:53:47
Marius Van Der Wijden (M):64 lets gooo
10:53:59
FLCL:why not 32?
10:54:03
Marius Van Der Wijden (M):65tx per block lets gooo
10:54:14
Marius Van Der Wijden (M):Replying to "65tx per block lets ..."
65k*
10:55:14
Karim T. (matkt):64 ok for Besu : already using that
10:55:30
Dragan Rakita:That is impl detail, right?
10:55:44
spencer:Merged in EELS. EELS is the truth. Must go 64 now sry 😓
10:55:46
felix (eest):hyperoptimizations shouldnt affect spec, we should go with intuitive 64 imo
10:56:05
Stefan Starflinger:you mean you could use 32 bit even though the spec says 64?
10:56:20
FLCL:better to use the same types across clients
10:57:00
Ben Adams:32? Or expecting more than 4Bn?
10:57:06
Marius Van Der Wijden (M):Lets use 32 Byte
10:57:15
Marius Van Der Wijden (M):256 bit :P
10:58:27
Ben Adams:u32
10:59:03
FLCL:is it very hard to implement 32?
10:59:53
FLCL:tyvm
11:00:25
Jared Wasinger:We can’t even push close to these limits because of the 8mb block size limits
ePBS Chat
10:54:33
Barnabas:@Eitan Seri-Levi whats your branch called?
10:54:50
Nico Flaig:that was from NC not me
10:55:05
Eitan Seri-Levi:Its not ready for kurtosis but its fix-pruning-gloas
10:55:07
Justin Traglia:Oh oops. Sorry about this. This is the PR: https://github.com/ethereum/consensus-specs/pull/5133
10:56:07
Potuz:When you guys say fork choice tests you mean pre-gloas right?
10:56:35
Barnabas:Replying to "Its not ready for ku..."
please reach out when you feel ready!
10:56:53
Justin Traglia:Replying to "When you guys say fo..."
No this is for Gloas
10:57:08
Eitan Seri-Levi:Replying to "Its not ready for ku..."
Will do!
10:57:13
Potuz:Replying to "When you guys say fo..."
What tests?
10:57:22
Potuz:Replying to "When you guys say fo..."
We definitely don’t have any gloss-specific tests
10:57:30
Justin Traglia:Replying to "When you guys say fo..."
I’ll find a link, one minute.
10:57:33
Potuz:Replying to "When you guys say fo..."
Only pre-gloas tests that are run under gloas
10:58:12
Barnabas:ethpandaops/geth:glamsterdam-devnet-0
10:58:17
Stefan Bratanov:Replying to "When you guys say fo..."
there are on_execution_payload_envelope tests and stuff
10:58:43
Justin Traglia:Replying to "When you guys say fo..."
https://github.com/ethereum/consensus-specs/blob/master/tests/core/pyspec/eth_consensus_specs/test/gloas/fork_choice/test_on_execution_payload_envelope.py
11:00:37
Justin Traglia:Would these be programmatic?
11:01:18
Justin Traglia:Tysm 🙂
11:01:32
Barnabas:Replying to "Tysm 🙂"
don’t reveal our secrets
Summary
8 highlights
· 3 decisions · 1 action itemsExperimental
Summary
8 highlights · 3 decisions · 1 action itemsExperimentalfork status and schedule
testing progress
execution apis
Decisions
Action Items
- EL teams: EL teams analyze security implications of CL syncing without payload envelopes00:51:52