Pooja Ranjan:We are, live.
Transcript
Mario Vega:Thank you so much. Hello, everyone. Welcome to Acdt number 47. Today's August 4.th
Mario Vega:We have a couple of items in the agenda starting by Fussaka. So yeah, let's just get started. I'm gonna ask, is Barnabas online?
Mario Vega:I think so. Hey? Yes.
Barnabas:Okay.
Mario Vega:Updates on Fuzaka. Devnet 3.
Barnabas:Yeah, currently, we have 2 that are struggling on business, 3 nimbus and roadstar.
Barnabas:And also captain, seems to be offline right now,
Barnabas:we have 90% participation. So every other client seems to be doing pretty well the all the initial issues with the suggested values seem to have been resolved. There's only one more issue on Low Star, where, right after the startup, the values are not set according to the finalized epoch
Barnabas:in the Genesis State. But I've notified them about this issue.
Barnabas:Maybe someone from Losar can give us an update in.
Matthew Keil:We're looking at getting the update for going off the genesis validator set. We're looking at how we can actually implement that. And then the Sync bug, I am still actively working on it. It's been proven to be pretty difficult. We're doing some refactoring in order to just improve the code base, because, it's what's making it challenging in order to actually get it resolved. But we are actively working on it. And we have a good plan to going forward, so we should be should be there soon.
Barnabas:Right, and on the number side we seem to have some super nodes that are having hard time getting peers. So most likely they got rate limited or banned by others.
Barnabas:Maybe someone from the nimbus team can also speak up.
Mario Vega:I don't think it's anyone from from nimus team at the moment.
Mario Vega:Maybe you can get back to this after they join.
Mario Vega:If they do
Mario Vega:anything else on the my, my question is, is vpo activated. Any Vpo activated on the.
Barnabas:Yeah, all all 5 have been activated. And we are currently hitting the 20 blobs. Usually, we had some issues regarding the mev relay.
Barnabas:There's some transaction propagation issues. And hopefully, we can get sorted.
Barnabas:But also the builder workflow seem to be working on every single client fair, which is very good.
Mario Vega:Anything that we need to test on the Meb
Mario Vega:workflow for Bpo. My, I mean, I guess it was fine on this. But is there any
Mario Vega:lesson learned, I guess, from activate the Mepo on on the Mev side?
Mario Vega:Or was it like smooth.
Barnabas:No, that's that's pretty smooth right now. The major issue on the Mev side is we're trying to create a private pool. And
Barnabas:we send the transactions the spammers, sending all the transactions to just the mev relay machine that is running the rest node. And this ras node is unable to share transaction with the R builder. And that's where we are struggling to basically rebroadcast these transactions and include them.
Bharath:Yeah, I have a, I like, basically an update on that. I think so, yeah, so basically, like, the our builder uses Eth new pending transactions, the subscription Api to get transactions from the Red Mempool
Bharath:and so we've disabled transaction pool
Bharath:like local transaction propagation in the red node and be. And you know, today I was just talking to the red team. And it's because of that flag
Bharath:transactions are not being streamed. So we currently have like, 2 ways to get around this right now. One is
Bharath:yeah. Our builder has implements et Ross and transaction. It implements its own Json. Rpc, server. I. We are like, we can try like sending transactions from spam or to that
Bharath:Rpc, to that Rpc server, and that doesn't go through the Red Mem pool. So that's an that's our bill. Their own transaction pool, if we can. That's 1 way yeah. So and the other way is like,
Bharath:like, I'm seeing I'm just talking to the red folks like, if they can probably like, just like, remove that flag. And
Bharath:we can implement. So that currently doesn't have a flag to disable transaction. Gossip can somehow get that flag and could probably remove the local transaction propagation flag and things should work. So we currently have, like a couple of ways forward. There's some complications with spam, or, I guess, with directly sending it to the builder. But but yeah, like, we've identified like, why, transactions are not getting included in the block with
Bharath:when we send only to the Mev relay machine as Barnabas mentioned.
Bharath:Yeah, I'm currently like exploring, like, what are the best ways forward? I think we have like 2 ways, and I'll I'll I'll see like I'll give an update like when it when we finalized on something.
Mario Vega:Everyone.
Roman:Yeah, quick question. You said that rath has a flag to disable the expo local propagation. You said that it's not working.
Bharath:No, no, that's working that's working
Roman:Okay.
Bharath:I've put in the Devnet. Mav. Tg, chat. There's another
Bharath:there's a flag for which you there's implemented for optimism to disable transaction gossip
Bharath:in the peer to peer network, which is not, which is not there for the red.
Bharath:Like the ethereum. l. 1 el that flag. Maybe, like you can have that flag. But yeah, the transaction pool flag to disable local transaction propagation that was working fine like.
Roman:Okay.
Bharath:You, we can follow like Roman, you can follow the the chat in the Devnet Mab testing. Tg, I think you have context there, like.
Mario Vega:Thank you.
Mario Vega:Any other findings on Defnet 3. From any other clients.
Mario Vega:debugging issues or or similar?
Mario Vega:Alright, was was that everything that you wanted to share Barnabas, or was there anything else that you want to touch upon?
Barnabas:May. Maybe Agnes can give us an update now from Nimbus as well.
Barnabas:cause he was just joining.
Mario Vega:Thank you.
Agnish Ghosh:Hi, so we had a couple of issues. Last week.
Agnish Ghosh:One was, I think, each das guardian
Agnish Ghosh:we were having a custody column mismatch.
Agnish Ghosh:So whenever it does guardian requested us with all columns, if we were a full node, we would return with
Agnish Ghosh:our custody.
Agnish Ghosh:The columns we would custody. But surprisingly, somehow, the column indices don't match.
Agnish Ghosh:I think the indices don't match after a couple of validator custody updates.
Agnish Ghosh:We'll just check into our code. If there is some hash function stuff going on such that the indices are being inaccurate after
Agnish Ghosh:couple of epochs, because I think we do pass ef test vectors. So
Agnish Ghosh:so I'm trying to looking, I'm trying to look into that apart from that. Yes, yeah. The the earliest available slot
Agnish Ghosh:it was was not moving. So right now that thing
Agnish Ghosh:looks fixed to me. We have a build
Agnish Ghosh:triggered. I think it should be picked up in Devnet 3 in a while
Agnish Ghosh:and general updates. We had mev issues they are fixed now. So we, I think Bharat can confirm, or somebody maybe Barnabas can confirm. Nimbus is. I think it. They are proposing Mev Block.
Agnish Ghosh:Yeah. So what else?
Agnish Ghosh:I don't think anything else I mean whatever. Oh, yeah, okay. So
Agnish Ghosh:there's been an issue being raised in discord where
Agnish Ghosh:members aragon one and get one are having a tough time to sync and and of course,
Agnish Ghosh:we are. We are basically being asked, why are we proposing? Even
Agnish Ghosh:when we are not caught up to the head.
Agnish Ghosh:But I think that's like that's something we have by design. If we are short range thinking, and we are like not
Agnish Ghosh:2 to 3 parks away. We don't turn off validator duties.
Agnish Ghosh:Yeah.
Agnish Ghosh:So if we are unable to sync, it's definitely a da or a peering issue, either, we are not getting all the required columns on time, or we are not being able to peer. Well. So
Agnish Ghosh:I'm currently looking into that numbers. Aragon one get one. They have some sort of peering issues. Yeah.
Agnish Ghosh:so yeah, nothing else. Nothing else from our side. Yeah. Overall. I think there were a couple
Agnish Ghosh:couple of peering issues. Mev issues indefinite 3. They're all fixed now.
Agnish Ghosh:And yeah, hopefully, I'll fix
Agnish Ghosh:the das guardian thing. By next couple of days, and what else?
Agnish Ghosh:I think there is a there is a crash Barnabas sent me
Agnish Ghosh:it was a seg fault. It's happening in a very specific scenario.
Agnish Ghosh:I suspect it's happening because of
Agnish Ghosh:indicating the earliest available slot incorrectly or something of that thought. I'm currently looking into that. So yeah, nothing else from us. Yeah, that's it.
Mario Vega:Thank you. Mark.
marek:Yeah.
marek:I just want to mention that. Never mind has an early integration with our builders. So it would be great if we can try this as well. And by early, I mean, of course, it does require more testing, and so on. And it is not being run on production.
marek:And also it's like integration with State. So never mind, can work with our builder as a State provider, but it still uses our evm. Us. For execution. So it's it's only replaces the state. Yeah. But just wanted to mention, because I heard some discussion about
marek:our builder and this kind of stuff let me share the link.
marek:Alright, that's all from my side.
marek:Is it fully ready? So, as I, as I mentioned the state is is ready, of course. We would appreciate more testing here and then that would be great here.
marek:And but Evm part is not being done.
Mario Vega:Thank you.
Mario Vega:Yeah. Any other updates from anyone else on any any team?
Mario Vega:If not. Yeah. Thanks. Thanks for the updates. And it seems like we're doing fine on the military. Then.
Mario Vega:What's the follow up plan here? Do you know, Barnabas, do we plan to keep definitely running? And what's the next steps? Or
Barnabas:So ideally, we want to have Nimbus and North Star.
Barnabas:basically bug free before we go on to the next step. And we have 2 more decisions to make today. And based on that, we're gonna decide. If you want them at 4 or not.
Mario Vega:Perfect awesome. Yeah? All right. Yeah, then, let's just move on to the next discussion topic, which is related to Eip 7,825, the transaction gas limit gap
Mario Vega:there was a discussion on
Mario Vega:last Thursday about rolling back the gap well up, rolling up the limit to 30 million instead of the 16 million that we have. Currently, I think there was an update from the person that shared this on act, and whose major concern was raised on Acd. Sorry on icde last week.
Mario Vega:and I can share that response here. If this person is is in the call. Currently
Mario Vega:but otherwise it seems like the main concern is not really concerned from from the read of the text.
Mario Vega:is there anyone here that can elaborate on this?
Mario Vega:I don't think I don't think this person is on the call.
Toni Wahrstätter:Yeah, maybe I can. I can stay a little.
Mario Vega:Okay.
Toni Wahrstätter:Yeah, it was a concern that there might be some apps that might be affected. For example, very complex defi transactions that go through many different pools.
Toni Wahrstätter:and Duncan brought it up, and that of course, it's
Toni Wahrstätter:it could be a backwards compatibility issues. But it seems like, after waiting. Yeah, the trade-offs.
Toni Wahrstätter:We can just keep it as is as we have it on Defnet 3.
Mario Vega:Sounds perfect.
Mario Vega:Any other opinion on on this matter, or this is gonna be like a very short discussion?
Mario Vega:Alright, yeah, if nothing else. I think this is the way to go. We keep the limit in 16 million.
Mario Vega:and which is great news for testing, because we don't have to perform. The analysis of the testing
Mario Vega:back doors are breaking again. So yeah, great thanks.
Mario Vega:All right. Next topic is, this is a recurring topic on gas limit testing updates.
Mario Vega:I think Barry is the main person to reach out to this. Or do you have any updates, Barnabas, that you want to share about Gatlin testing, or there's nothing to discuss. Maybe today.
Barnabas:Not from my point of view. No. And I think part is available today.
Mario Vega:Alright, no worries.
Kamil Chodoła:Only thing.
Mario Vega:Come in. Yeah.
Kamil Chodoła:I can chime in, yeah. Hi, Albert, Hi, everyone. So yeah, basically, we are still trying to achieve the stateful testing being capable. But we are tackling that from various perspectives, and we still need to verify few things. Marching also is trying to do some work around our test. Generators in gas benchmarks to see if this will be any more helpful and will speed up the development.
Kamil Chodoła:In meantime, esta integration working with you guys to to make it to runnable on our infrastructure, and I I believe I am almost done with that, like I need maybe one or 2 more days just facing some unexpected issues with that.
Kamil Chodoła:And in the meantime Pari was discussing with with all of the teams, checking our readiness, maybe for any other gas limit upgrade in future.
Kamil Chodoła:So we have another version of gaff checking. If modex scenarios are being improved.
Kamil Chodoła:and in general I would say yes. A lot of modex scenarios are even twice or 3 times better than it was before, but still the worst cases are not being properly attacked. Maybe there is some
Kamil Chodoła:some tweaks which need to be added on the, on the adjustment made on Gef. Plus there are some regressions. So there are scenarios which are performing 50% worse. Comparing to the to the previous results. So, yeah.
Kamil Chodoła:so
Kamil Chodoła:this would be like the major concern about any further increase. But if this will came up, being good, which is, looks like it is right now, but just needs few more tweaks. Then we can consider doing kind of thing forward.
Kamil Chodoła:But still, also, the stateful scenarios are really really needed. And also we have a bloatnet which
Kamil Chodoła:have already stayed on 2 x 2 x size, and we are preparing for some test cases for some testing.
Kamil Chodoła:Yeah, so hopefully, some results will be this week.
Mario Vega:I can also update on the East side. So what we propose on doing there on the stateful testing is so we have the execute command that we use to run our test in in in running devnets.
Mario Vega:And the idea here is that we prepare some test cases where we rely on a contract that has a very large storage or state size to be already present in the chain, which I think is the case for Blotnet, and a lot of the contracts are in there. So basically, it's just a matter of updating our repository to receive some flag.
milen | Erigon:To point.
Mario Vega:To point it to to the contracts that are already existing there. So it's gonna be. It's not a trivial change, but it's also not not not breaking. But yeah, we'll continue working on that. And and we can keep you updated.
Mario Vega:Marshine brick. Yeah.
Marcin Sobczak:Hi, I would like to say that we are developing one more tour tool to to check the potential issues with modex repricing. It's Rpc. Endpoint price block. But with custom spec. So what we are doing
Marcin Sobczak:we have a script which is getting a trace from like current chain. So on, Petra and another one doing the same tries, but with Fussaka spec.
Marcin Sobczak:and we are comparing receipts like. The difference in gas usage is expected but we are trying to to check if there will be some other changes like broken contracts, etc.
Marcin Sobczak:for now it's I think, like proof of concept. It's it's working. But we didn't perform the actual test yet. But yes, we are planning to do it soon. So we'll keep you updated.
Mario Vega:Thank you. Men.
Ben Adams:Would it be 60 million? The next level that we'd consider.
Kamil Chodoła:I believe if we'll consider anything like 60 should be the next step. But we need to perform a lot of testing around that. But yeah.
Kamil Chodoła:small but still significant. Jump.
Mario Vega:Yeah, I think we can prepare testing for next couple of weeks and gather some benchmarks before continuing our decision.
Mario Vega:Is there anything that you need from the tests
Mario Vega:side, Camille, a new release? Or how are you looking? There.
Kamil Chodoła:As I mentioned on some of the Tg channels, is like new release with new tests. Which you you added Louis added, around modex as well, plus change of the gas limits. But this is also like being discussed right now on on how many of those we need. But yeah, to unify the it a little bit more.
Kamil Chodoła:And yeah, basically the recording stuff around genesis files which I'm fighting right now. But yeah, after after being able to run those tests on our infra. I will let you know if there is anything else to be improved.
Kamil Chodoła:And then from the statefuls perspective. Yeah, we need to
Kamil Chodoła:think a little bit more around that, and hope to help to make it. The workaround, like execute the capability, is a good workaround, for now, which probably we can use, but will be very time consuming to to execute everything properly.
Kamil Chodoła:But yeah.
Kamil Chodoła:we just need to figure out we, and also the tool which Martin is trying to revive and adjust, to, to be able to generate on top of existing state and the custom Genesis files would be also helpful there. But yeah, we we are
Kamil Chodoła:fighting with that right now.
Kamil Chodoła:So whenever there will be anything to to to help us with, I will let you know.
Mario Vega:Excellent. Thank you so much. Great and just bringing this up, surfacing this this message from Barry. He kind of speed right now, but it seems like the get team
Mario Vega:would take some time to upspring any of the changes if they decide.
Mario Vega:Is there anyone from the get team here that can
Mario Vega:a little bit more context to that. Or
Mario Vega:if not, that's not a problem.
Mario Vega:I don't see anyone from Geth.
Mario Vega:All right. I think we can just follow this Async.
Mario Vega:Alright cool.
Mario Vega:All right. I think. We can just pass until on on the next
Mario Vega:Next discussion topic, I'll leave the intervals per slot discussion topic at the end. We can just jump into right into the eips, 79, 51 benchmarking results. These were shared from the Bessu team. Anyone from the Bessu team that would like to expand on this
Mario Vega:this topic.
Justin Florentine (Besu):Hey there! Unfortunately, the folks that are curious about that are in the aus time time zone.
Justin Florentine (Besu):This is in reference to the gas improvement that we found with the r. 1 curve. Is that correct?
Mario Vega:Yeah. Exactly. Yep.
Justin Florentine (Besu):Yeah,
Justin Florentine (Besu):I don't really have anything in addition to add other than the notes that have been posted. Sorry for that.
Mario Vega:Yeah, no worries, but it seems to me I I've read through the comment, and it seems to me that it's
Mario Vega:It is not a concern of how the the new precompile is.
Mario Vega:Is is what's the cost on it? So it is not. It doesn't seem that it's a concern. Actually, it seems like the
Mario Vega:The cost is is too high, maybe from the new precompile. But I think that's not something that we're willing to discuss today. Maybe.
Mario Vega:Yeah, if there's nothing else on this topic.
Mario Vega:Maybe we can continue to sync.
Justin Florentine (Besu):I think that would be fine. Thank you.
Mario Vega:Yeah, alright. Thank you so much. Yeah. With that, I think we can just jump into the intervals per slot. Justin, do you want to. Take it up.
Justin Traglia:Sure. Hello, everyone. I would just like to raise awareness for this. Pr, I just shared the link.
Justin Traglia:It essentially defines explicit slot component times in milliseconds as a percentage of the slot duration time and as a configuration variable instead of a preset like youth, Panda Ops requested, and potus
Justin Traglia:I'm not asking for us to decide on this today, but I would like to
Justin Traglia:make a decision, maybe at Acdc. Whether or not to merge it sooner than
Justin Traglia:like 3 months, 3 or 4 months from now. So that we can start working on other specifications. It would be beneficial for epbs like Efp, 7, 7, 3, 2, and 7, 7, 8, 2 shorter slot times.
Justin Traglia:it wouldn't actually change anything. It would just add additional configuration values or sorry. Yeah, additional configuration values.
Justin Traglia:but no substantial changes. It would deprecate intervals per slot.
Justin Traglia:But once again, you don't have to stop using that immediately, so
Justin Traglia:I see no reason why we couldn't merge it now.
Justin Traglia:Fucking down.
Mario Vega:I think the main question here is whether this breaks Fussaka or changes anything in Fussaka.
Justin Traglia:It really shouldn't. No, I mean, clients might have to update
Justin Traglia:code to handle the new configuration values, but they could essentially ignore them until Clamsterdam.
Justin Traglia:or until we start working on claim studio.
Justin Traglia:Yeah. Shouldn't
Justin Traglia:do clients have any opinions on this or
Justin Traglia:maybe just look into it and discuss it later.
Justin Traglia:Please leave reviews.
Barnabas:There was another question regarding possibly moving some values from config to presets especially the number of columns. I think.
Barnabas:Yeah, this is.
Justin Traglia:That is a separate issue.
Barnabas:But yeah, yeah, it's a separate issue. But I'm I'm just curious. If we should actually take a look at all the different config values. And consider moving some of them to preset, because I feel like there's more than just one.
Barnabas:Something in the.
Justin Traglia:Yeah, a number of columns could be a preset, probably should be.
Justin Traglia:It also doesn't actually have to exist, since it will be all it'll always be the same as
Justin Traglia:Salisberg's, tended Bob, but it is a nicer, variable name.
Barnabas:Yeah, then maybe we can just get rid of the unused variables.
Barnabas:It would also just clear things up a bit.
Justin Traglia:Yep.
Mario Vega:And Pawan mentions in the chat that he will review it soon.
Mario Vega:but it looks good at 1st glance.
Justin Traglia:Thank you.
Mario Vega:Yeah, if the rest of the teams can just give it a look. And I guess Acdc is the way to continue this conversation and see if it gets merged.
Justin Traglia:Yes, that's right.
Mario Vega:Cool.
Mario Vega:Yeah, thank you very much.
Mario Vega:alright. I think that's all of the topics we had to today. So I can open up the floor to any other points or discussions that we have.
Barnabas:There's 1 more
Mario Vega:Yeah.
Justin Traglia:Yeah.
Barnabas:Regarding 4, 4, 7, 7.
Mario Vega:Is this related to sorry.
Barnabas:This is related to Futaka. Basically the the proposal was from Lido to increase the balance for additional custody groups.
Barnabas:So basically, every single 128 Eth. Would increase the number of columns that the validator would need to custody, and
Barnabas:they push the decision making to today, instead of making a decision on Thursday, because it was very last minute on Thursday.
Ben Adams:And this would need a new devnet.
Justin Traglia:Personally, I'm against this change. I think we just keep it. How it is currently.
Mario Vega:Yeah, I think this is the only feature that requires a new right. It was before this call. It was this, and also 78, 25. But now it's just reduced to this only change for a new.
Mario Vega:So with that in mind, with that in mind, do we?
Mario Vega:Is anybody Bro.
Mario Vega:Doing this at this call the ones to raise the the point.
Barnabas:There was lots of support from different cl devs. So that's why I thought
Barnabas:this should probably bring it up again.
Mario Vega:I see
Mario Vega:I think the moment is now, because otherwise, if we delay this further, I don't think it's gonna be any any good to the development processing. It's already too late. So either we.
Mario Vega:either someone from the Cl that is supporting this
Mario Vega:speaks up now, or we simply just reject it?
Mario Vega:If so, if there's anyone in this call that wants to support this, please speak up now.
Csaba Kiraly:I just just wanted to say the time
Csaba Kiraly:I'm against the change, because it's so. The this is last minute.
Csaba Kiraly:because it we might with Bpo scale to a high number, and we think that with the Bpos we will not have a chance to do it later, so we would have to do it. Now.
Csaba Kiraly:my take on. This is that if if we, if we will need these kind of changes.
Csaba Kiraly:then we should do this together with the Bpos, and not quickly now.
Csaba Kiraly:So that's why I would, I would say, don't do it now, and.
Barnabas:I'm actually super against doing that. I wouldn't want to include anything else in Bps.
Mario Vega:Yeah, if we add this, it would be like a new fork. Actually, I don't. I don't see how it could be. Only vpo.
Csaba Kiraly:Yeah, it wouldn't be proposed, because not. It's not not a broad parameter. So by definition, it's not a Bpo. Then
Csaba Kiraly:I'm just saying I'm I'm skeptical about Bpos being Bpos.
Csaba Kiraly:For because of these and and and the other things that will come up similar to this.
Barnabas:I think if we say that we're gonna reject this from Saka, we might consider it for Amsterdam, but not before that.
Barnabas:I wouldn't consider any substantial code change for any Bpos.
Mario Vega:Yeah, I think that's the general sentiment right now. I don't see anyone pro
Mario Vega:Agnish.
Mario Vega:First, st Anisha nico.
Agnish Ghosh:Yeah. So I'm against the change. Simply because I think if we raise it to 1, 28, the number of super nodes decrease, and the number of nodes with a variable dynamic da of sorts which would keep swaying based on attached validators. That thing is tricky, usually like
Agnish Ghosh:on on a Cl implementation level. I have discussed this with multiple people, Jimmy. Manu
Agnish Ghosh:It's tricky for all of us to sort of
Agnish Ghosh:declare da and keep blocks resolved unresolved across various stores. Based on the dynamic da. I have. I have probably raised concerns about this
Agnish Ghosh:with Francesco and a lot of other people over the last couple of weeks. I know that. Okay, we have this design and this design is there for a reason. But
Agnish Ghosh:I would definitely want to see more super nods.
Agnish Ghosh:Hence I'm against this change.
Agnish Ghosh:I I just don't want more nodes with this swaying, dynamic da condition.
Mario Vega:Got it. Nico.
nflaig:Yeah. Just want to bring up that. Because I saw in Glamsterdam we could change the value. But I think, since we cannot reduce the Cgc. I think this would not be possible right.
nflaig:because the existing notes would just keep the previous custody they had.
nflaig:But I'm also not strongly for this change. I think it's a bit late, and I guess, too, too little analysis is done
nflaig:to do this. Now I think.
Mario Vega:Thank you.
Mario Vega:Alright. Yeah. It seems like the general consensus is that we do not go forward this change so that would mean that we wouldn't have them before.
Mario Vega:That means that just going back to the point that was raised earlier by Barnabas, we only need nimburs and load. Start to be bug free before continuing. So I guess instead of any changes, we just focus on fixing
Mario Vega:box this week, and we follow up next week with the next steps on this.
Mario Vega:All right.
Mario Vega:Yeah, thank you. Thank you. Everyone.
Mario Vega:Sunnyside lives. They wanted to share something.
Mario Vega:Is anyone from Sunnyside lives here that wants to present.
Jae Hee Lee:Yeah. Hi, so I think I shared the link a while ago. So I will just reshare it.
Jae Hee Lee:Yeah, we've just released a new report, which contains mainly on
Jae Hee Lee:high block loads, plus regular transactions, and perfect perfect pdos.
Jae Hee Lee:So for the 1st one, we sent
Jae Hee Lee:high block Britwood on the around 22 million gas.
Jae Hee Lee:and we were able to find find that we could reach up to around 60 blocks per block with that load, and there wasn't big impact on bandwidth usage.
Jae Hee Lee:So it was going quite well.
Jae Hee Lee:And then we further limited the bandwidth
Jae Hee Lee:of each node for both shipper node and full nodes as per eip. 7, 8, 7 0 hardware requirements.
Jae Hee Lee:and it the network
Jae Hee Lee:performed poorly that it only got 20 blocks per block, but later Francesco suggested that
Jae Hee Lee:maybe we should be less restrictive on supernodes, that
Jae Hee Lee:100 mega megabits per second for download and 50 for upload is too low, so
Jae Hee Lee:from next round we will retest this with unrestricted bounds for sugar notes.
Jae Hee Lee:Assuming supernodes will be available for one Gigabit
Jae Hee Lee:network will be available for Schmutz
Jae Hee Lee:and for the second network for the perfect periods which is having 16 8 validator nodes
Jae Hee Lee:uniquely custoding one each column.
Jae Hee Lee:and we were also able to achieve 60 blocks per block on the 22 million gas usage, and also 30 about 30 blocks per block with bandwidth limits.
Jae Hee Lee:However, we found that Genesis sync was failing from full app books on perfect theaters network. So
Jae Hee Lee:We are looking into it, but it seems like
Jae Hee Lee:There are only 8 peers on sync nodes, and
Jae Hee Lee:the sync nodes are trying to find
Jae Hee Lee:the columns their peers don't have, and
Jae Hee Lee:it just fails at the very early stage at full
Jae Hee Lee:we also did backfill test on Teku
Jae Hee Lee:we found an issue reported to tech team, and they are now fixing it. We also would like to test other sales backfill once they are ready.
Jae Hee Lee:and very lastly,
Jae Hee Lee:for nimbus and load star. Please let us know when you guys are ready for
Jae Hee Lee:block group protesting, and other
Jae Hee Lee:validate validate the cost of the test tests that we have, because for numbers I heard that
Jae Hee Lee:in versus having pending optimizations so that
Jae Hee Lee:hype high block triplet isn't available yet, and for loadstar.
Jae Hee Lee:Full notes are subscribing to more than
Jae Hee Lee:8 validators. Nearly as same as super nodes. So please let us know when it's not ready?
Jae Hee Lee:Yeah. Oh, sorry. I say, hands up. So.
Mario Vega:Yeah, not. Just.
Jae Hee Lee:First, st yeah.
Mario Vega:Yeah. Manu Manu was 1st and then Agnesh. Thank you.
Manu:Hi, I can quickly give an explanation about perfect pureness, about syncing in perfect pureness.
Manu:why it fails, and why at least, prism fails. So by the definition of perfect, peerless
Manu:another has absolutely 0 peer
Manu:clustering the same command than the node. So if you start from Genesis, or even from from
Manu:even from a checkpoint. Then you want to fetch some some columns. And yeah, you don't find any peer. So the only
Manu:possibility you have the knowledge to
Manu:is to retrieve any 64 Coltons, you know any one of them from from your peers and then reconstruct the colon. You you need
Manu:yeah, prism doesn't do that. Currently, I'm I'm we are working on it.
Manu:Target is this week and end of this week.
Manu:And I'm not sure that a lot of other clients already implemented. This this feature, sinking in perfect purdues.
Jae Hee Lee:Yeah. We also tested lighthouse, and Granddyne and
Jae Hee Lee:lighthouse doesn't seem to sync any fully slots, while Grandine, I think Granddyne also stops at very early
Jae Hee Lee:fully. Epoke.
Manu:Actually, I think everybody can can sync flu, but without any blob. So the the connection, which is not okay now is post flu with blobs.
Jae Hee Lee:Right, right, right.
Mario Vega:Yeah, thank, you.
Jae Hee Lee:I'm.
Agnish Ghosh:Yeah. So currently, members, supports up to 50, I think 52 blobs from the last
Agnish Ghosh:from the last of test that Sunnyside labs ran,
Agnish Ghosh:so is there. Is there an actual priority to
Agnish Ghosh:actually work on optimization that can raise up like upwards of 50 right now, like for Fusaka. I I just wanted some clarity on that, because.
Agnish Ghosh:Yeah. Yeah. Okay, go ahead.
Jae Hee Lee:Yeah. But I wouldn't say that was a stable achievement.
Jae Hee Lee:that the changes failed after that flow right.
Agnish Ghosh:Right, right.
Jae Hee Lee:I'd say 40 plus is a safe margin for Nimbus when we test it.
Agnish Ghosh:Okay.
Jae Hee Lee:At that time. But yeah, are we targeting something higher, at least for in-in the idea that not seen scenarios, I think we should get something higher.
Agnish Ghosh:Do we actually need something above 40 for Fusaka mainnet
Agnish Ghosh:like? That's the open question. I'm not.
Agnish Ghosh:I'm not sure.
Jae Hee Lee:I I think someone else could clarify.
Agnish Ghosh:Yeah, exactly. If someone else can let me know, because, this work can be done. It's just not on priority. So
Agnish Ghosh:oh, yeah.
Mario Vega:I'm gonna pick on you, Barnabas. Do you feel like this is a requirement for next devnets or main shadow forks, or
Barnabas:That really depends on what kind of Bpo values we expect to do before Amsterdam.
Barnabas:We haven't really been down the rabbit hole of deciding Bpo values. Yet
Barnabas:I think it's gonna be a totally different question. But I would expect that we want to be able to do 72 blots before transgender them.
Mario Vega:So it feels like right now. It would not be a blocker, because we have not decided on Bpo. But before deciding on Bpo's we have to have this working right?
Barnabas:Yeah, I mean, we don't need to ship right now. 72 blob support. But technically,
Barnabas:without any physical hard work, we should be able to scale up to 72.
Mario Vega:Okay. Manu.
Manu:Yeah. Sorry.
Mario Vega:Yes.
Manu:It was because it was before. So, thank you.
Manu:Actually, yeah, there's a question. There is some question in chat about syncing, in
Manu:thinking, in perfect paradise. Francesco is asking if it is.
Manu:If do we really need this thinking in perfect, careless mode?
Manu:I think we need? I mean, I will be more
Manu:comfortable launching flu with this feature.
Manu:Because I think that Evan is superno that are really helpful.
Manu:I think a nod should be able to think.
Manu:even without any super node in its peer. Maybe because not enough supernode will be in the network. Maybe so, maybe super node will reject incoming peers.
Manu:And so that's why I think that we should be able to think in perfect parallel mode.
Jae Hee Lee:Yeah. Adding some data that I got which which is also mentioned in the report. But what what I saw was
Jae Hee Lee:what we could guarantee so far was that with the tested Ceos
Jae Hee Lee:for the 1st 1,000 slots.
Jae Hee Lee:Every node seems to have their columns correctly so.
Jae Hee Lee:But it would be definitely much more feeling, much more safer to have this feature and imperfect theaters think available.
Mario Vega:Okay?
Mario Vega:And there's another comment from Francesco. Francesco, do you want to raise this comment or.
Francesco:Yeah, I mean, basically, I was just saying that I'm not saying that you necessarily need to have or like, I wouldn't expect that.
Francesco:Okay, let me backtrack a second. Yeah, I I wouldn't want the requirement that you need a super node in your peers that that's not what I'm saying more just. I'm wondering
Francesco:if it is reasonable to expect things to work when you don't have anyone in your peers that has the columns that you want, and not only you don't have in your peers, but you cannot find someone in the network to be your peer that has those columns
Francesco:like. I just don't understand how you would even exist in the network afterwards. Like you want columns, you know, 1, 7 and 8, and you can find anyone that wants those columns that look like also kind of has those columns, and generally wants those columns like you could maybe get the other ones and and sync by doing this trick of getting more of them and reconstructing. But eventually you'd just not be able to
Francesco:keep going on the network.
Barnabas:So you're basically saying that a full node shouldn't be required to reconstruct in order to sync.
Francesco:I just unclear to me that
Francesco:there is necessarily that much value to it like I
Francesco:seems to me that if you cannot find anyone that is willing to give you the Comms that you have. You have a much bigger problem.
Francesco:Like. You're just even if you manage to to solve the kind of immediate problem by doing this reconstruction thing that doesn't. That still doesn't seem like it leaves you in any kind of a good position like in in this perfect beard. Us? Setting. That's maybe okay. Because yeah, by by design, there isn't anyone that has your columns, and I don't know the only you know you have, like.
Francesco:yeah, you'll you'll be like, alone in your topic. And you, basically, I guess you just receive the columns directly from the block proposer every time, but like in a realistic setting that doesn't really make sense for you to just like exist independently without anyone that you connect with in the topics that you care about.
Mario Vega:Alright just for some out a little bit. It seems like we don't have the capability right now to go up to
Mario Vega:30 plus blobs for Fussaka
Mario Vega:So we don't need this feature right now. But we do need to ensure right now that we are gonna perform this testing again the moment that we want to.
Mario Vega:Increase the Vpo limit right.
Barnabas:This has nothing to do with that.
Barnabas:as far as I understand. Like, it doesn't matter if you have 4 blobs, or 30 blobs, or 300 blobs. If you if your full node cannot reconstruct and sync up using columns
Barnabas:with the user co-coding, then that's yeah. That's that's a totally different topic. That's a perfect topic. Now.
Mario Vega:Alright. Then do we have any other opinions on other sales about this matter?
Mario Vega:How do they fill chicken, Fussaka without this without this salt?
Mario Vega:Yeah. Agnesh.
Agnish Ghosh:So if a full node chooses to reconstruct because it doesn't find the accurate peers.
Agnish Ghosh:I don't know. How is that a problem? Because if it catches up to head, then it just again progresses normally right from like by getting columns from either
Agnish Ghosh:the blob pool or the proposal. No, I mean, I mean, it's a it's a it's a problem when it's sinking, of course, but reconstruction has several sort of guarantees to end.
Agnish Ghosh:Let's say the da of a slot within a given timeframe.
Agnish Ghosh:So I would. I would obviously prefer reconstructing than actually actively going and trying to find peers, because I think the latter would take me more time.
Agnish Ghosh:Yeah, I agree with Manu.
Agnish Ghosh:Actually.
Mario Vega:Alright. It feels like the sentiment right now is that we do need this feature. Is that is my reading correct?
Agnish Ghosh:Yes, yes.
Mario Vega:Okay?
Mario Vega:So yeah, I would. Like the Cls to speak up and and say, how much time do they need to to get this implemented.
Mario Vega:It seems like it's a blocker issue for Fuseka.
Mario Vega:Any other Cls that have not expressed anything today about this issue.
Mario Vega:No, we don't, Jacob.
pawan:We don't have this feature right now, but I think, like we have the mechanism to implement this.
pawan:so I'll have to look into. I'll have to look into it. Maybe I'll take a look this week, and
pawan:by next week I'll have something ready, or like at least know how long it'll take.
Mario Vega:Thank you anybody else.
Mario Vega:Alright! I feel like this is pressing enough that we have to
Mario Vega:Talk about it again on Thursday.
Mario Vega:Please to the rest of the Cl teams. Please look into how much time it would take to implement this, because it seems like it's a blocker issue, and we need to talk about this on on Thursday, so we can. We can touch upon the topic yet. There!
Mario Vega:Again, and see the raise the status. Does that sound? Okay for everyone.
Jae Hee Lee:Yeah. And if you guys need testing, we can just launch new Dev notes
Jae Hee Lee:at any time. Or just please let us know.
Mario Vega:Thank you. Excellent, all right.
Mario Vega:thank you. Is there anything else that we need to talk about? Right now? Any other.
Barnabas:Did you choose?
Mario Vega:Sorry. Backfield. Okay.
Barnabas:Backfill. Yeah, I'm curious. If any clients have implemented backfill for lapse.
Mario Vega:Manu says, prism is in progress, same from lighthouse.
Matthew Keil:Lowestar is actively working on it, but it's a pretty big body of work, like everybody else.
Barnabas:Do we want to actually wait for backfield implementations to be done before we launch them at 4, in case we do want to launch them at 4.
Mario Vega:It seems so at.
Barnabas:It would. It would be good if we had at least 2 clients that have done it, and then we could test those at least. And then we can actually go on the Shadow Fork and whatnot
Barnabas:could we get like an eta on when clients think they would have backfill implemented.
Mario Vega:Any of the clients. Is this
Mario Vega:bandwidth issue? Or is it like something intrinsically complex about the the feature that anyone wants to share.
Agnish Ghosh:I would like to go ahead. So the issue with backfill is
Agnish Ghosh:more of a CPU usage thing.
Agnish Ghosh:I would say, like it really depends on how quick do we want to get this backfill done
Agnish Ghosh:like, if we want to get it quick enough, then we are basically allocating
Agnish Ghosh:a significant amount of the CPU to do this thing rather than focus on.
Agnish Ghosh:Let's say, increasing.
Agnish Ghosh:yeah, it doesn't. Yeah. Okay, so rather than focus on let's say, higher block count or a bit more stressed environment when full is actively running.
Agnish Ghosh:So yeah, that is, that is one second is
Agnish Ghosh:while the backfilling do we really want to
Agnish Ghosh:advertise earliest available slot in a really very granular way? Or are people just doing it? Something like, let's say, when backfilling is in progress. I just keep it at local head slot, and then, when backfilling is done, I again
Agnish Ghosh:push it back to the slots behind. I
Agnish Ghosh:I was really curious about this. The current crashes. Some of the current crashes in members is
Agnish Ghosh:because we are trying to sandal, the earliest available slot in a very granular way. Like every slot we complete backfilling, we want to update the earliest available slot
Agnish Ghosh:that way. It's just more helpful for the network. I know there's no incentive. But
Agnish Ghosh:yeah, so I was really curious about that as well.
Mario Vega:Yes, pawan.
pawan:Yeah for us, like, we fetch the earliest available slot from our database. So every time backfill updates the earliest available slot in a backfill. Basically what we advertise also updates simultaneously. So at least for us, like we, we basically had to change a dB a little bit to add that particular earliest available slot point.
pawan:So right now, it isn't a problem for us to keep updating it after every backfill batch is processed.
Mario Vega:Thank you. We have less than a minute left. Ainish. Do you want the response or.
Agnish Ghosh:Yeah, just a question for pawan. What is an average backfill bat size? I'm just curious.
pawan:It's 1 epoch for us.
Agnish Ghosh:Okay. Okay.
Mario Vega:All right. We are out of time. I think we're going to close it now and thanks everyone for joining, please, on Acdc. This Thursday, we need updates on perfect beer desk syncing. So please, teams focus on that. And thank you. Everyone for joining. And yeah, have a good week.
pawan:Call queue.
Mario Vega:Thank you. Everyone.
Agnish Ghosh:Thank you.
Agnish Ghosh:Bye, bye.
Justin Traglia:Bye, bye, everyone.
Fredrik:Well.
Chat Logs
00:09:45
Barnabas:reth builder flags: https://github.com/ethpandaops/fusaka-devnets/blob/master/ansible/inventories/devnet-3/host_vars/mev-relay-1.yaml#L178-L188
00:10:09
James He:Have people already updated for remote signer needs? ( web3signer) opened https://github.com/ethereum/remote-signing-api/issues/20
I think we just need to handle fulu block as a version but reached out to uzi from web3signer
00:11:30
Barnabas:Replying to "Have people already ..."
we haven’t verified web3signer or other external signer
00:15:07
Bharath:Can you point to any repo related to this @marek
00:15:22
Barnabas:is it fulu ready?
00:15:27
marek:Replying to "Can you point to any..."
https://collective.flashbots.net/t/announcing-multi-client-support-for-rbuilder/4895
00:15:42
Agnish Ghosh:Replying to "Have people already ..."
We have it going on in a PR.
00:16:46
marek:Replying to "is it fulu ready?"
More testing would be appreciated, a devnet sounds perfect for this
00:16:57
nixo:important question: which striped animal is fusaka’s emoji? 🦓 or 🐅 ?
00:17:21
Barnabas:Replying to "important question: ..."
🦓
00:17:23
Ben Adams:Feedback was that 16M was fine? https://ethereum-magicians.org/t/eip-7987-transaction-gas-limit-cap-at-2-24/24746/22
00:17:32
Mario Vega:https://ethereum-magicians.org/t/eip-7987-transaction-gas-limit-cap-at-2-24/24746/21
00:17:44
Toni Wahrstätter:Yes, we keep it as-is
00:18:17
Agnish Ghosh:Replying to "important question: ..."
🐅 😛
00:20:37
James He:Replying to "Have people already ..."
We have one too just for the block wasn’t sure if something else was neeeded
00:20:45
Ben Adams:60M update before fusaka? 🤔
00:21:54
Parithosh Jayanthi:I can’t speak, but the geth team also said it would take some time to upstream such changes if they even decide to go down this route. So it won’t be something that would help us out asap.
00:22:14
Louis:Related issue: https://github.com/ethereum/execution-spec-tests/issues/1976
00:24:18
Parithosh Jayanthi:Depends on the timeline, but yes the next step would likely be 60M
00:24:59
Agnish Ghosh:Is it a good idea to get a builder api release out?
00:25:06
Agnish Ghosh:/a good time?
00:25:32
Louis:Related PR: https://github.com/ethereum/execution-spec-tests/pull/1983 for gas limit configuration for release
00:27:51
Louis:It seems to be safe for current gas cost
00:28:36
Justin Traglia:https://github.com/ethereum/consensus-specs/pull/4476
00:30:10
Barnabas:**shouldn’t** famous last words 😂
00:30:45
pawan:Will review it soon. Looks good on the first glance
00:32:05
Jae Hee Lee:https://testinprod.notion.site/Sunnyside-Devnet-Updates-08-04-2458fc57f546808ab2c9e34480e0b7a9?source=copy_link
00:32:10
Barnabas:https://github.com/ethereum/consensus-specs/pull/4477
00:32:12
Jae Hee Lee:Can we discuss Sunnyside Labs Report?
00:33:04
Parithosh Jayanthi:It was also sort of tied into if we would increase tx limit per tx
00:33:14
stokes:i think we leave alone for now
00:33:14
Parithosh Jayanthi:If we needed a new devnet there, then we wanted to do both together
00:33:29
pawan:Against the change as well
00:34:10
Toni Wahrstätter:It's not a bad idea but too late in the process now and not urgent enough
00:34:30
Francesco:Given that there won’t otherwise be a new devnet, I would rather leave it as is
00:34:38
Barnabas:potuz/tersec/nflaig?
00:35:30
nflaig:slightly for the change but I can see why we don't wanna do it this late in the process
00:35:32
Parithosh Jayanthi:If anything I’d say we do it in glamsterdam
00:35:44
Raúl Kripalani:BPOs were designed to not include any code changes
00:38:02
Barnabas:node operators can always yeet their db
00:38:08
Barnabas:so its not really an argument
00:38:16
Parithosh Jayanthi:It can be like history, tell them to resync if they need the benefit immediately
00:38:27
Barnabas:its also a very small percentage of users that would be affected.
00:38:40
Parithosh Jayanthi:I think we still wanted devnet-4 right? Just to see a perfect transition without any bugs
00:38:47
Parithosh Jayanthi:Just no spec changes in the devnet
00:39:02
Barnabas:Replying to "I think we still wan..."
I think we can discuss that later
00:39:08
Jae Hee Lee:https://testinprod.notion.site/Sunnyside-Devnet-Updates-08-04-2458fc57f546808ab2c9e34480e0b7a9?source=copy_link
00:39:15
Matthew Keil:Replying to "I think we still wan..."
I agree with Pari
00:39:46
Barnabas:We are looking to do a shadowfork sometime next week. Ideally we should have stablish branches from clients.
00:41:04
Agnish Ghosh:Replying to "We are looking to do..."
Yep hopefully should be bug-free by then.
00:41:20
Francesco:(Specifically because it’s just not enough for the CL propagation path to work for supernodes, we know it needs more bandwidth than 100/50, even in theory)
00:44:55
Francesco:Do we need this to work? I think it would also be ok to just say that if no one can serve you your columns, you can’t sync
00:45:33
Barnabas:Replying to "Do we need this to w..."
so basically we accept that nobody can reconstruct?
00:45:40
Barnabas:Replying to "Do we need this to w..."
so whats the point of having 128 columns at all?
00:45:44
Manu:Replying to "Do we need this to w..."
I will be definitely more confident if this perfect peerdas sync works.
00:45:46
Barnabas:Replying to "Do we need this to w..."
we could just do 64 columns, and call it a day
00:45:53
Francesco:Replying to "Do we need this to w..."
There are no supernodes in this devnet
00:46:05
Francesco:Replying to "Do we need this to w..."
Supernodes would reconstruct
00:46:35
Barnabas:Replying to "Do we need this to w..."
we haven’t verified that either yet
00:46:54
Manu:Replying to "Do we need this to w..."
The main issue I see if we don’t do that is:
A full node will sync
A node with validators won’t sync
00:47:12
Parithosh Jayanthi:We also don’t have to ship fusaka releases with support for 72 out of the gate
00:47:22
Agnish Ghosh:exactly
00:47:26
Parithosh Jayanthi:We could do these things later and then decide the bpo to go that high
00:47:34
Agnish Ghosh:Replying to "We could do these th..."
I agree
00:47:39
Manu:Replying to "Do we need this to w..."
The question is not “is there enough super nodes in the network” but “is there enough super nodes in OUR peers"
00:48:51
Francesco:Well, you don’t need supernodes in your peers, you just need someone that has your columns
00:49:19
Manu:That’s true.
00:49:31
Francesco:Another way to say it would be: do you need to be able to sync if you can’t find any subnet peers? How would you even stay on the network without subnet peers?
00:49:35
Raúl Kripalani:Replying to "Do we need this to w..."
i think yes. it gives us better load distribution when resolving non-finality
00:50:04
Agnish Ghosh:Replying to "We could do these th..."
Right we scale upto 20 blobs, it’d be nice to scale upto 30+ blobs for Fusaka? 72 is insanely aggressive imo
00:51:21
Raúl Kripalani:isn't this a scheduling problem? i.e. if clients can do this reconstruction during idle time, it seems worth it
00:52:39
Manu:2 different issues
00:52:41
Francesco:Yeah it’s not related to blob throughput
00:52:51
Agnish Ghosh:Different issues
00:53:19
Raúl Kripalani:yeah this is an issue about resilience / functional guarantees of DAS
00:53:58
Francesco:To be clear, I am not saying this is useless (though I haven’t thought through exactly how much it helps in general), just wondering if we need all clients to implement it before Fusaka
00:54:00
Manu:I wish we do not need this feature.
But I prefer to be prepared in case we need it...
00:54:14
Minhyuk Kim:In larger networks like mainnet, is it highly(or enough) likely that a syncing full node will reliably hit a peer which custodies a column identical to itself? Given the larger number of available peers and supernodes?
00:54:20
Barnabas:I agree on Francesco that its not a must have during finality, but during non finality, we might have some non finality scenarios where we do need this feature
00:54:39
Agnish Ghosh:Replying to "I agree on Francesco..."
We definitely need this feature
00:54:47
Manu:Prysm: ETA end of week
00:55:08
Raúl Kripalani:i think this is part of the broader topic of reconstruction (and which scenarios trigger it and when)
00:55:14
Francesco:Fwiw it’s not obvious to me that this isn’t going to end up causing problems because all of a sudden everyone is requesting way more columns than they normally do
00:55:30
Agnish Ghosh:We do we use this feature at times, for short range syncing, and sometimes even while gossip
00:55:33
Francesco:Replying to "Fwiw it’s not obviou..."
e.g. network has issues, and then they are compounded because everyone is trying to get 64 columns to reconstruct
00:55:37
Barnabas:you would only request more columns if the requested columns are not available
00:55:50
Francesco:Replying to "you would only reque..."
Yeah, so likely when the network is already stressed
00:56:09
Raúl Kripalani:Replying to "Fwiw it’s not obviou..."
i'd rather hit those issues sooner to understand how clients are rate limiting column requests (and decide how to harden)
00:56:35
Minhyuk Kim:I’m personally curious about the implementation of validator custody backfill for fusaka. Given further devnet/testnet upcoming, will it be included in future devnet requirement?
00:56:47
Agnish Ghosh:Ideally even for gossip we gossip accept 65 columns, pass DA and reconstruct only when nimbus is the most idle through the slot, the last 3 seconds
00:56:58
pawan:Tbh, I’m not fully convinced that this is required. Will talk about it async a bit after exploring it a bit
00:57:05
Manu:Backfill: Prysm in progress
00:57:10
Manu:(soonTM)
00:57:11
pawan:Same for lighthouse
00:57:21
Agnish Ghosh:In progress
00:57:37
Agnish Ghosh:We can wait sure
00:58:52
pawan:I don’t think it needs to be quick
01:00:04
Barnabas:update it once an epoch should be fine?
01:01:14
Matthew Keil:Our design is to backfill and updateearliest available slot on a epoch by epoch basis