Parithosh Jayanthi:So we're gonna start with fusata devnet 3 today.
Transcript
Parithosh Jayanthi:So the devnet went live late last week. So we have the I'm gonna share the link for Dora in a second. The networks finalizing, and we've had 3 dpo since then.
Parithosh Jayanthi:So we've progressively just been increasing. So we had fullu as a fork, and then we had the Vpos on top. So we're currently at 18 blocks per block. However, Bpo 4 is scheduled to reduce the block count so that we can also test that workflow. It should go live tomorrow and additional information on network. We've added some rate limiters. So we have about 30% gigabit nodes. And these are all super nodes.
Parithosh Jayanthi:and the remainder of the nodes have 100 m bit down and 50 mbit up. So this is based on the eip that defines the recommended hardware spec for a testers
Parithosh Jayanthi:the distribution of the keys is also not uniform. However, what we've done is, we have one of every client there, and when we've done top ups to cause a difference in what state runs on each node, and this is also allowing us to check Cgc changes with top ups, I think. Barnabas had a few points to note there, but he's out of office today, so he'll definitely bring it up with the clients later.
Parithosh Jayanthi:Yeah, Amanda, do you wanna go.
Manu:Yes, I had a question about rate limiting
Manu:do all the node has the same right limiting, or is it depending of
Manu:client implementation or super node, or full node.
Parithosh Jayanthi:It depends on. If it's a super node or a full node, so if it's full node, it gets a hundred Mbps, and if it's a super node. It gets gigabit.
Manu:Okay. Thank you.
Parithosh Jayanthi:Yeah. And these numbers are from the recommended hardware eip. So if for whatever reason, we notice that this is insufficient, then we have to
Parithosh Jayanthi:raise a bag and figure out what to do with that.
Parithosh Jayanthi:But this is the recommended hardware we have for stakes right now.
Parithosh Jayanthi:Besides this, we see that Nimbus and Nimbus el seem to be having a couple of shoes. Do we have on the call to expand?
Parithosh Jayanthi:Okay, I don't see them immediately. But yeah, they essentially were having some issues with peering, and were unable to connect to the Bootnodes.
Parithosh Jayanthi:They're trying to debug it with Dcli. But I'm not sure what the current status is, but I guess we take that we take that async.
Parithosh Jayanthi:Besides the nimbus issue, most clients are performing
Parithosh Jayanthi:quite, quite well so far. We do have a higher than expected rate of orphan blocks. I know that Manu was starting to look into it. Do you want to give an update on that Manu.
Manu:Yeah. I notice there is that prism often one block
Manu:and when looking into the logs, I saw that
Manu:for the block of slot just before all the colons were received quite late, like
Manu:more than 6 or 7 seconds into the slot.
Manu:so that explain why prism didn't chose to
Manu:build its own block on the param block, but on the grandfather
Manu:and yes, I was wondering if it wasn't due to the weight limiting. Maybe
Manu:So I have to check more into details. Why these colons were received very late.
Parithosh Jayanthi:Yeah, I can also have a look at the metrics. As far as I know, I don't think we were
Parithosh Jayanthi:rate limited on all nodes, and especially with such a small network like roughly 100 nodes. Having 30 of them, Gigabit is
Parithosh Jayanthi:should be enough to propagate everything quickly.
Parithosh Jayanthi:yeah, just looking at a quick 3 h metric on network usage where, at around the 30 Mbps
Parithosh Jayanthi:on not even 30 megabits per second, actually even lower. But it's roughly 20 Mbps sent and received.
Parithosh Jayanthi:and there's no apparent saying so. It's very random as to which cl and which el it is. So we don't actually know what's going on the only change we have from earlier networks is that the map workflow is now present, so it could be that there's some interactions there that we don't really know what's going on. It could be that the map node is
Parithosh Jayanthi:releasing the blocks late or nodes are waiting longer for timeouts. Not fully sure we have to investigate all of this, I guess.
Parithosh Jayanthi:Any other immediate info you wanted to gather, Manu, or are we all good to? Just look at this, Async.
Manu:No, that's all good right now.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:I've also made sure that 4 keys now working it. Was missing some dependencies. So it's up and running. You should be able to see when something's wrong. I did notice there were at least one or 2 instances where Nimbus was trying to build on a very old parent, and Grandine also tried building on an older parent.
Parithosh Jayanthi:so I'm not sure why that necessarily happened. I don't easily see a trend.
Parithosh Jayanthi:a quick example, is slot C, 5, 7, 7, 6. Where Grandine builds on a parent that's about 5 slots earlier or 4 slots earlier, instead of an immediate parent.
Parithosh Jayanthi:and then gets offered.
Parithosh Jayanthi:But yeah, I'm sure that we find a lot more examples and figure this out.
Parithosh Jayanthi:Besides this, do we have Bharat on the call to give us an update on the map. Workflow.
Bharath:Yeah, I'm there. So in terms of the update, like, my workflows.
Bharath:yeah, so where we are currently is that we have like 6 72 validators registered out of the 8 18 validators on the network. And we have a block production. We just got a slot delivered. We just got a block delivered for the latest slot.
Bharath:So blocks are being delivered of.
Bharath:So I am currently on top of my mind, for the workflows are 2 things. One thing is, why are all the validators not registered? I think the the Devops team we have set up like we've connected all the beacon nodes to the relay. So that shouldn't be an issue. So I so based on my initial investigations, there's some issue with like
Bharath:Loadstar, like, for some reason the validated registrations. When when lodestar sends it like there's a issue with the timestamp in the validated registration. It's a little weird like I was talking to Nico about that about it, and it seems like everything is right in the lodestar code. I don't know what's happening. I'm just going to investigate that I'm checking. If there's any caching around the stack. I don't know but I don't think so. But I'll investigate that. So that's there. Second on top of my mind is
Bharath:for some reason. When block publishing for nimbus fails with the State root mismatch, and this happens for Nimbus, only I don't know where the issue is. Here. That's another thing I'm gonna investigate. But yeah, so these are the 2 issues that I'm looking at in terms of my workflows outside that like. We've been having like very decent like rate of block delivery like if I just page to the
Bharath:link to the whatever. Oh, yeah.
Parithosh Jayanthi:Yeah, Justin just shared it.
Bharath:Yeah. So based on that, we seem to have pretty decent like, block delivery, like like today. Morning, we had an issue. We had a bunch of issues even over the weekend, like when we hit Bpo. One block building failed because we're using the block parameters of Fusaka and not of the Bpo's
Bharath:in the builder. So we updated that. And today we had some rate limiting issue with block simulation to fix that. And I think
Bharath:we have regular blocks coming in. But these are the 2 issues that
Bharath:I'm I'm looking into like right now, without my workflows.
Parithosh Jayanthi:Awesome.
Parithosh Jayanthi:Thank you for the update. There's, I think, one more thing. The map workflow now unlock unlocks for us. They've added the ability to have a private member, a private blog mempo.
Parithosh Jayanthi:And the benefit is that if we submit blobs here and the block gets produced via the map workflow. We've essentially turned off the get blobs optimization. So we can actually network would perform if we have blobs that have never been propagated. Ahead of time. I think we should try this over the next day, or have we already been trying this.
Bharath:I'm actually not sure to be honest. I think one of us had configured it, but I I.
Parithosh Jayanthi:I can look into it. I didn't get an update from him as to what the status was there. I can look into it, and if we haven't been trying it already, then I'll make sure that we tried. Now.
Bharath:Sounds, cool.
Parithosh Jayanthi:Okay. Anything else related to Fusaka Demnet. 3.
Parithosh Jayanthi:Anything people wanted to discuss. I
Parithosh Jayanthi:either. There's mostly 2 or 3 active threads
Parithosh Jayanthi:on the interrupt channel. But I think we spoke about everything so far.
Parithosh Jayanthi:Okay. There was a bunch of discussion last week about eth config. I didn't fully follow the discussion. Could someone summarize this? We have any volunteers?
Parithosh Jayanthi:Yeah, Mario, do you wanna go.
Mario Vega:Yeah, I think this summary is that we have it. Underscore config checks now in East, but we do have to enable them, I think, assert would be a nice place.
Mario Vega:Currently, the clients. What they can do is just use East to test their testnet configuration. And yeah, testnet and mainnet configurations, but I think it would be nice for us to coordinate and add this to assert, or probably adding, it's just a matter of creating a file that East understands with the devnet configuration. And then what we can do is basically just start querying the client's life
Mario Vega:during the net during the devnet. So we can like just just consistently see if there's anything breaking in need config or in the configuration itself.
Mario Vega:Yeah, that's that's basically my understanding. So far, I don't know if there's anything else.
Parithosh Jayanthi:Okay, thank you. Yeah. I,
Parithosh Jayanthi:at least on a quick reading. It looked like there was something about serializing. Rlp.
Parithosh Jayanthi:or is that a completely unrelated discussion.
Parithosh Jayanthi:Yeah, Marius.
Marius van der Wijden:Not really to that point. But I don't think matters here, but one of his
Marius van der Wijden:problems with this was the the hashing that it introduces. And
Marius van der Wijden:the problem is that we are hashing it in
Marius van der Wijden:we are hashing the Json Rpc
Marius van der Wijden:response. And that means everyone has to have the same Json, Rpc, response. That means we have to like strip all of the all of the wide space and stuff. And
Marius van der Wijden:I feel like this makes it extremely brittle. So I don't have
Marius van der Wijden:a hundred percent context on this.
Marius van der Wijden:So it's very unfortunate that Matt is not here. But
Marius van der Wijden:I think we should find another way of verifying this, not
Marius van der Wijden:like, not this weird way of hashing the Json.
Danno Ferrin:Do you have any proposals that aren't? Rlp.
Marius van der Wijden:So this is this. Okay, this is related to the Rp, and then I think we should just hash the Rp.
Danno Ferrin:Rlp is a format that we're supposed to not be using anymore. Why should we introduce another use of the format? Should we do an SSD. Encode against that.
Marius van der Wijden:Everything is better than the Json.
Marius van der Wijden:I would prefer the Rrp.
Marius van der Wijden:But if other clients feel strongly that Sse is the way to go.
Marius van der Wijden:I don't think anyone wants to prove anything against this Rp. And against anything against this hash, and that's why I think Rrp.
Marius van der Wijden:As the preferable option. If we were to prove stuff against it, then, I think Ss.
Marius van der Wijden:Might be better. But since we want to extend this configuration
Marius van der Wijden:in the future potentially a lot. I think Rfu would be the the better format.
Parithosh Jayanthi:Yeah. Roman.
Roman:That's not exactly a fair assessment that we're not supposed to be using Rlp, because one of the upcoming headliner proposals for the next hard work is block access list, and it will be in Rlp. So since Rlp. Is used everywhere, I agree, it's strictly better than the Json that is currently specified in the.
Danno Ferrin:When are we gonna do the Sfd shifts?
Danno Ferrin:I mean, we keep pushing it off. Are we gonna live with rop necessity forever?
Roman:Well currently on El. We have only Rlp, well, we we don't have Scc anywhere, so it would make sense to to change the config once we do that, and when we do that I'm not sure that this is the topic for this call to discuss.
Danno Ferrin:Cool. I would feel better just dropping the hash in general and doing adjacent dip before moving it to Rlp. To be honest.
Roman:That also sounds amicable.
Roman:I agree it might be a little bit more difficult for for the testing tools
Roman:to to sort everything and like to see that there were no extra keys. But but yeah.
Roman:everything sounds better than sorry in Json, as it is right now, I agree.
Danno Ferrin:If the testing tools can alphabetize the
Danno Ferrin:the fields underneath each each Javascript object that the diffing should be easy.
Mario Vega:Yeah, we we can even just I mean, we get the, we get the response and we don't do the hash ourselves. So I think we can do the hashing ourselves. If that's a a really big problem for the clients.
Mario Vega:So yeah, basically, Json, we do the hashing. We compare. It's not a problem.
Danno Ferrin:Okay, I'll drop the bits of the hash from the spec.
Parithosh Jayanthi:Awesome. Thank you. And Daniel will send an updated Pr.
Danno Ferrin:For the eip. And what else do we want for the Pr. Just eip, and let the teams do the rest.
Parithosh Jayanthi:I guess Eip and I'd assume it's in the execution. Api, as well.
Danno Ferrin:Yeah, those 2.
Parithosh Jayanthi:Yeah, okay, I think that should be it. Yeah.
Parithosh Jayanthi:awesome. Thank you. That works well. And we can still ping like client, and see if he's okay with the outcome. But
Parithosh Jayanthi:some sort of good outcome.
Danno Ferrin:There's also, while we're changing things. There is request to normalize the keys and values in the addresses.
Danno Ferrin:I don't want to lead with the addresses, but I'd be willing to lead with like the names of the precompile, so it'd be precompiled. Name Colon address. They wouldn't be statically sorted, but they would show quite nicely on the desk.
Danno Ferrin:So if the right time to do that change, that was another event concerns.
Parithosh Jayanthi:10 itself. That sounds like a sane change to me. Do we have any opposition.
Mario Vega:Could you repeat the change? I didn't quite get it. What's what's the change?
Danno Ferrin:So so in the precompiled contracts and system contracts. I have it ordered in what's being configured in the value in the configuration. That means we have 2 different keys and precompiles. It's the address of the contract, and then the value is the name of the contract and system contract. It's the name system contract you're looking for in the address it's at. I would propose for pre compiles. We flip it. It's always the name of the precompile, and always the address. The consequences. They're not going to sort in order and adding, won't always be at the end.
Danno Ferrin:But if we're diffing Javascript, it's not going to matter. It's going to be obvious that you're missing one when a diff happens
Danno Ferrin:so that would make the keys consistent across the precompiles in the system contracts.
Mario Vega:Yeah, I think either way is fine. We're still gonna sort them internally, just to to make a comparison. So it doesn't matter.
Danno Ferrin:Okay, I'll put that in my Pr. If anyone objects we can undo it.
Parithosh Jayanthi:Perfect. Thank you.
Parithosh Jayanthi:Do we have any other eips or Prs? We need to look into cause as of the last time. I checked the notes. Document the spec sheet for for 7 3 I think we had. Everything merged in.
Parithosh Jayanthi:The only outliers were 2 metric Prs. And I'm assuming we're still looking into those.
Parithosh Jayanthi:Are there any other eips we should be paying attention to?
Parithosh Jayanthi:Okay, that's good news time.
Parithosh Jayanthi:I guess the next question is, what's the timeline we want steps of testing. We have a shadow fork planned as well as a really large network.
Parithosh Jayanthi:I wager that we should at least wait until the nimbus issues are figured out as well as the map. Workflow issues are figured out, and the often rate are there any other points that client teams want to wait for before we start the next phase of testing?
Parithosh Jayanthi:And do we have an estimate we wanna come up with
Parithosh Jayanthi:this time next week or earlier.
Parithosh Jayanthi:Do we have any strong opinions?
Parithosh Jayanthi:Okay, then I guess we'll make a post on the interrupt channel. As to what the plan should be, but for now it would be great if clients focus on the orphaned issue as well as any in any client. Specific bugs that we point out.
Parithosh Jayanthi:or you can already find.
Parithosh Jayanthi:Okay, and then we move as soon as we can as a follow.
Parithosh Jayanthi:So
Parithosh Jayanthi:the next topic on the agenda is a gas limit. So we have Mainnet at 45 million for about a about a week. Now, is there anything particular that client teams have noticed on Mainnet. Any analyses, any
Parithosh Jayanthi:bugs. I issued notice in the last week.
Parithosh Jayanthi:Okay, I think we're investigating cl validator stability. Still. But we'll we'll probably have an analysis out soon about that.
Alexey:Sure if it's worth mentioning. We have some peer issues on my net. I mean
Alexey:Those are my notes become less visible
Alexey:in a peer set may not for some reason, and we are trying to figure out
Alexey:maybe it is connected to an update of some certain execution. Client? Maybe not.
Alexey:So we are investigating. If you, if some other clients see this similar picture and came to some conclusions.
Alexey:ping guests, too, please. Oh, and Sarah
Alexey:And say, it's just.
Marius van der Wijden:Yeah, we are also looking into this.
Marius van der Wijden:We have a theory that it might be because of Geth, because we introduced a feature where we
Marius van der Wijden:are asking the connecting node for
Marius van der Wijden:to do the note Key Exchange
Marius van der Wijden:or the United Exchange. I don't. I don't remember what what we do. But yeah, that
Marius van der Wijden:improves peering on on some level, but it
Marius van der Wijden:maybe if this, if this exchange fails, then it's also not good for
Marius van der Wijden:others. I think Jabba made this change
Marius van der Wijden:so maybe he can. He can speak to that a bit better than I can.
Parithosh Jayanthi:Do you wanna unmute or.
Csaba:I'm on mute. Yeah. Do you hear me?
Csaba:Yeah.
Parithosh Jayanthi:We can hear you.
Csaba:Okay, yeah. So the change was on this before, I haven't seen the issue yet. The change was on this before.
Csaba:When we are getting the Dnrs
Csaba:before just blindly trying to connect to to whatever node
Csaba:is behind that IP address physically that we have first.st
Csaba:We are getting the actually another with the folk Id. And and we are filtering, and we are only connecting
Csaba:to those notes which I was connecting.
Csaba:So that's only an outgoing. And that's what happens in this week. 5 already. It's just that this week 4 that we changed so that it's it's only trying to connect to nodes which which makes sense to connect.
Csaba:So I don't see how that can create issues. But if there is an issue on some pointers, I can try to look into it and and see if it's related to that.
Csaba:Yeah, that that was the.
Csaba:I suppose my perspective on that change, the other changes that we introduced are are more just.
Csaba:what was it? Yeah. So in in this before and and in in this, before we just changed. How we are
Csaba:looking up emails from the from the 3, 4, this 3, 5, so that we always have enough notes in the in the queue for for dialing out.
Csaba:But did you?
Csaba:Yep.
Parithosh Jayanthi:Has this been? Live?
Parithosh Jayanthi:It's not hard.
Csaba:Yeah, it has been live for. No, not not not that much, not not one month. I think I think a bit less.
Csaba:But I have to understand what's the issue that that you're seeing
Csaba:and and the timeline, and then we can
Csaba:try to see if it's edited.
Parithosh Jayanthi:Got it. Thank you.
Csaba:Yep.
Ben Adams:Oh, never mind!
Parithosh Jayanthi:Yeah. Sorry. Yeah. Ben, go ahead.
Ben Adams:I was just thinking, I mean, cause it also depends on
Ben Adams:when people upgrade their nodes so smooth.
Parithosh Jayanthi:That's fair. I was.
Parithosh Jayanthi:Yeah. I was also just wondering, because never mind, did have a release. About a week ago I wanted to mainly know if it was some to the Never mind release, or
Parithosh Jayanthi:if it's something that's been sitting on Mainnet for a month.
Ben Adams:It was around the time of our release, I think. Rep also did a release.
Ben Adams:But is happening with prior prior versions. Also.
Csaba:Can you send the link issue description?
Csaba:Well, something related that you see.
Ben Adams:Sure. I mean, it's sort of coming from Crawler, so we'll just hit
Ben Adams:I'll I'll send it to you. Then.
Csaba:Kim.
Parithosh Jayanthi:Okay? Yeah. If any client teams also notice any business with their notes, then please reach out and we can see how widespread it is, and see if there's a pattern.
Ben Adams:Yeah, I mean, we. We've not had any complaints from any users.
Ben Adams:Yeah, it's quite. It's just weird.
Parithosh Jayanthi:Yeah, that makes sense. Yeah, I was also just checking my note. And it seems to have, yes, so
Parithosh Jayanthi:okay, cool anything else on the peering topic.
Parithosh Jayanthi:Okay, is Carlos on the call. He wanted to give us an update on blueprint.
CPerezz:Yep, yeah, go ahead
CPerezz:so we finally arrived to the 2 X State landmark. Now, what I wanted to ask
CPerezz:especially to never mind breath and ergon, or see if they could. Take a look to this document that I've sent the link which contains a bunch of questions that would be really useful for us to finalize all of the test cases that we want to test in blood
CPerezz:so far. This
CPerezz:which I'm sending here. This second document is all of the scenarios that we have come up with. But hopefully, after reviewing it with all of the teams. This will grow and get much more precise and and correct. Probably.
CPerezz:Aside from that. There's an issue with breath in implemented, which is actually not proposing. If I recall correctly, which haven't been, which hasn't been tracked yet.
CPerezz:and another one with gift. Although this has been tracked already, and I think Gary just sent a link for it so hopefully it will all be resolved soon. So yeah, please, if we can have a review on these documents, and we can try to meet by this week, it would be it would be extremely useful coming from the teams.
CPerezz:Yeah, what is.
Marius van der Wijden:Yeah. So the the issue that we're tracking there is not really the the issue that actually happened.
Marius van der Wijden:Basically, this is just a symptom. The real issue that happened on Bloatnet was that somehow get would ignore the some some flags for the tracing, and so it would trace with the stack and the state enabled so whenever, like, we made
Marius van der Wijden:something like a like, we change something in the State, it would dump out all of the State.
Marius van der Wijden:which is because of Bloatnet extremely heavy.
Marius van der Wijden:we just updated to current master. And it seems like current master has fixed this problem. So that was because performance, the performance branch had like some weird old image.
Marius van der Wijden:Yeah, so, and because this used so much memory, the node would keep crashing every now and then whenever we had this like really big traces.
Marius van der Wijden:And because of that, I think we had around 55 crashes. And at some point the database was just fried. And this this issue that we're tracking right now is is basically
Marius van der Wijden:the database being fried. But it's not the root cause. The root cause was just the node running out of memory because of tracing.
Marius van der Wijden:And we kind of fixed the issue
Marius van der Wijden:already. Basic. Well, it wasn't really an issue was just an outdated image.
Marius van der Wijden:But we are also reworking our tracing to use less memory. So even if you enable these bloating techniques and enable state tracing, then it will not use that much memory.
Marius van der Wijden:Yeah.
CPerezz:So just just to be clear. Then this was sent by Gary saying that there's
CPerezz:in parallel maximum payload that you can actually push into the database so
CPerezz:like, how is that with running out of memory.
Marius van der Wijden:Better.
CPerezz:I don't know if swap, but.
Marius van der Wijden:That is, that is a different issue. That also happens when the database is fried. When we don't have the log indexer anymore, we will run the log indexer, and the log indexing
Marius van der Wijden:creates might create a batch that is more than 4 GB, and then pebble refuses to to write that batch.
Marius van der Wijden:but that is also something that we already had fixed on master. So
Marius van der Wijden:the the only thing that is not.
Marius van der Wijden:or that is questionable right now is like, if there's a way of fixing this this fried database, basically.
Marius van der Wijden:But it's not really an issue in Geth, it's more of an issue in in pebble itself.
Parithosh Jayanthi:Yeah. So for right now I've started re-syncing the note with the fried database, so we should have get back. Attend soonish, and we should be able to make a snapshot with get as well.
Parithosh Jayanthi:But yeah, we still have to, I guess. Look into the tracing issue. And there's already organizations.
CPerezz:So aside from that, there's 1 last thing which is the last link the last hacking, the link I sent which contains the scenarios we have so far, I was wondering whether this is something we can deduplicate and basically agree on with the yields team
CPerezz:mainly because they they have this issue which is related to Ck vms, but is the one I was referred to for state growth. Related tests
CPerezz:I can signal the redirection issue here. They started here.
CPerezz:So yeah, so the idea is that we can basically use all of these tests as state growth and state road computation once for both just testing and also benchmarking. So I think Mario is here. So it would be nice to see whether you are interested in working on those or not.
CPerezz:and especially because we're interested in running those when compaction or or other sub processes in notes are triggered, and I'm not sure to which extent this is functionality that yields should have, or we just need to build something outside and and monitor ourselves.
Mario Vega:Yeah, it's pretty interesting. I think the main problem with the tests that we produce is that we cannot do like a
Mario Vega:super huge state because of various reasons.
Mario Vega:But I would need to take a look into this this list. To be honest, I'm not. I'm not very familiar with it. But yeah, in general, just like the
Mario Vega:the things that we can add like right now, maybe you can expand this in the future is like computationally intensive
Mario Vega:benchmarking tests
Mario Vega:for state growth. We still like trying to decide how we can implement those if we can implement them.
Mario Vega:I don't know if these are all state growth and state specific tests? Or is there anything like computationally? Because if there's any computationally intensive tests that you have in this list, those like would be like the low hanging fruits, and we can just port them right away to to east.
CPerezz:So there, there will be computational, intensive, but just for state road calculation, but not, I mean, you will not be doing a lot of computation within the Edm itself. It's more related work and state state computation.
Mario Vega:I see, I see. Do you? Do you need to have like the state has to be huge in the in in this case, or can we start with a with a with a small state.
CPerezz:Ideally, we should test it with broad net state, which is 2 x mainnet at this point, and in the future we should do it with even bigger state sizes. So that's the ideal scenario.
Mario Vega:I see, I see. Thank you. Yeah.
CPerezz:It's just that yields was going to be integrated within spammer by Philip. So it just seemed like all the pieces came together. But yeah, maybe.
Parithosh Jayanthi:It is, it should still be coming together. I don't think comments on the call, he texted, saying he won't be able to make it. But what they've what the Netherland team has done. They is they've used a specific type of file system where you can copy very quickly from snapshots, so he was able to run a test and revert the test back to the old snapshot within a matter of seconds. So they wanted to use
Parithosh Jayanthi:that along with their gas benchmarking tool and the yields integration such that it doesn't matter if yields can import the State because they just have to do it once, and then you do these file system tricks to do the remainder of the work.
Parithosh Jayanthi:He's not here on the call, so I'd say we take that discussion, Async, but they have been putting some time into thinking about.
CPerezz:Okay, that makes sense. So I'll follow up asynchronously. That should be all for me.
Parithosh Jayanthi:Perfect. Thank you.
Parithosh Jayanthi:Yeah. Anything else from the guesstimate testing discussions.
Parithosh Jayanthi:Mario, do you wanna give an update on
Parithosh Jayanthi:east integration into the gas limit testing tool.
Mario Vega:Are you referring to? Clc, I this? So, yeah.
Parithosh Jayanthi:I meant the other tool, the Nethermind tool integrating to use the East test.
Mario Vega:Oh, yeah, okay, yeah. Basically, I think the update there would be that we have a new release.
Mario Vega:we have a new benchmark release. Let me share the the link. And basically what this thing does is
Mario Vega:here's link.
Mario Vega:Yeah. Basically, this thing contains all of the benchmark tests that we have produced in East. And then from here we have this, what is called a consolidated
Mario Vega:genesis file.
Mario Vega:which means that never mind can now use those genesis files to start their testing. I think they're still working on the integration. But basically, the starting point would be the benchmark releases from now on, we're going to try to make it like as seamless as possible for them to generate these Genesis files where they can just run every single benchmark that we have.
Mario Vega:But yeah, it's it's still a work in progress. It should be. It's almost. It's almost done. I think
Mario Vega:the main blocker right now is that we have to update some of the benchmark tests because we just updated the format to make them like all seamlessly work. But yeah.
Parithosh Jayanthi:So.
Parithosh Jayanthi:Yeah, I guess the next topic on the list is the Clc benchmarks. Do either you or Lewis want to talk about it?
Mario Vega:And Lewis is the person. Yeah.
Louis:Oh, okay, I will send the slides in the chat
Louis:and let me share my screen now.
Louis:Okay.
Louis:Oh, can you? Can you see my screen.
Parithosh Jayanthi:Yes, we can.
Louis:Okay?
Louis:Okay, today, I want to present the benchmark result for calculating 0 of code and
Louis:the p. 25256. Verify recompile introducing Fussaka.
Louis:and we run benchmark from East
Louis:Is included in this folder, and
Louis:this is the steps away from benchmark. 1st we fill the test cases.
Louis:And we want a hype server with different client configurations and run as an engine, and to observe the execution time for the payload.
Louis:And if we want to check more details about each test cases.
Louis:I have prepared a pull request for list so you could check detail here and for
Louis:the visualization results. There are some notes for it.
Louis:First, st we're not comparing with client performance, which with each other, we're comparing the counting, leading 0 up code
Louis:to other.
Louis:Of course, with similar gas cost.
Louis:And currently, we do not have client warmups in the East benchmark infrastructure. So there is no client warm up in neither, my and baseu. So in the result, it will take much longer time for these 2 clients.
Louis:but we believe with more help, it will be much faster.
Louis:And for the benchmark we we do. Take the Vip 7, 8, 25 into account.
Louis:So there is transaction gas limit cap in the benchmark test result.
Louis:And in the benchmark test we have 2 cases. The 1st one is for 7 to
Louis:1 million gas, which is twice amount of the default gas limit now in East
Louis:and the other one is the 100 million.
Louis:and for for list test cases
Louis:is slightly like it. It is for worse block benchmark, so
Louis:we are not like running to upcoast
Louis:for 1,000 times and compare the time spent for the benchmark test
Louis:we in in our case, we try to fill a block
Louis:as much up code as possible.
Louis:as much certain, of course, as possible, and we use this to benchmark the
Louis:the operation so it might affect how you interpret the result.
Louis:And this is the visualization graph we have now, for the red one.
Louis:It takes 3 gas costs, and this is the operation. It is listed here and full of blue
Louis:boot. Route one! It takes 5 gas costs.
Louis:and for cld, we have 2 cases, one for
Louis:executing, counting, leading 0 with different input and the other one is executing, counting, leading 0 with same. Input, it is labeled in green.
Louis:so you could see the result here and.
Marius van der Wijden:Alright!
Louis:This is the.
Marius van der Wijden:This is the time. Oh, yeah.
Marius van der Wijden:Do you use Clz with 3 or with 5?
Marius van der Wijden:Guess. Oh.
Louis:We're we're using with 5 now.
Louis:So it is the updated version
Louis:in for the Y-axis. It is the time spent to run this benchmark and is measuring second.
Louis:And this is the result for 100 million guest limit.
Louis:And this 2 graph is for calculating 0.
Louis:And this one is for the frequent pile, and this is the comparison, and there is a no.
Louis:the comparison in the time is spent but in person percentage. And this is a formula.
Louis:how we derive these values.
Louis:Forget that that's it from my side. Do you have any question about these results?
Parithosh Jayanthi:So one of the takeaways would be that with a gas limiter with a Z price of 5, we're able to do both 72 as well as 100 million gas. Right?
Parithosh Jayanthi:Guess limits.
Parithosh Jayanthi:because the execution time of all the clients are below 4 seconds.
Louis:Yeah, you know, below sports seconds.
Ben Adams:And this is without warming up for basically. And Netherlands.
Ben Adams:Yes, yeah.
Louis:It is without warming up.
Ben Adams:So they should be. They should be better on after some time.
Ben Adams:It looks like we're good.
Louis:Yes, we have opened an issue for adding warming up in the East so.
Louis:but we will do it later.
Parithosh Jayanthi:Perfect. Thank you. Does anyone have any other follow up questions? Any other benchmarks we should be collecting for Clc, or are we okay with status quo for now.
Parithosh Jayanthi:Okay, that's it for
Parithosh Jayanthi:for status quo. Thank you so much for this. And addressing Roman's point of dropping to 3 I think right now we should freeze and focus on shipping and stick to 5, and we know that we have a bit more slack for reducing the price at Amsterdam. So repricing. I know that there's a bigger discussion around repricing at Amsterdam, anyway.
Parithosh Jayanthi:Okay, any other comments on Cfz.
Parithosh Jayanthi:Perfect. Thank you for presenting and collecting the data. Lewis.
Louis:Thank you.
Parithosh Jayanthi:Yeah. Mario.
Mario Vega:Just one comment on the on the methodology. I think for this we use eels, and even though, like we are filling blocks, it's it's a little bit of a bottleneck because of the it's a little bit slower than clients, of course, but I think this is doable for this and for future hardworks. So basically, the methodology should be that we implement 1st in yields.
Mario Vega:and then we can just fill the benchmarks and just try them out with the with the clients. So now we have, like the the approach, on how to do it, on, on the, on decent feature forks.
Mario Vega:Yep.
Parithosh Jayanthi:Perfect. Thank you.
Parithosh Jayanthi:cool. The next item on the agenda is the Sunnyside Labs team. They've been working on a few things and they had an open issue
Parithosh Jayanthi:talking to the tech crew team where they found something, I think. Do you maybe want to give a present? Briefly, give us an update.
Minhyuk Kim:Yeah, we're preparing the reports right now. But just to give a update on our progress, we are running 2 type. We ran 2 types of devnets. So one was, we are running the previous test that we did in the last report, and we are rerunning the test with regular transactions as well on top of the pop transactions so that we can monitor how the
Minhyuk Kim:larger block sizes can affect the data column sidecar like propagation across the network.
Minhyuk Kim:And we are also running the other network, which is composed of 16 nodes. So with each
Minhyuk Kim:note studying a data column. So
Minhyuk Kim:that is a 1 28 in total. So we have one client concerning one calling, each like
Minhyuk Kim:one column started by one client each, so that we know
Minhyuk Kim:we we maybe we could find any bugs or
Minhyuk Kim:problems with that kind of setup.
Minhyuk Kim:And we also started running the test for validator custom backfill and in the Fussaka devnet 3 spec. Page. I noticed that Teku was the only client which said, it supports backfill right now. So we ran the test for Teku. But there were some problems. So we are communicating with the Techo team, and if any other client are ready for the backfill implementation we. We'd be glad to help with testing that as well.
Minhyuk Kim:And for when we get done with the new report we'll share it in the discord.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:thank you very much. Yeah. If any clients also need some help saying testing, then please either reach out to the Sunnyside lab team or to us. And it 1st glance, is the perfect column definite working. Okay.
Minhyuk Kim:Yeah, it's working. And it was it handled like 40 50 blobs without any issues@firstst Yeah, so yeah.
Minhyuk Kim:nice, perfect. I know we've had some trouble with the with that conf specific configuration in the past. So that's good to hear.
Minhyuk Kim:Okay.
Parithosh Jayanthi:Nice.
Parithosh Jayanthi:Okay? Any other questions for Sunnyside labs?
Parithosh Jayanthi:Great? Any other open discussion points
Parithosh Jayanthi:or anything I've missed from the agenda today.
Parithosh Jayanthi:Okay, then please engage with the 2 open metrics. Prs as well as the hack. Md, that Carlos has shared. And
Parithosh Jayanthi:thank you guys for attending the call, have a nice week.
CPerezz:Cheers.
Marius van der Wijden:Thank you. Bye.
Parithosh Jayanthi:And.
Pooja Ranjan:Thank you.
Louis:Thank you.
Chat Logs
00:03:20
Parithosh Jayanthi:https://dora.fusaka-devnet-3.ethpandaops.io/
00:05:58
Manu:Asking because Prysm orphaned a block and noticed that all columns were received very late (>7s) into the slot
00:06:55
pawan:Are the orphaned blocks from a specific CL/EL ?
00:07:38
Manu:Replying to "Are the orphaned blo..."
no
00:07:45
Manu:Replying to "Are the orphaned blo..."
No clear pattern.
00:07:57
Manu:Replying to "Are the orphaned blo..."
No clear pattern for the parent block as well
00:09:01
Parithosh Jayanthi:35776
00:09:45
Justin Traglia:http://mev-relay-1.fusaka-devnet-3.ethpandaops.io:9060
00:14:04
Parithosh Jayanthi:@pk910 could you look into that with the testing team?
00:14:14
Marius van der Wijden:Is lightclient not here?
00:14:24
pk910:Replying to "@pk910 could you loo..."
yea :)
00:15:42
Roman:the EIP does specify that, but i agree that it’s kinda brittle
00:16:04
nixo:Replying to "Is lightclient not h..."
he’s ooo
00:16:31
Tim Beiko:Why are we not supposed to use RLP?
00:17:31
Marius van der Wijden:But again, I'm a bit low context on this
00:18:20
Mario Vega:The EIP does specify a json standardisation https://www.rfc-editor.org/rfc/rfc8785
00:19:20
Mario Vega:I liked the hash btw 🙁
00:21:13
Marius van der Wijden:I think either way is fine
00:21:25
Marius van der Wijden:but they should be the same
00:21:55
Parithosh Jayanthi:https://notes.ethereum.org/@ethpandaops/fusaka-devnet-3
00:22:59
stokes:As soon as we can
00:23:08
stokes:This week if we can get the other stuff sorted
00:23:11
Alexey:we will check secp256r1 real gas price I hope this week. Doubt we have time to update price in case of interesting findings though
00:25:56
Roman:what’s the change again?
00:27:18
Ben Adams:Kamil says "Gogogogogo 60M"
00:27:23
Roman:can u link the PR pls?
00:27:41
Marius van der Wijden:We have a PR here to add some more debug logs here https://github.com/ethereum/go-ethereum/pull/32287/files
00:28:37
Alexey:we started noticing it like 5 days ago
00:29:01
Louis:Replying to "we will check secp25..."
What’s your approach, are you creating some benchmarks for this?
00:29:05
Trent:Catching up, the nethermind and geth issues are related or not related to 45mm?
00:29:11
Parithosh Jayanthi:Replying to "Catching up, the net..."
unrelated
00:29:43
Marius van der Wijden:Replying to "can u link the PR ..."
https://github.com/ethereum/go-ethereum/pull/31944/files#diff-5ad3eda22c65151285dcc5c2b67964cf633636bf5cc80e86773da555c5150fa8R178
00:29:58
Raúl Kripalani:Maybe if you have peer IDs we can investigate if there’s a common pattern
00:31:01
CPerezz:https://hackmd.io/WUe1KgwSS3iT8xoIbDg3Ng
00:31:21
CPerezz:https://hackmd.io/9icZeLN7R0Sk5mIjKlZAHQ?both
00:32:00
CPerezz:https://github.com/ethereum/go-ethereum/issues/32290
00:36:26
CPerezz:https://github.com/ethereum/execution-spec-tests/issues/1926
00:36:27
Marius van der Wijden:Interesting tidbits, we allocated ~10TB because of tracing and < 1TB of RAM because of the other chain operations
00:36:46
CPerezz:https://github.com/ethereum/execution-spec-tests/issues/1923
00:39:25
Marius van der Wijden:It would be ideal to write the tests in a way so that they can be replayed on arbitrary networks
00:40:27
Marius van der Wijden:How much do these file system tricks impact performance?
00:41:08
Mario Vega:https://github.com/ethereum/execution-spec-tests/releases/tag/benchmark%40v0.0.3
00:41:15
Parithosh Jayanthi:Replying to "How much do these fi..."
Yet to check tbh, we’re learning about this fs ability as we go
00:41:23
Parithosh Jayanthi:Replying to "How much do these fi..."
But its sort of the only realistic way to do state tests at scale
00:41:36
Parithosh Jayanthi:Replying to "How much do these fi..."
Moving tbs of state around will kill SSDs otherwise
00:41:48
Parithosh Jayanthi:Replying to "How much do these fi..."
Eth price isn’t high enough for that
00:43:47
Ben Adams:Replying to "How much do these fi..."
Yet
00:44:42
Ben Adams:Replying to "How much do these fi..."
@Marius van der Wijden sounds like low to none
00:47:10
danceratopz:Louis' slides:
https://docs.google.com/presentation/d/1jjLM4cWQghMpiyPcQzZ_3di-Euz65idgZ45BUPndKBs/edit?slide=id.g373c3cf0dc8_0_69#slide=id.g373c3cf0dc8_0_69
00:47:35
Ben Adams:So looks like we are fine with CLZ
00:48:19
danceratopz:Replying to "So looks like we are..."
if anything a gas cost of 5 is slightly conserative
00:48:49
Roman:are we dropping to 3?)
00:49:00
Marius van der Wijden:Replying to "are we dropping to..."
I don't think we should
00:49:08
Ben Adams:Ok with status quo; but should be considered to be repriced in next fork
00:49:31
Ansgar Dietrichs:Replying to "Ok with status quo; ..."
I think next fork we’ll reprice most opcodes
00:49:31
Marius van der Wijden:Replying to "Ok with status quo..."
already on the list 🚀
00:50:08
Ben Adams:Replying to "Ok with status quo; ..."
Calibrated Opcode Costs?
00:50:47
Roman:@Louis can you remind pls how to interpret the y axis on your charts?
00:51:10
Louis:Replying to "@Louis can you remin..."
It is the time spent on the benchmark test! In second
00:51:49
Roman:Replying to "@Louis can you remin..."
is it in seconds?
00:53:18
danceratopz:Replying to "@Louis can you remin..."
Yes, seconds. It's the time to execute the payload - a block that has been filled (to the specified gas limit) with the opcode being benchmarked.