unnawut:Okay, cool. I think we can get started. So, yeah, welcome to PQ interop number 33. As I mentioned earlier, Will is traveling, so I'll be, taking care of this call today. So, yeah, so as usual, let's start with client updates. So that'll be, Zoom.
Transcript
Anshal:Hey, hi, Anshulia. So, I'll be updating, regarding the Zoom client updates, since Kajinder is also traveling. So on the Zoom side, like, we finally merged the DevNet3 matrix PR. Apart from that, we have also, updated the,
Anshal:payload and gossip signatures map on the store side of things. There has been a PR that, has been merged on the spec side as well. Similar changes have been ported to
Anshal:side as well. Apart from that, we were also facing some issues during the, long-running interrupts with Checkpoint Sync. That has been resolved, and apart from that, there has been some networking updates,
Anshal:on the same client side to make it, like, make, gossip propagation more, robust. So, that has been done.
Anshal:Yeah, that's pretty much it from our end.
unnawut:Cool. Thanks, Anshal. Next is, Reem sharing?
Shariq Naiyer:Sweet, sweet. Yeah, hopefully my laptop doesn't die in the middle of this update, but
Shariq Naiyer:So, on the DevNet 3 side, we've updated the metrics, to include the, attestation Subnet committee, metric that Katya was requesting, and we've been working on fixing a lot of our sync issues, so Derek has been, influential in fixing some of them. We were running into stalls, and we've been… we've been continuing DevNet, debugging what's happening over there.
Shariq Naiyer:on DevNet 3. On the DevNet forefront, we have… we are currently working through the key separation PR, the attestation and proposal key separation. I believe we would have all the changes from that PR merged in by today or the latest tomorrow. So if,
Shariq Naiyer:if the cryptography stuff from LeanSig is ready to go at that time, I haven't tested that stuff, I read that something was happening on that front. So if those things are ready to go, we should be able to interop on DevNet 4 by end of this week.
Shariq Naiyer:And on the spec side, we have been working on fixing, some of the test vectors on the spec side, as well as the ream side. So, O has been taking, the foot forward there. He's been, he's been making great first issues, good first issues for new contributors to take up. So, that's what we've been doing.
unnawut:Cool, thanks, Shariq. Next in the reverse alphabetical order is, CLEAN.
Ruslan Tushov:Hello. There is currently no updates for Kulin. We're starting the DevNet 4.
Ruslan Tushov:work, and… but we took some time to try and make automated, interop testing.
Ruslan Tushov:These are small tests, like start network, run for some slots. I sent the link in the tuning chat, and so it's…
Ruslan Tushov:allows to run test fast and in different combinations, for example, Zuum-Zuum, Zuum-Reem, and we'll add later option to run local builds.
Ruslan Tushov:Let's allow, fixing the tests.
Ruslan Tushov:Faster.
Ruslan Tushov:Thank you.
unnawut:Cool, thanks, Rasaline. Next is, Enlin.
unnawut:Okay, if no one from InVin, then there's, Grandine next. I think RTM said that Mercy will be taking care of the Grandein's update today, right?
unnawut:Right.
Mercy Boma Naps-Nkari:Yeah, good afternoon. Good afternoon. So, on Garden's side, we had some internal fixes this week, so…
Mercy Boma Naps-Nkari:We were able to patch the synthetic anchor block we had implemented, also removed a duplicate attestation data check that was rejecting valid blocks from pairs. So, locally, it's fine, and I'm trying to, like, see how that plays out and how it holds up on the full demands.
Mercy Boma Naps-Nkari:Thank you.
unnawut:Oh, thanks, Mercy. Next is, Gene. Gene, I think?
Shaaibu Suleiman | gean:Yeah, hello, hello.
Shaaibu Suleiman | gean:Yeah, hello now. Yeah, so, I guess from the last, update yesterday, we've completed, sorry, the last update last week, we've completed, DevNet 3, and, started participating, in the DevNet 3, multi-client, interop DevNet run.
Shaaibu Suleiman | gean:And, I guess this, this is the first time we are participating, in the multi-client, DevNetron, yeah. And, it's coming up, fine. currently, we are currently investigating, an issue, a stalling issue.
Shaaibu Suleiman | gean:That, Katya brought, our attention to. I think, 3 clients, so we're actually currently investigating, that, and probably other, other issues that might, probably arise, end-to-end, in the DEFNET3, multi-client, DEFNET run.
Shaaibu Suleiman | gean:And also, we've started taking a look at DevNet, 4 already, the specifications for DevNet 4, to see how we can, start also facilitating development, for DevNet 4 also. So yeah, basically, that's it from our end, Gene. Thank you.
unnawut:Sounds good, thanks. Next is, ETHLambda.
Pablo Deymo | Lambda:Yes.
Pablo Deymo | Lambda:We were working on some peers for metrics. We added a link at the Stage 1 Community subnet.
Pablo Deymo | Lambda:tracking the decision committee account per node. We also fixed the registration of the metrics to have zero values as defaults in the startup.
Pablo Deymo | Lambda:And we added some metrics for the size of the tables.
Pablo Deymo | Lambda:Rocks TV.
Pablo Deymo | Lambda:To improve the…
Pablo Deymo | Lambda:pruning strategy, because we have a big increase of our storage, so we are working on that.
Pablo Deymo | Lambda:And on DevNet for…
Pablo Deymo | Lambda:aspect, we continue working on this PR. We are reviewing, and we think we will merge this PR soon, maybe today or tomorrow. The separation,
Pablo Deymo | Lambda:Between the… of the two keys, the station and proposal.
Pablo Deymo | Lambda:And also, this week, we migrated
Pablo Deymo | Lambda:And the library we use for SSZ to our internal library, leave SSZ, so we already matched that.
Pablo Deymo | Lambda:And we contributed with us with that library as well. And we started doing a kind of POC of a protarray fork choice, like the idea from LMD Ghost, from the Beacon chain.
Pablo Deymo | Lambda:To this… to this chain. It's just a POC, but we… we want to improve that.
Pablo Deymo | Lambda:Thank you.
unnawut:Right, sounds good.
unnawut:I hope I covered all the clients. Are there any other clients that I missed out and would like to share updates?
Mihir Faujdar:Yep, hi everyone, I can share an update.
unnawut:Oh, sorry.
Mihir Faujdar:So yeah, basically, since the last call, we've just been working on general fixes, bug fixes based on the feedback logs collected from the long-running DevNet.
Mihir Faujdar:Currently working on making,
Mihir Faujdar:Lots of improvements for syncing. Basically, like, I plan to basically, have a local long-running DevNet, using stable client images, so I can basically use that as a way to test, checkpoint sync locally.
Mihir Faujdar:As well as early this week, we added support for, state printing, because we've been noticing that the memory usage was growing, and as well as have finished adding support for DevNet4PR449. I note that there are
Mihir Faujdar:two or three other PRs that we have to also look at, so we'll be looking at those this week.
Mihir Faujdar:And yep, for the current issue that we're still investigating the…
Mihir Faujdar:current checkpoint sync issue. We should be…
Mihir Faujdar:there should be a new image available, with the fixes by tomorrow. Yep, that's our website. Thanks.
unnawut:Oh, thanks, Mihir. Any other clients?
unnawut:Okay, I don't see any other kinds in the list, so, next topic would be on metrics and DevNet running.
unnawut:Any updates from… yeah, Katya?
Katya:Yeah, hello, Ron. So, currently, DevNet is running since yesterday, and it looks stable. Some of the clients are stalled. After the call, I will share the logs and restart the clients. If new clients don't have access to Grafana Dashboard, please DM me. I will provide you the access.
Katya:Yeah, I can see the new subnet metrics from the clients, thanks. Currently it's zero, but, yeah, as soon as… as it will progress, we already have a metric.
Katya:Yeah, that's it from my side. The next week, I'm also traveling, so we'll probably restart clients once a day or so, or Partha will do this. Yeah, that's it from my side, thank you.
unnawut:That's good, thanks. Actually, I wanna share a bit more about metrics as well, based on, the Grafana.
unnawut:So I think maybe…
unnawut:yeah, I hope… hope the sharing is okay. So I think you… you all are familiar of, like, this Grafana that we are using right now, thanks to… to Katya and, and Partha, who set this up and have been running all these dev nets. One thing I kind of, like, want to figure out is that after, like, we have achieved, like.
unnawut:100,000 slots, like, what are, like, the small details that we can find in there, hopefully, like, build up onto the next DevNet?
unnawut:So I created this, Observatory. I think I mentioned it before, but haven't really demoed it, but since we already have, like, really, like, you know, 5-6 days running DevNets, I thought it might be worth sharing right now. So I can share the link later, but it's basically Observatory.linroadmap.org, and basically what this does is it's taking the Prometheus metrics that are from
unnawut:Grafana here, and, put that into Jupyter notebooks, compile them, render them, and generate the, you know, the kind of analysis for one specific period for the DevNet.
unnawut:I broke down into 6 notebooks, you know, PQ signatures, aggregation, consensus, state transitions, networking, and resource utilization. I'm just gonna go through this quickly so you guys can kind of, like, see an overview.
unnawut:It basically would compile, for example, it would kind of, like, cut down the long length data into specific dev nets, with its starting time. For example, this one, was run… running for, like, 115 hours.
unnawut:You can see in here all the metrics broken down by each of the clients.
unnawut:you know, attestation, signing, verification, and so on. Some data is, like, in the tables as well. And this…
unnawut:It's available for aggregation, consensus, like, you know, what's the head slot, current slot, justified, finalized slot, and so on.
unnawut:So if you guys are interested, feel free to come to this link and then look through all. Currently, it's basically just only GitHub build and connecting to Prometheus, so likely it's gonna only have, like.
unnawut:data for the latest two weeks, so if you need more data back, then just, like, let me know, maybe I can set up some infrastructure. But for now, like, I try to keep the Unifar minimum, so we are only using Kitapa actions, so it only lasts for two weeks of data.
unnawut:But the… kind of, like, the thing that I want to bring to your attention is… so after all these, like, notebooks compiled, I specifically picked this, like, 115 hours
unnawut:DEFNET, because I guess, like, the data is, like, most stable. And just, like, go through details a bit. I want to share this here, because I don't… I think, you know, we don't… so that we don't have to, like, spend time on this, on the weekend, on the, the, the fork mode.
unnawut:day to do this, but basically, I think, like, the… in terms of, like, PQ signatures right now, the… the signing time and verification time looks perfectly fine. It's only taking 16 milliseconds to sign, and this is basically gathered from all the clients, and only, like, 32 milliseconds, at 99th percentile, so it's basically only using, like, maybe 6% to 3.2% off the interval, so off the 1 second interval.
unnawut:That… that someone needs to sign a block or a test station. So, I think that's good.
unnawut:aggregation data right now is a bit problematic. Some data are missing. I still need to go look into, like, each specific metrics that's missing.
unnawut:It could be from the client itself, but it could also be from the… how the Promethe is gathering data, or from… from my notebook script itself, so I need to dig down into that.
unnawut:But one thing that I want to kind of, like, bring, like, point out is, like, the number of signatures aggregated per slot, and this is, like, per slot, right, per 4-second slot. We are looking at about 1 to 5.3 signatures per slot.
unnawut:But I noticed that for this specific DevNet, we are actually running up to, like, maybe 13 validators.
unnawut:The median… actually, so this is a range, right? But if you look at, like, the graph, the median is actually around, like, two signatures per slot. So, I think something is missing around here, and I think this is, like, one specific point that… that's gonna kind of, like.
unnawut:reflect the… the performance of, like, or the result of the DevNet the most. So this is something, if anyone knows what's going on, or has any clues, like, feel free to kind of, like, share that in Telegram group. I would be, like, interested to… to dig down a bit more.
unnawut:The rest, I think the numbers are still… because, like, we have only, like, 13 validators, we are getting, like, really small number of, like, signatures per… per slot or per second here in the aggregation, so I think the only way out here is just to basically, like, like Ajinder said before, we should, like, expand our DevNet to… to, you know, like, tens of or hundreds of validators, when possible.
unnawut:Some to-dos are actually that I'll conclude at the end.
unnawut:In terms of consensus, I think we are doing great. Like Katya shared before, like, we are able to finalize up to, like, 100,000 slots, so I think that's, like, really good.
unnawut:I noticed one thing, I'm not so sure if this is a data problem as well, but, the finalization delay, or I'm not so sure what's the right word for that, but basically this number of starts from here to finalize right now.
unnawut:The difference is around 20 up to 82 slots for most of the clients.
unnawut:Head to justify takes 11 to 27, and justify to finalize could be, like, 8 to 64 slots, which is very odd, because we are supposed to be doing 3 slots finality, but it's taking up to 82 slots to… to finalize, so something is going on there.
unnawut:Also, from the metrics we gathered so far, I noticed there's a lot of reorgs as well,
unnawut:in best case, no rearcs, but I think we are seeing up to, like, 0 to 12 rearcs happening, and it's happening constantly as well. So if you look at the graphs here, you can see that it's always, like, almost always non-zero, so that's also something worth looking into as well.
unnawut:State transition, nothing much. This is basically the time it takes to process a block. We are hovering around 125 to 250 milliseconds. Given interval is 1 second, I think we are still fine, but it's very lightweight, because we are doing very little to process a block.
unnawut:Networking-wise, I… I'm seeing some clients reporting one connected peer, even though it seems like it's gossiping to multiple peers, so maybe there's something wrong with the reporting there. But in terms of, like, networking,
unnawut:Receiving and transmitting.
unnawut:For this DevNet, it's a bit weird, where, like, the first half of the DevNet duration, like, it fluctuates quite a lot, but after, like, the first 48 hours, things kind of, like, stable out.
unnawut:But all in all, I think it's still kind of okay, for the fluctuated time. We are looking at about…
unnawut:1 to 5 megabytes per second of network writes, and half to 4 megabytes per second of read. And if you kind of, like, sum that up, that combines up to about 24 megabits per second, and the EIP7870 recommends 50 megabits per second, so I think that's still good for us.
unnawut:But again, like, we are only running 13 validators, so assuming that it goes up linearly, then…
unnawut:it looks like we can only double our validators to 26 validators before we saturate the actual, like, recommendation. But obviously, like, we are running on an infrastructure, so we can go much higher than that, but just kind of, like, heads up that if we are just only doing the way we are doing the networking right now, we are only supporting
unnawut:on 26 validators, I would say.
unnawut:Resource utilization, aggregator is doing 2 to 4, on load average. I think this is okay. Non-aggregators are using 0.2 up to 1 load average, so I think that's okay, too. Memory-wise.
unnawut:Okay, there's nothing outstanding.
unnawut:Yeah, so those are the resource utilization. So, oh, and one thing that I have to point out is, I think from the current infrastructure setup, the…
unnawut:the aggregator is fixed, right? I think from this, like, 120 hours defined, it's fixed to ETH, like, ETH lambda, validator 0, or ET lambda 0. So, that…
unnawut:had a performance toll on the… on the… that instance of ETHLambda, but I think another thing that we might want to try out as well for future DevNet is rotating aggregators, because right now we are only basically, like, you know, having ETHLambda client or implementation doing the aggregation job, and we are not actually testing out how other clients are… are doing with the aggregation, or… most of the clients are doing aggregation.
unnawut:But it's not the primary aggregator, so there might be, like, behavior differences there.
unnawut:So I guess, like, kind of, like, a quick…
unnawut:suggestions on what we can do further after, like, learning about this DevNet is obviously metrics reporting, there's some… some that needs to be fixed. There's the, like, low signature aggregation rate. I think that's odd, and I think that's probably kind of, like, skewing the DevNet results, so that needs to be investigated as well.
unnawut:the finalization gap, the frequent rearcs, and then I think after we have a stable DevNet 3, like Ajinder proposed before, we should increase validators, and yeah, so, I think we… before we're actually going to DevNet 4, if we are able to kind of, like, bump up the validator count for DevNet 3, I think that would be great.
unnawut:rotating aggregators, I think that's also something that we can explore. I'm not so sure, maybe DevNet 4, maybe after that, and then perhaps maybe have multiple aggregators?
unnawut:Within, you know, like, the same slot.
unnawut:That could also be possible as well. And then, yeah, I guess one of the kind of, like, bumped-up priority is once we get to, say, 20, 30 validators, we probably have to start adopting new networking design before we can actually scale the defects up again.
unnawut:So, yeah, that's my quick update on the metrics. Again, thanks to Katya and Patha for setting up all the infrastructure. This one was, you know, basically all the information that's coming from Prometheus and Grafana that those two set up. So, and yeah, it's been fun to actually, like.
unnawut:all this data.
unnawut:So that's my update on the metrics,
unnawut:Next section, unless there's any, maybe, quick questions or quick thoughts?
unnawut:Yep, gotcha.
Katya:So, regarding the network, metrics, so currently there are a lot of unstable clients, and they, request a lot, so, please keep it in mind. So, I think we need a syncing metric, so to see which client is currently syncing.
Katya:And
Katya:If you need any other metrics, maybe, maybe we missed something important, please let me know, or open an issue.
Katya:And regarding aggregators, I was trying to run with a clean aggregator yesterday, but just stable… the DevNet was so unstable, so I had to just stabilize it to turn back. I'm…
Katya:I… I don't say that that was the reason, but, just to…
Katya:went back to… to the stable DevNet, so we can experiment later as well with this.
Katya:Thank you.
unnawut:Sounds good, Katya.
unnawut:If that's all, then, next would be, specs and research. Toma, or anyone?
Thomas Coratger:Before we go.
Shariq Naiyer:the.
unnawut:Oh.
Shariq Naiyer:Mr.
Shariq Naiyer:I just want to say, oh, that website looks amazing. Thank you, thank you for all the work you're doing on that Observatory website.
unnawut:Well, thanks. Oh, I have to give credit to, Raul as well, so I saw Raul, from networking team.
unnawut:have this for, for East PandaOps, I think. So I just basically, like, forked that up and turned it into a PQ DevNet thing. The hardest thing about that is, like, they use a different data pipeline. I don't remember. I think they used ClickHouse or something, so I had to…
unnawut:Change that, that, in just… Yeah, to… to, Prometheus.
unnawut:But, you know, AI agents have a lot.
unnawut:Hey, thanks, Chaduke. Yeah, sorry, Thomas. Thomas?
Thomas Coratger:Yeah, so, let me decouple into…
Thomas Coratger:Two things. First on the research, update.
Thomas Coratger:We have continued to… improve, like, CONKY3 and LeanVM.
Thomas Coratger:We had also, like, a discussion with, Blonky 3 recursion project guys, especially Robin, who is maintaining the project. This is, like, a similar approach to LeanVM.
Thomas Coratger:with a bit of a different, like, vision and things, because this is not really a VM, but this is more like a circuit builder approach. And so, we are very interested with this approach, because this
Thomas Coratger:allows us to make a fair comparison between LeanVM and this approach to see what are the advantage, drawbacks, and stuff to aggregate signatures. We are continuously improving,
Thomas Coratger:Plonky3. We have a very big PR with, Onur, who is working with us. We submitted this big PR yesterday. Let me send it here, to integrate mostly, everything from here, inside Plonky 3 there.
Thomas Coratger:So that soon we'll have, we're available directly in Plonky3. We also had a couple of PRs.
Thomas Coratger:To make, Poseidon faster, and the arithmization of Poseidon faster, especially this one from Emil this morning.
Thomas Coratger:So all of these PRs are, like, improvements and making things faster. As discussed also during the last ZK proof call with Justine, we have, like, Monolith, which is this very modern, hash function.
Thomas Coratger:That we are also trying to make fast. Yeah, this is a modern alternative to Poseidon that is not very battle-tested, but something modern that we want also to check. So, we are doing all of this.
Thomas Coratger:In the meantime, I think that Emil, maybe we'll talk about this later, but Emil did a DevNet 4 branch for a lean VM that supports all of the DevNet 4 features, including, like, so recursive aggregation.
Thomas Coratger:Poseidon1 support instead of Poseidon2, and the new ZK-friendly encoding.
Thomas Coratger:So that is on the research front. We are also working on, on its, research post about, the
Thomas Coratger:post-quantum pub key registry, about sharing some ideas, some thoughts about how we could proceed, how we could handle that as a preparation event to the full post-quantum transition, let us say.
Thomas Coratger:So we have a post in preparation. If you are interested, feel free to DM, and I can share for… we have a group with this, and I can share for feedback.
Thomas Coratger:On the spec side of things, a couple of nice news. We have, like, DevNet 4 PRs ongoing. A big one has been merged last week, and we have these big other ones with POSEIDON on one switch, and,
Thomas Coratger:the new ZK-friendly encoding that will be merged, along with everything related to the new LeanVM update.
Thomas Coratger:So that will complete a lot of stuff that should be in DevNet 4. Then I noticed that you guys opened a lot of issues and PRs, especially good first issues, so we have seen a new, wave of collab… or, like, let's say, new collaborators, entering the game.
Thomas Coratger:submitting some small PRs, this is super nice.
Thomas Coratger:Because this makes, more A's into the code.
Thomas Coratger:With more testing, small, good first issues that are easy to tackle and easy to review, so that is also very good news to split the work that way.
Thomas Coratger:I love this, and I think that we should continue. I didn't expect that so many people could be interested by Python issues there, but I think that we reach a point where we have a bit of stability inside the spec, so we are able to do these kind of things and delegate a bit of work to external contributors, and that is very nice.
Thomas Coratger:So yeah, that is, my update, we should continue like this.
Thomas Coratger:And, yeah.
Thomas Coratger:Emil, maybe, if you have anything to… I'm not sure you are here, but if you have anything to add about LeanVM, or…
Thomas Coratger:Nope, so… but in any case, I guess I will see you most of… most of you in Cannes, next week.
Thomas Coratger:And especially next weekend, with the fourth mod events.
Thomas Coratger:Where we have a bunch of talk and nice presentations to discuss all of this.
unnawut:Sounds good. Thanks, Thomas.
unnawut:Artha mentioned, quick question, is there an impact to LeanSpec as a result of supply chain attack on few Python packages from yesterday?
unnawut:I haven't looked into those, not sure, Toma, do you know?
Thomas Coratger:No, same, I haven't looked, but… I don't…
Thomas Coratger:I would say there is not that much impact. Why? Because we have, like, at most 3 or 3 dependencies, or stuff like this, so I hope that these are not the wrong one, but…
Thomas Coratger:Yeah, I have not checked deeply into this.
unnawut:Yep, so, that's research updates. Any other, questions or updates, maybe on DevNet… 4.NET 5…
Parthasarathy Ramanujam:I mean, before we move on to DevNet 4, if I may, I believe the next PQ interop call, wouldn't go through because of, HCC, and we have effectively two weeks, for us to run, probably, a long-running DevNet.
Parthasarathy Ramanujam:I was wondering, I mean, Gajinder had one PR that he's raised on Zoom, I think it's merged, correct me if I'm wrong, Anshal, which introduces a new mesh,
Parthasarathy Ramanujam:topology for, multiple, I mean, validators that we are setting up. I would probably, raise a PR for the lean spec as well to introduce that if the clients can probably implement that, we could have probably multiple subnet,
Parthasarathy Ramanujam:testing for the long-running DevNet, which would probably highlight additional issues that you just raised in your spec. So just wanted to run it past all clients if they are okay to implement this before we move on to DevNet 4 testing.
Parthasarathy Ramanujam:It seems to be a minor change, but yeah, if everyone is agreeable to that, we can go ahead with that.
Parthasarathy Ramanujam:Alright, I'll then probably…
Parthasarathy Ramanujam:Yes, I'll do that, Mihir. I'll be raising the PR on lean spec later today or tomorrow, and then, yeah, we can use that to test it.
Parthasarathy Ramanujam:just to say that in the current setup for multi-subnet, you have one aggregator per subnet, and the idea is for the subnet to listen to attestations from other subnets as well, so that all of that could be aggregated. It's a minor change that can be introduced either in the config or the client, where you… an aggregator, you could specify the list of
Parthasarathy Ramanujam:subnets they wish to listen to attestations. So, it's theoretically a small change, but I'll let you know as soon as I have the PR ready.
unnawut:Oh, sounds good. So just to repeat again, you'd like this PR in and implemented in the clients as well, within,
unnawut:Did you say, like, next week or something?
Parthasarathy Ramanujam:whenever it's possible, because I'm not sure if all clients would be ready for DevNet for Interop, at least for the next two weeks. So I was wondering if we would… if the clients manage to implement, this feature, we could probably have a long-running DevNet with multiple subnets as well.
unnawut:And just to clarify again, so, like, based on DevNet 3, and add this to networks… Correct. Correct, yeah.
unnawut:Cool, so, so that, part that will raise the PR, will ask all the clients to, to chime in, and we can figure out, after that PR whether we can make it into DevNet3, and by what date.
unnawut:Cool, yes, Toma, you, you…
Thomas Coratger:Yeah, yeah. Also, like, about the spec, as I mentioned yesterday in the chat, I think that the big, the big point right now is the new, lean VM update and the recursive aggregation.
Thomas Coratger:So that I am trying to avoid touching
Thomas Coratger:that much things besides this, so adding tests with the small PRs for adding testing coverage is okay and pretty encouraging, but I have tried to avoid any major change until we reach the point where we have everything merged.
Thomas Coratger:For example, in the last PR of Anshell, I think, we had, like, this discussion around, like, block production. I think that in the build block function.
Thomas Coratger:We have, like, four, nested levels of logic, so that is super complex. Yesterday, I tried to go over it. This is a discussion there.
Thomas Coratger:this is super complex and pretty difficult to follow. You have
Thomas Coratger:Two wiles with two fours, so that is very complex.
Thomas Coratger:I will touch it, and I will try to simplify it, but I am waiting for… for the recursive aggregation to land.
Thomas Coratger:before I touch anything, because I don't want to make the work more complex than it is.
Thomas Coratger:But this is DevNetly something that I want to check.
Thomas Coratger:Because I think that this will cause some problems in the future, because this is too complex to have this level of nested logic into the spec, so I should touch it, but not now.
Anshal:Well, on that side…
Anshal:Yeah, I want to add something. So, on that side, Thomas, so, as I mentioned, we can completely skip the greedy aggregation part. I think that is a sort of an optimization that can be limited to client teams, or, whichever client team
Anshal:want to, implement it, so I think that can be removed. Apart from that, like, once we have, like, I don't know, like, what's the…
Anshal:exact benchmarks there, but, if I have, like, multiple childproof, what I tried doing there was greedily select the,
Anshal:proof based on the number of new candidates that I'm able to find, for every other proof attached to a particular attestation data. If I simply,
Anshal:if I simply loop over through the proofs and only, look for, unique valid… look for proofs that add a unique, validator bit, then I think I can completely skip the greedy part. But,
Anshal:again, here we have the trade-off if we… if that is, like, I can, supply all those child proofs directly to LeanVAM for recursive aggregation, but then the number of children proofs will be higher, as compared to when I do it, greedily.
Anshal:So, if we, like, skip the greedy part, then, we'll have, like.
Anshal:preset of nested pools, which was, like, the case before introducing this change. So this is something that, we can look into. Like, we'll have to, I think, benchmark it and see that, what
Anshal:what needs to be done. Apart from that, like, I have another idea that, we had discussed in the Zoom call as well last week. So, right now, all these aggregator… all these proposers who are not aggregators are simply relying on the aggregators and not aggregating anything of their own.
Anshal:But with recursive aggregation, can we at least,
Anshal:Aggregate the aggregates that we, that we get while, producing the block, so that for a particular attestation data, there's simply a single,
Anshal:Single, payload attached to a particular attestration data.
Anshal:So, that is one thing that I was thinking of adding in the, in the next PR. I think I can wait as well for that PR to, for that feature to roll out. I can simply add the…
Anshal:recursive aggregation right now, and in a follow-up year at that thing, but that is something that we can do, and I think, like, based on the observability metrics that O shared, so we can, basically determine how much load on the number of posts that we are using
Anshal:For the non-aggregators, so if it is, like, within the range of the EIP, then we can, go ahead with that.
Thomas Coratger:And yeah, I think that we already mentioned that in the chat, but I think that's…
Thomas Coratger:to update the bindings and the new API of LeanVM, you will have to work on the PR that I made about the cryptography, because right now, in the lean spec, you have POSEIDONDEN2 implemented, and so you will have a mismatch between…
Thomas Coratger:what is in LinVM and what is inside the specs, so you need to work on top. But feel free, like, to do anything you want, like, to duplicate the PR, to push on my PR, like, to open a new PR, like, anything you want is good for me.
Anshal:Yeah, yeah, I'll create a PR on top of your PR today.
unnawut:Okay, cool, sounds good. Any other updates or discussions?
Pablo Deymo | Lambda:I wanted to, to, to mention something.
unnawut:Yes, please, go ahead.
Pablo Deymo | Lambda:Yes. We, Lambda, or, at least,
Pablo Deymo | Lambda:Thomas and me, we are not going to… to Cannes.
Pablo Deymo | Lambda:other guys from London, yes, but not us. So we wanted to… we had the idea to host a kind of site event for the FCC, a remote site event.
Pablo Deymo | Lambda:Our idea is to talk about clients, and we would like to invite And you, and… Thanks.
Pablo Deymo | Lambda:builders, will be stuck, it will be a Twitter space.
Pablo Deymo | Lambda:So, we will… Put me in an 8.
Pablo Deymo | Lambda:And we can communicate via Telegram, but if you're interested, we will be happy to talk with you.
Pablo Deymo | Lambda:Thank you.
unnawut:That's good. Any other updates or announcements?
Mercy Boma Naps-Nkari:Sorry, I wanted to ask about this link that was shared on the chat.
Mercy Boma Naps-Nkari:the interrupt test.
unnawut:Yep, Mercy?
Mercy Boma Naps-Nkari:Yeah, I mean, there was a link on the chat about interrupters, so, like, I'm confused on what it is.
Mercy Boma Naps-Nkari:I shared a link on the chat.
Ruslan Tushov:I can explain. So, this repository implements simple tests, like, currently, Link Quick Start allows starting individual nodes, or the whole network.
Ruslan Tushov:it starts it manually, and usually you check the logs, metrics, Grafana, to see if the network behaves as expected.
Ruslan Tushov:this, in this repository, if you look into tests directory, there are production and finality and restart tests.
Ruslan Tushov:These are scripts which run some small network.
Ruslan Tushov:and then check the metrics manually. For example, if we have 3 slot finality, S…
Ruslan Tushov:was mentioned in the Observatory report.
Ruslan Tushov:We should see that a head metric is… and a finalized metric, are different by no more than 3 slots.
Ruslan Tushov:So we can find such assertions in the test, and run all the combinations, for example, two zim nodes, two RIM nodes, and see if there is some
Ruslan Tushov:Problems in some configurations.
Ruslan Tushov:And it does it automatically.
Ruslan Tushov:And also, this currently is a project in development.
Ruslan Tushov:If someone wants to try, please feel free to share your feedback.
Ruslan Tushov:Or request some features.
Ruslan Tushov:I tried to add, clients, but, there was…
Ruslan Tushov:some problems running its Lambda and Lighthouse. So, if someone could contact me.
Ruslan Tushov:Or we could discuss what are the arguments or docker image.
Ruslan Tushov:to run.
Ruslan Tushov:For the Thank you.
unnawut:Great, any other… Questions? Updates?
unnawut:Okay, then, some housekeeping items. Next week is HCC, so we won't have the next PQ interrupt. The next PQ interrupt will be on the Wednesday 8th of April, so 2 weeks from now.
unnawut:And then this weekend in Cannes, like, some of us have already mentioned, there's Beast Mode and Fort Mode events, run by Justine Drake, Ladislaus, and Will.
unnawut:Beast Mode is happening on Saturday in Cannes, and Fort Mode, which relates to us, leaving Census, is, on… on… on Sunday.
unnawut:Hope to see everyone there.
unnawut:Yep, and I guess that's… that's all the updates we have for this week. Thank you, everyone, for joining the call. See you in ETC. If not, then PQ interop in the next two weeks.
Katya:Goodbye.
Ladislaus:Very much, oh.
Mihir Faujdar:6 o'clock.
unnawut:Thank you.
Thomas Coratger:But…
Pablo Deymo | Lambda:Bye-bye. See you. Bye.
KaydenML:But…
alan xu:Ed? Hello, everyone.
Chat Logs
00:39:32
Parthasarathy Ramanujam:Rotation is already ready in lean-quick start, we can target that in the next devnet3 run
00:43:48
Thomas Coratger:https://github.com/Plonky3/Plonky3/pull/1477
00:44:04
Thomas Coratger:https://github.com/Plonky3/Plonky3/pull/1480
00:44:27
Thomas Coratger:https://github.com/Plonky3/Plonky3/pull/1478
00:47:48
Parthasarathy Ramanujam:Quick question, is there an impact to leanSpec as a result of the supply chain attack on few python packages from yesterday?
00:49:56
Mihir Faujdar:Hey @Parthasarathy Ramanujam , could you please share the PR for it?
00:52:53
Thomas Coratger:https://github.com/leanEthereum/leanSpec/pull/464#discussion_r2970421525
00:59:05
Mercy Boma Naps-Nkari:https://github.com/turuslan/lean-interop-test
00:59:15
alan xu:Where can I see the meeting summary?
01:01:28
Ladislaus:Antworten auf "Where can I see the ..."
on forkcast!
https://forkcast.org/calls
--> Breakouts --> PQ Interop
or also on GH ethereum/pm issues
01:01:50
Katya:Lighthouse doesn’t support devnet-3
01:01:55
Mercy Boma Naps-Nkari:Thank you
01:02:09
Mega | Lambda:@Ruslan Tushov Let’s talk in tg
01:04:58
alan xu:Replying to "Where can I see the ..."
Thanks!
Summary
17 highlights
· 3 action itemsExperimental
Summary
17 highlights · 3 action itemsExperimentaldevnet status
- Devnet-3 stable since March 24; 115+ hours runtime, 100K+ slots finalized00:28:15
- Multiple clients experiencing stalls and checkpoint sync issues requiring investigation00:34:04
- Median 2 signatures aggregated per slot; expected ~13 with current validator count00:35:41
- Finalization delay: 20-82 slots observed vs expected 3-slot finality00:37:39
- Frequent reorgs occurring (0-12 per period) requiring investigation00:38:07
devnet4 progress
research and spec
infrastructure and metrics
- Observatory tool launched: Jupyter notebooks analyze Prometheus metrics by devnet period00:29:31
- Network usage 24Mbps (13 validators); EIP-7870 recommends 50Mbps minimum00:38:30
- Multi-subnet mesh topology PR incoming; aggregators listen to multiple subnets00:49:02
- Automated interop testing framework available; runs client combinations programmatically01:00:08
Decisions
- Defer major spec refactoring until recursive aggregation PRs merged00:52:00