Thomas Coratger:So… Let's get started. Welcome to Post Quantum interrupt number 38.
Transcript
Thomas Coratger:So, first, so we have different topics on the agenda. We will discuss, I think, mainly a devnet 5 proposal, but first, let us start with client updates. So, Zeam first.
Gajinder Singh:Hey, doing well.
Gajinder Singh:Hello, everyone. So, yes, on Zoom front, as I mentioned last time, we have been working to optimize our client as an aggregator, and to basically maximally parallelize and optimize all the flows, so we did some work regarding that.
Gajinder Singh:And, we are… we added some metrics also to see our processing of, the interval, and on basis of that, we are doing further optimizations.
Gajinder Singh:And, basically resolving the paths where we don't want mutex to unnecessarily hold the critical paths, the hold paths. So we are evaluating them, and we are trying to
Gajinder Singh:get, to resolve the hard parts and make sure that we can be a very optimized aggregator. That is what has been our focus. Apart from that,
Gajinder Singh:devnet PipeFront, we have been
Gajinder Singh:looking into, the spec for aggregating multiple payloads into one single signature, which also, and, Anshul has been talking to Emil regarding this, so that, to get the API interface,
Gajinder Singh:cleaned out for, for basically de-aggregating slash splitting the payloads as well, so that we can then independently re-aggregate it on, front of…
Gajinder Singh:on front of Goldfish, running Goldfish as the consensus availability mechanism. We have been digging into
Gajinder Singh:how, basically, Goldfish works, and what are the issues.
Gajinder Singh:that it can have with, 3SF, so basically, we figured out that, finally, we realized that it is incompatible with 3SF as such, because 3SF also has some sort of
Gajinder Singh:availability running, and what we actually want is a finality
Gajinder Singh:gadget which runs independently from the availability protocol. So we are basically further digging into it and figuring out, what is the best mechanism that isn't…
Gajinder Singh:we need finality is… because as the nodes proceed, right, we want block 3 to be pruned out, and we want caches to be cleared out, otherwise we won't be able to sustain the nodes beyond a day, the way we earlier saw on devnet0, when we didn't have
Gajinder Singh:These sort of prunings.
Gajinder Singh:So, with regard to that, we need some strategy to figure out how to
Gajinder Singh:prune the nodes, or how to keep, basically, move… how to keep moving some sort of a stabilization… stabilization point further and further down the chain. And the other point that we also realized is that we want something that is also
Gajinder Singh:easily, codable into, into the straight transition function, and that safe stability should…
Gajinder Singh:should basically show across any of the forks downstream that, whatever that finalized last table point that we, that we calculate and figure out. So… so we have… so keeping all those things in mind, we have been
Gajinder Singh:digging into some research, and thanks, Jan, for it, and all the discussions on it, and I think we'll basically take that up
Gajinder Singh:In this call later on with Jan.
Thomas Coratger:Great, yeah, thanks a lot for the update. We'll discuss that in more detail just after the update of all of the team.
Thomas Coratger:rim.
Shariq Naiyer:Okay, so, we've been working on three, three things. Our primary goal is stability in the devnets, and then scaling the devnets.
Shariq Naiyer:So we've been working on 3 things, testing, planning, and executing. And I think most of these things are better shown in a demo, so I'm gonna let Derek take the part for, testing.
Shariq Naiyer:Go ahead, Derek.
Derek Sorken:Hello? I just want to double check my mic's working well here.
Shariq Naiyer:Yeah, it's working well.
Derek Sorken:Perfect. Okay, I'll go ahead and share my screen.
Derek Sorken:Okay…
unnawut:Your voice is a bit low, though, than usual.
Derek Sorken:If I adjust my mic is a little bit better?
unnawut:Yeah, much better.
Derek Sorken:Perfect, okay.
Derek Sorken:Are you able to see the Hive screen?
Shariq Naiyer:Yep.
Derek Sorken:Okay, perfect. So, one issue that we kinda…
Derek Sorken:Like, a non-issue we've noticed as we were running is that… the default…
Derek Sorken:HiveView is kind of messy as we kind of run through a lot of Hive stuff.
Derek Sorken:And it gets kind of more and more difficult to navigate through.
Derek Sorken:What exactly is going on?
Derek Sorken:And to get to the nearest stuff, it's always, you know, I have to scroll all the way down.
Derek Sorken:And even then, I'm just gonna move my…
Shariq Naiyer:filter, and… yeah.
Derek Sorken:right here.
Derek Sorken:it's a little bit difficult to say, okay, which client's doing what? So you say, okay, maybe we want to look at gossip, and we have to go and load gossip, and be like.
Derek Sorken:Okay, now we can see, okay, given client, okay, now it's Gean, or now it's Enlina, or whatnot.
Derek Sorken:And so, we came up with kind of a solution to this, and we've improved upon the Hive UI.
Derek Sorken:And I think that…
Derek Sorken:even then, like, we'll probably go and add this improvement to this main page, but for now, we have it hidden under Lean Latest.
Derek Sorken:And so it's a… maybe a much more digestible way to see what's going on with all the tests.
Derek Sorken:And so…
Derek Sorken:it's presented in a grid view for each client. You can see the client. For example, Client Interop is probably the
Derek Sorken:the only outlier in terms of how this works, but I guess just for, like, a regular suite for RPC Comp, we can see…
Derek Sorken:For a given client, we can see if it passes a given test in the rows.
Derek Sorken:And if it…
Derek Sorken:I suppose if it does pass, we can of course navigate to that exact test information, and click on logs as normal.
Derek Sorken:Or… If we want, we can see, of course.
Derek Sorken:In the case of, like, we have Qlean here, I think it's just the,
Derek Sorken:the API endpoint, but we can also see, okay, what's failing, and then we can go exactly why is it failing? It's just, they're returning Qli API instead of LeonRPC API, and if we wanted, we, of course, go back to the…
Derek Sorken:Logs.
Derek Sorken:We now have this for all of our sweets, and so it's maybe a lot more digestible to see.
Derek Sorken:What's going on, which… how many tests are failing in total, but then, of course, which clients are failing, and…
Derek Sorken:I think as we go along as well, in the, the main Hive page here, we'll probably display, like, a…
Derek Sorken:instead of all this, we'll have the Lean Latest page, and then if you scroll down, you'll be able to see this.
Derek Sorken:But I think it might be, as well, something to consider to add, like, a little scoring at the top, just to make it…
Derek Sorken:even more clear what's going on for each client. Northink as well.
Derek Sorken:For client interop, it makes things way easier to see, you know, who's working with who.
Derek Sorken:And so, I might add a little legend here, but, for example, we can see Ethlambda majority, and so two Ethlambda nodes, and say one key node works, and, like, one end lead node doesn't work, and if we want to go from the minority side instead, we can say, okay.
Derek Sorken:You know
Derek Sorken:when there's only one Zeam node, but then there's two Qlean nodes, things are failing, and of course, we can, again, navigate to see
Derek Sorken:You know, what the logs are, what the failure reason is.
Derek Sorken:I think maybe an oversight, we'll add clients against themselves, we can fill these empty gaps here, so instead of just blank for each lambda against itself, we'll put it there just to make it…
Derek Sorken:A little bit nicer looking as well, and…
Derek Sorken:To, of course, tests we can interrupt against ourselves.
Derek Sorken:But yeah, that's the, the Lean Latest page. We'll probably maybe re-navigate it here, but I think this should make…
Derek Sorken:things a lot easierly, easier to digest in terms of, you know, you can quickly check what's going on. And the way this works is, it'll just pull the most…
Derek Sorken:The most recent test run. So…
Derek Sorken:For, this is the most recent run of Client Interop.
Derek Sorken:In my time, 5.05 a.m.
Shariq Naiyer:Yep, I mean…
Derek Sorken:But…
Derek Sorken:this should be, like, serve as a status page to see how everything is running in Hive, and should be ways to check, okay, what's failing, and then you can easily go and navigate, of course, to why it's failing, and it should be a lot easier to clean everything up and get things working
Derek Sorken:Without that, maybe, extra navigational step.
Derek Sorken:But yeah, just a quick demo on that.
Derek Sorken:That's all for me, I think.
Shariq Naiyer:Sweet.
Thomas Coratger:And so, if I understand correctly, right now, in this page, you have, like, the classical Hive tests that are for consensus debugging, but not yet.
Thomas Coratger:the lean spec test vector, I guess, because I haven't seen any PR to link things together, so my assumption is that these tests are not in this suite, right?
Kolby ML:Yeah, still working on that.
Derek Sorken:process, yeah, he's… I think he's getting… getting there, but… This will then… as… once that's added, that'll again just be just an additional suite displayed here, and you can quickly navigate.
Derek Sorken:To, what's working for the lean spec test with each client.
Derek Sorken:What it shouldn't be.
Derek Sorken:Much more addition into this page.
Shariq Naiyer:So that, one, it allows us to test all of the clients, see all of the results of the testing, publicly, and then what it allows us to do is, you don't want to be running devnets with clients that you know won't interop, so you could run…
Shariq Naiyer:just because everyone has limited resources, most people do not have access to, like, 100 GB RAM servers. So you can run the clients that are actually interoping with each other, in your devnets. The other factor we're working on is planning out these devnets, what topologies we should have, what subnets we should have, if you want to scale up. I'm gonna let O speak to that.
unnawut:Okay, thanks, Shariq. Let me share my screen as well.
unnawut:Yeah, so there's actually a few things that I want to share, but just quickly, so, as Tomo mentioned, like, last week, this is the observatory site, so I just checked it again. It's still working, it's still running, for the past few weeks, so, feel free to come in here and check out the data. I did some edits here, because previously on the, like, list of previous, devnets on the left here.
unnawut:There's no clue on, like, how long it ran or how many clients, so I just added, like, a small visualizer here, so that, you know, if there's, like, something that's, like, not too significant, like, you know, 33 minutes definite, then you can discard that. And then if you're… you want to look into something that's, like, more long-running, then you can basically look into the orange bar here, so you can see this one is 24 hours.
unnawut:This one is 4.5 days, so the longer the bars is probably the more interesting to look into. But I haven't did any, like, further analysis in this, so I'll just, like, share this here, that it's available for anyone to see.
unnawut:The second thing that I want to share is, so Ream did a study session, about 2-3 weeks ago, so we made also, like, this kind of, like, static… the static JavaScript page. There's quite a bit of, like, interactiveness in there, where you, you know, you can play around with, like, read Solomon. I'm pretty proud of this one, where you can, you know, kind of adjust the tampering level here, you can adjust the number of samples you want to do, and you can kind of
unnawut:We wrote samples and see, like, how many times you actually caught the sheater, and how many times you will not catch it.
unnawut:And then there's, like, more, yeah, different…
unnawut:ways to play around with this, you know, with the sum check. You can also, like, tamper the output and see, you know, all the changes that happened, so you can, like, know how… how basically, like, WER works, and how WER, like, detects tampering, stuff like that. So that's the second thing.
unnawut:And then the last thing I want to share is this thing called LeanBench. So this… this is what I mentioned yesterday, that I want to kind of, like, use this to… to feed back into devnet 5 as well.
unnawut:in the team, like, room team, we want to figure out, like, how far we can scale up the validated count on the devnet, right? And I think to answer that, that…
unnawut:Basically, depending on… on three things.
unnawut:versus, like, the performance of LeanSig, lean multisig, of course, like, proving and verification. But more important in terms of DEFNET is, the performance of lean multisig on a specific target machine, right? We need to pick the target machine on… on what instance type, though, we want to run the DEFNET on.
unnawut:And lastly, the topology for… for… basically the topology for the aggregation of the signatures. But all of these are moving factors. The mother sick is still moving. We don't have, like, a strict, like.
unnawut:specification on, like, what instance type we should use. It's flexible, we can, like, do smaller instance with, like, more numbers, or maybe less number of instances, but bigger, bigger instance. It's also… I think we are not too sure whether we want to stick with, like, EIP7870, for… for the… the number of cores and so on as well. So, because, like, these are moving factors,
unnawut:I figured out, like, having some kind of, like, a harness where you can automate this, and it's basically, like, one single command to run, and then it will trigger,
unnawut:it uses the GCP CLI to basically spawn up the different instance types. You can configure that. It waits for all the…
unnawut:the test or all the benchmarks to run, and then collect all those results. It compiles them into a JSON, along with profiling the machine, so it takes all the CPU information, the RAM, the memory information there, and combine it into a JSON. And then, if you choose to, you can also, like, basically just, like, commit and push it onto the repo. So basically, like, all these runs are basically the runs that
unnawut:had, and they're all on GitHub repo. So it's pretty, like, pretty simple to maintain.
unnawut:Then there's some interesting stuff that I want to show. For example, like, I think this is probably pretty obvious, or I guess, like, all the… actually, researcher, like, Emil, you probably know their performance profile anyway, but I think, like, having this is kind of, like, easy for us to digest as well. So, in terms of, like, LeanSec, or even LeanMartistic XMSS, signing, I think that's pretty…
unnawut:like, pretty clear that as long as we are using something that's a bit more, like, modern CPUs with 4 cars, you are getting performance that are pretty similar, no matter, like, how many cars you have. The only one difference is N1 standard, but I think N1 is probably, like, I don't know, 6 or 8 years old CPU, while, like, the C4 are more, like, 2 years, 4-year CPU. So as long as we avoid that, then… then in terms of, like, an actual
unnawut:just a tester client, or closer client, I think we can make do with, like, C4 standout 4.
unnawut:Another interesting thing that I found out is that, yeah, so scaling in terms of aggregation is pretty linear,
unnawut:the more cores you have, then the faster it runs. So these are all, like, the raw numbers here. Again, like, the only one is the N1 start that stands out, but all the modern CPUs kind of, like, scale up linearly. There's some more, like, interesting bits down here. So those are raw numbers, but there are some… I derived some charts to show the relationship as well.
unnawut:So, for example, I showed in here the relationship between the leaf size and the fanning. So, basically, the number of, like, recursions, the number of aggregations, number of proofs you want to recurs into one proof. And this pretty much scales linearly, for the…
unnawut:Interestingly, for different leaf size, so if you aggregate more signatures, like, more raw signatures into one aggregation, and you want to do recursion on top of that.
unnawut:it doesn't matter whether you're aggregating 125 signatures or doing 500 signatures, the timing for that is pretty much the same. So, if you are doing just, like, 2 aggregation recursion, then it's taking 6,000 milliseconds. If the lead size is 250, you're also around, like, 6,000 milliseconds. Again, for 500, it's also, like, 6,000 milliseconds.
unnawut:Similarly for… for, like, doing, like, four aggregations, recursion, yeah, one…
unnawut:12 seconds, 12 seconds, 12 seconds, and so on. So, another way to visualize this is basically, for the same number of recursion
unnawut:number of, like, fan-in recursion you do, you are getting pretty static results with the number of signatures. So, we know that the biggest performance hit is the number of fan-ins. The more number of aggregations that we want to do recursion on.
unnawut:That's the more performance hit that we'll get, but the number of signatures, is… is okay.
unnawut:There's also, I also compare it in terms of scaling on, same instant types, but more, like, different numbers of cores. So you can see here for the same instant type, but for a different number of cores, for signature, signing, and verification, that's pretty much, like, kind of, like, flat, for most of the aggregation stuff. Interesting, I don't know, maybe…
unnawut:Tomorrow, you probably know why this happens, but the scaling happens linearly for 4 cores up to 16 cores. Like, you're basically halving the time that it takes to aggregate something, whether it's, like, a flat or a tree, like a recursion. But once it gets to 32 cores, I don't know if this is, like, something, like, specific to Lean MotoSig, or is it, like, something to do with, like, the cloud
unnawut:instances, but I did multiple runs on this, and it continues to happen, so I'm not so sure yet why, but once I get to 32 cores, the performance kind of, like, dropped, so you're not getting the linear, optimized, like, linear improvement in time, the more number of cores you go up. So another, like, kind of, like, thing we can set here is that for our definets, I think, like, maximal
unnawut:like, 16 cores or 32 cores is probably, like, the maximum number of cores that we want to test with, because, you know, if we go up to 64 cores, the performance gain is not worth it. So we probably want to stick to 16 or 32.
unnawut:And, yeah, basically, that. And then I also list out the results by machine type, so we can also, like, dig in here if you're interested in some of the instance type. You want to see whether, like, an actual operation is
unnawut:using or saturating the cores or not, then you can also come in here, to see at a specific, machine instance. You can see here, like, for… for,
unnawut:say, signing, yeah, so signing takes, like, about one core to do, while, like, verification takes, like, pretty much, like, 0% of the time, and so on.
unnawut:So this is the first part of LeanBench. Then there's the second part, it's called the trend. So basically, what I was kind of, like.
unnawut:here is when I did the first run of this thing about, like, 2 weeks ago, is that, Lee Model 36 is changing so fast, like, there's, like, so many comments each week, so… and I'm not so sure whether, like, there's any performance
unnawut:like, gain or impact, depending on the comments there. So, when, like, if…
unnawut:I run, or if anyone runs and commits the results here, then it can also plot a graph over the specific, like, commit combo.
unnawut:on the benchmark. So you can also see here that for, like, the aggregations, most of them are pretty much, like, constant over time. You can also see the dashed line here. I plotted the proof size as well in here, so you can see, kind of, the regression over time. XM is a sign, I think this is, like, one of the useful things that… that I already, like, gained from this, is because I think ML,
unnawut:or one of the contributors into Lean Autistic did an optimization where you no longer have to do, like, some kind of, like, grinding, to sign the XMSS, so avoiding that, it dropped from, you know, taking about, like, 200 milliseconds to sign. It dropped to, like, 5 milliseconds to sign. So, that's pretty cool.
unnawut:And then the last thing to kind of, like, inform into devnet 5 is… so I mentioned yesterday, there… I… I said that we should be able to push up to, even in DefNet 4, maybe a combination of two aggregation subnets with 250, validators each.
unnawut:And basically, that's because, like, from all the numbers that I showed so far, we could actually derive some models out of this. So I, white-coded this simulator up as well.
unnawut:So we can fill in, you know, the interval time. So we have 5 intervals right now. Each of the intervals is 800 milliseconds, so it's here. I made a really kind of, like, optimistic assumption. The propagation budget is 200 milliseconds. I think this is low, but as long as we are in, like, a closed, definite in a single data center, then maybe we can kind of, like, just get away with this.
unnawut:For now, but you can always, like, change the numbers, and then… And then the… oh, yeah.
unnawut:the… the… the simulation here will go up. I also… we have this kind of, like, internal target that we want to kind of target, like, 100, one… sorry, 10,000 validators, so that's why, like, all the numbers are targeting 10,000, and you can kind of, like, see here from the simulation that
unnawut:Actually, we might be able to fit in a…
unnawut:Like, a 4 to 6 second slot.
unnawut:But the interval might have to be, like, kind of, like, shrink or expand a bit, and the topology here is probably not the best that we want. So for example, there's no way that we could fit, like, say, 10,000 validators, in a, like, two-level or three-level topology. You need
unnawut:About, yeah, at least 4 levels, so, like.
unnawut:One of 200 aggregations, aggregate to, like, 5, and then to 5, and then to 2, and then arrive at one, root, proof.
unnawut:And there's some visualization here that, you can basically kind of play around with. But just to stick to devnet, devnet 4 and 5, where it's a reasonable number, I would say, I was saying probably, yeah, is it 500?
unnawut:Because basically, for… for 500… oh, actually, I think…
unnawut:We could actually even do 10,000. So for… for example, like, if we aim to have, like, 10,000 validators in the devnet, then we can actually make do with just, like, one single leaf aggregation and one recursive aggregation. So basically, the leaf aggregation would take about 500 milliseconds, 550 milliseconds to do for each of the 200 signatures.
unnawut:And then, for these 5,
unnawut:proofs to aggregate into one. It takes about 2.23, which still fit in the two or three intervals that we have within the sort before the
unnawut:It's like we…
Shariq Naiyer:For a second. Your screen is covered by a different screen.
unnawut:Oh, just now?
Shariq Naiyer:Yeah, your one browser is covered by a different browser, and I think you're browsing on the first browser.
unnawut:Can you see the, like, the…
Shariq Naiyer:I think that's what you're.
unnawut:Laura?
Shariq Naiyer:Yeah, yeah.
unnawut:Oh.
unnawut:Interesting. I haven't actually moved anything.
Shariq Naiyer:Okay, I don't.
unnawut:Okay, but, but…
unnawut:Actually, I can shoot… I can see my screen. Oh, okay. Oh, there are some… oh, oh.
unnawut:I think it's a Zoom bug.
unnawut:Let me maybe…
Thomas Coratger:maybe try to unshare and reshare. I think this is very recent, like, from the last 20 seconds or something like this.
unnawut:Oh, okay.
unnawut:Soon.
Shariq Naiyer:Nice.
unnawut:Okay. Yeah. Yeah, so…
unnawut:So I… yeah, so with the simulator, let's say, like, we target, like, a thousand, a thousand validators, so a thousand signatures per slot. I think we can make deal with this, because it's, yeah, the each of the 200, signature aggregation, it's gonna take about less than 600 milliseconds, which, if we account for 200 milliseconds propagation, then we can probably fit it in.
unnawut:And then the root, aggregation here, the recursion here, will take about 2.23 seconds, and so we should be able to…
unnawut:yeah, basically due to the availability, Shane had update.
unnawut:By basically just propagating the leaves, so you can… you can update the head with that, and then within, at least, like, within the slot, or maybe a bit, like, overrun into the next slot, you get the root aggregation for the 10,000 signatures. I think this should be able to achieve
unnawut:So with the calculations, yeah, I'm looking at, like, less 3.96 for the slot version, overall. But, yeah, I also show here the block inclusion delay, so it's, it's the next two blocks that you will actually get, get this route.
unnawut:Like, included into the bot.
unnawut:If anyone is interested, again, like, these are pretty much, like, interactive. You can… you can add the number of, like, fanhon you want, and then you can see here the scaling up of the numbers here. The number of, like, layers that we want to do, like aggregation, is also possible.
unnawut:I think one interesting thing that's not also… that's not captured in here is that for, like, the deeper aggregation like this, it doesn't mean that we need, like, 1, 2, 3, like, 7 aggregators here, because, in order to do, like, say, a Tier 2 aggregation.
unnawut:the Tier 1 aggregation should have finished before. So, technically, you'll always… the maximum number of aggregations you will need is on the number of leaves, and then for each of the upper layer, we should be able to reuse the same aggregator instance to do the next level aggregation. So, the machine's count is not,
unnawut:The biggest worry.
unnawut:Another thing to also mention is that for the bigger fan-ins, I think, like, the bigger I go for, like, 8 or 16 fan in, like, there is actually a penalty there. So the, like, compared to, like, a 2 fan in versus, like, 8 fan in, 8 fan in will take more time to aggregate for the same amount of signatures. So it's always kind of, like, easier
unnawut:To do a small fan in, and it also, like, reflects here on most of the calculations here, that you'll see the number of fans in around, like, 2 or 4.
unnawut:So, all in all, like, the tool is already up on bench.beanroadmap.org,
unnawut:it's secure, it's… something is wrong with my browser, but yeah, you can access it. And for my recommendation, at least for… even for devnet 4 that we are still running, I think, like, targeting something… I think if I can remember correctly, yeah, so… so doing, like, a 200…
unnawut:Yeah, there's a few numbers, but I think something like 2 subnets of 250 validators, that's a very safe bet. If you want to stretch it a bit, then, you know, like, 5 subnets with 200 validators each. I think that's also something that's well within the thought timing that we have.
unnawut:Yeah, and so I think that is it for me.
Shariq Naiyer:Thank you for that wonderful.
Thomas Coratger:Great. Amazing.
Shariq Naiyer:Amazing. Amazing.
Shariq Naiyer:Okay, up next, what do we do with that information? We… we now know we could run it in these topologies. So here's where… I'm gonna let… I'm gonna let the cheers come in before… for Ro, before I go into this.
Shariq Naiyer:Sweet. What I'm working on is… here.
Shariq Naiyer:I hope you guys can see my screen.
Thomas Coratger:Yep.
Shariq Naiyer:So…
Shariq Naiyer:Patha has been running devnets, he runs all the clients at once, and tries to find any issues, and tries to raise those issues. I'm gonna be running another series… my goal is to reach stability among all clients. Now…
Shariq Naiyer:we have, like, 8, 9 clients, so it shouldn't be that hard. So, I was trying to… I was trying to… so I built this tool called Lean Start. This uses Kubernetes in the back, and has a very nice UI. You do Lean Start.
Shariq Naiyer:Or lime, zeem E lambda.
Shariq Naiyer:let's just ensure that we don't have a lean start, lean start destroy… so that scales down previously… that deletes previously running devnet. You clear this, then you do lean start…
Shariq Naiyer:So you could also say, oh, I want one… one of Zeam, two of Zeam, so I could run two Zeam, I could run them in two subnets. So there's a lot of… there's a lot of, like, playing around, but let's just run Ream, Zeam, and Ethlambda, one subnet.
Shariq Naiyer:So, this will now start a run with Ream, Zeam, and Ethlambda, and would save these logs in these timestamp format, as well as the latest logs will be in the latest folder.
Shariq Naiyer:Now, I was trying to run all the clients at once, I was trying to run Ethlambda, Lantern, Qlean, Ream, Zeam, as much as my, my, home piece, personal laptop would allow. But what I was finding is, like, it's hard to debug there, it's hard to figure out what's going wrong, and…
Shariq Naiyer:So, here's what I started doing. I started running smaller subnets, like, smaller…
Shariq Naiyer:One thing to note is, when running in one subnet, a lot of clients are pretty stable. However, when running in two subnets, like, clients start to break quite easily. So, what I've started with first is running Ream, Zeam, and Ethlambda. This tool is open source, and it's available on,
Shariq Naiyer:like, Ream Labs slash lean start, so anyone can run these devnets and start running them. You could also go to Clients, and you could change what Docker image you're using, so you could test with your local Docker image and stuff. So, what I've started doing is I'm running 3 clients right now, Ream, Zeam, and Ethlambda, figuring out what the issues are. If it's small enough, I'll just make it, test it on a local build, and push a PR to the client, and…
Shariq Naiyer:we are in, but if it's slightly larger, I will let,
Shariq Naiyer:I will let the client team know to fix it. One thing is, one thing is with, like.
Shariq Naiyer:with 3 clients with, like, these less amount of logs, it's easier to debug any issues, and it's easier to debug any issues for me and for my LLM. It's like, an LLM can't take, like, 100, 40 log files and try to do anything from it. It's just lost.
Shariq Naiyer:So I'm starting with Ream, Zeam, Ethlambda. Once this is stable enough, stable, I'm gonna see… currently, what I'm finding is Ethlambda does not reach finalization, while Ream and Zeam
Shariq Naiyer:comfortably reach finalization, but the issue might not be any… the issue may not be… Wait, it's not this one, let's go to this one.
Shariq Naiyer:So, yeah, so… Zeam reached finalization, both Zeam, reached finalization, both,
Shariq Naiyer:lean reach finalization, but Ethlambda did not, did not catch up. Now, one thing to note is this is likely not a problem in Ethlambda. Like, this may be a problem in one of the three clients, but it's an interop problem. Clients are not interoping with each other. I'm going to figure out the problem here, and see the step forward. Then, I'm going to integrate Grandine.
Shariq Naiyer:This is in no specific order, but I go see the visualization on Hive, I see, oh, which is the next probable client I should integrate, I'll do that, and then I'll go up from there. But my goal is to have all our clients being able to interop with each other on two sub… on two subnets, and then I'm going to go to three subnets.
Shariq Naiyer:And… And we'll have a framework to scale this to 500 validators as we're planning.
Thomas Coratger:That's great, like, it seems like you have done a lot of work within the Ring team, like, thanks a lot.
Shariq Naiyer:One other thing to note, one other thing to note. Because this is Kubernetes-based, now, I haven't, I haven't tried this out, but I think… My brother's calling me, he may be having something, but, let me mute this one thing.
Shariq Naiyer:he was in the hospital for, like, 10 hours yesterday. So, I have no idea. Okay, so, one other thing to note is, what Partha, I think, does right now is he starts the Genesis, he goes on each of these servers and starts the client.
Shariq Naiyer:Maybe you can confirm or deny that. But when it's Kubernetes means you could put all those nodes, all those physical hardware you're running in a cluster, and you could have one master control server, which could deploy to all those clients, as many nodes as you want.
Shariq Naiyer:In, like, one command. So, I think, I think from a user, from a developer experience, infide experience, it's a pretty nice experience. I'm gonna test those things out once I reach a little more stability between my current runs, and we'll go from there.
Thomas Coratger:Great.
Thomas Coratger:Amazing. Thanks a lot.
Thomas Coratger:For this.
Thomas Coratger:Let's go to the next team, it's Lambda.
Mega | Lambda:Hello.
Mega | Lambda:As for our update, we…
Mega | Lambda:The last thing we added was, support for the LibP2P Identify Protocol that we noticed Shin was expecting from
Mega | Lambda:I think Tama just froze.
Mega | Lambda:Can everyone hear me?
Thomas Coratger:Yeah.
Thomas Coratger:Sorry, I had a bad connection, so…
Mega | Lambda:No problem.
Mega | Lambda:So as I was saying,
Mega | Lambda:we… we think we fixed interoperability between Lambda and Jin by adding that protocol. We… we are running Jin on a lot of test nets, and it's working.
Mega | Lambda:We also added… improved the duplication of upgraded payloads, which should improve the quality of the… of aggregation with Islamo.
Mega | Lambda:And we had an external contribution improving the fork choice visualization we have in our node.
Mega | Lambda:We are also testing, still testing .NET 4 in our servers, but we also started work on devnet 5, we are trying to study…
Mega | Lambda:More on goldfish and stuff.
Mega | Lambda:And we are also trying to write down a proposal, and we aim to have some code… some code example.
Mega | Lambda:of the… Some proof of concept of at least a heartbeat working soon.
Thomas Coratger:Great.
Thomas Coratger:Thanks a lot.
Thomas Coratger:Clean?
Ruslan Tushov:Hello. Sorry, no updates from Colleen this week.
Ruslan Tushov:Thank you.
Thomas Coratger:No problem.
Thomas Coratger:Long term?
Mihir Faujdar:Yep, thanks, Thomas. Hi, everyone. So I've just been working on the redesign of the P2P library,
Mihir Faujdar:It can achieve, interop with Go and Rusty P2P via the test plans repo, and as well as it's, compatible with Shadow Simulator Tool. And, yeah, and currently I'm testing some memory fixes. Thanks.
Thomas Coratger:Greats… Lighthouse?
Thomas Coratger:Grandin?
Mercy Boma Naps-Nkari:Hello, good afternoon. So, we've been working on, improving the out-of-memory issue we've been having, so we tried to, like, move most of our CPU interpretive tasks to, a dedicated executor we built in-house, and…
Mercy Boma Naps-Nkari:Hopefully, by end of tomorrow, we should have a new image.
Mercy Boma Naps-Nkari:Thank you.
Thomas Coratger:Great, Guin.
Shaaibu Suleiman | gean:Yeah, hello, hello. Yeah, so, on our site, we've been, running, devnet4, basically, Gean, with a couple of other clients, basically, doing some testing.
Shaaibu Suleiman | gean:And, stabilizing the clients, for devnetwork, basically.
Shaaibu Suleiman | gean:And, also working on a couple of, other optimizations, basically. And, also, we are working on implementing, the new update for LeanMetrix, basically for devnetwork.
Shaaibu Suleiman | gean:We are working on, putting that also in place.
Shaaibu Suleiman | gean:And, part of the work we've been doing also is around, basically looking into or considering, the proposal for devnet 5, with the whole stuff around, Goldfish, the finality ticket, moving away from 3SF, and…
Shaaibu Suleiman | gean:And all that. So, yeah, basically, those are some of the things we've been, doing lately. Yeah, thank you so much.
Thomas Coratger:Great, thanks a lot. Nimbus?
Thomas Coratger:Did I miss anyone?
Thomas Coratger:Okay, so… now, like, First, let us do, like, a brief spec update and a lean VM update.
Thomas Coratger:So, on the spec.
Thomas Coratger:This week, we haven't done that much things new, because, like, we were in preparation of DevNet 5, and we wanted, like, to stabilize things a bit, so we have just added, like, a bunch of new tests, otherwise, like…
Thomas Coratger:we just did, like, a movement of file, especially with Leo from the Steel team inside the EF, in order, like, to satisfy this multi-fog system where no, like, you have a better system of inherents, and everything is pretty smooth right now.
Thomas Coratger:So this is mostly, mostly done. Nothing has changed in the spec, no lines of code or no documentation, but just, like, a movement of file in order to reorganize things a bit to accommodate for future potential, like, fork or, like, a system of inerrance between the staff.
Thomas Coratger:So that is one thing. Otherwise, it doesn't move that much. We have done, like, a bunch of updates on Wheel. We have, like, run this benchmark of WE Air VS, Fry on Plonkey 3.
Thomas Coratger:we have did, like, a bunch of small optimization on the sum check, on various primitives in BrunQ3. That was very kind of minor, but, like.
Thomas Coratger:still exist, and yeah, I think that maybe Emil wants to give an update on leanVM's situation right now.
T. Wambsgans:Hello, everyone. Yes, some,
T. Wambsgans:improvements on the XMSS being used on the main.
T. Wambsgans:And some, implementation with type 1, type 2 signature, and about this…
T. Wambsgans:basically, I will probably… we should probably continue to talk on the devnet 5 group.
T. Wambsgans:to discuss exactly what API is needed. I believe, currently we have everything we need, we just have to build some helper functions, I believe, and it should work. So that's it.
Thomas Coratger:Great, thanks for the update. Now, maybe, like, it's a good time to talk about devnet 5.
Thomas Coratger:So, like, to briefly summarize.
Thomas Coratger:in devnet 4, like, we, we had this, this stuff where we aggregated,
Thomas Coratger:Like, the same attestation data, but we still had potentially, like, different proofs for one block.
Thomas Coratger:And the goal of, the goal of the devnet 5 is to have, like, a single
Thomas Coratger:posted on-chain for each block, so this system of recursive aggregation. And, like, we have this very nice devnet 5 proposal from Anshal, like, that I shared in the GitHub issue of the meeting.
Thomas Coratger:Where we have, like, these two types of aggregation type that Emil was just mentioning, the Type 1 and the type 2, where in the Type 1, you have a single message, single sort aggregation, and in the Type 2, you have this multi-message aggregation, and you are able to recover
Thomas Coratger:the information from… from Type 1.
Thomas Coratger:So that is one part, and the second part is indeed, like, goldfish, because, like, we want to add, PQ a bit, as soon as possible, because this is one of the topics of the Stroman roadmap that can potentially reach mainnet.
Thomas Coratger:So we want to test it as soon as possible. But for me, like, the PQ things remain, like, the first objective, because we want to test PQ attestation, and their propagation and their aggregation, so this is the first thing that we should tackle.
Thomas Coratger:So maybe we will proceed in this order, like first having a PR merge about this, single proof for one block.
Thomas Coratger:And then we can see goldfish, what we can do about that.
Thomas Coratger:Maybe we need some discussion, but I have, like,
Thomas Coratger:a proposal about this regarding the fact that PQ, threats is coming,
Thomas Coratger:in a more urgent way right now, especially with all of the papers from Google and all of this. So, probably, like, we want to accelerate a bit on this, and maybe potentially accelerating the roadmap on this.
Thomas Coratger:So, what does it mean? It means that we would like to have, like, real numbers as soon as possible, simulate, iterate, observe, and all of this, with a growing number of validators, so a couple hundreds, and 1Ks, and 10Ks, and 100K, potentially, like.
Thomas Coratger:We want to experiment about this.
Thomas Coratger:so that means that, in my mind, the consensus protocol that we choose is
Thomas Coratger:important because we want to test, but not the primary importance, because if we want to ship PQ as soon as possible, we want to test the propagation and the aggregation of PQ, so my proposal here is that we can add, like.
Thomas Coratger:PQ first, and then, like, recursive PQ with one proof per block first.
Thomas Coratger:And then, potentially goldfish.
Thomas Coratger:And then we can potentially freeze, further updates for the foreseeable future.
Thomas Coratger:And just test, over this, reach a more stable point of view, with all of the clients.
Thomas Coratger:Check, like, stability, testing.
Thomas Coratger:metrics, observation, and all of this. This is why I liked a lot, like, the presentation of the Ring team, with all of the benchmark tooling and visualization.
Thomas Coratger:Because this is what we want to share with the community, what we want to share with the existing, like, other clients, potentially, with the execution client and all of this. Also, within the EF, with the PandaOps team, and all of this, this is…
Thomas Coratger:potentially what we want to share, so maybe a good idea could be to add this. Once we have that, we have, like, a kind of,
Thomas Coratger:full features that we want to test PQ, at least.
Thomas Coratger:And then we can iterate over this.
Thomas Coratger:and, like, HUD stability and check.
Thomas Coratger:just let me know what you think about this. Maybe this is the right time to discuss with the client teams.
Gajinder Singh:Anchili, if you are there, can you speak about whether the requirements for devnet 5 single, signature block…
Gajinder Singh:Are sort of there, and how ready we are about it.
Anshal:Yeah, so as far as, like, the APIs are concerned and the discussion that I had with Emil, we are mostly done. We just need another…
Anshal:helper function that, Emil was talking about, so that we can extract,
Anshal:approved for a particular message based on the index. So this additional helper function will basically help us know which particular index that, particular message was added to, and using that, we can get a
Anshal:proof for the message and the included validators that are, that are there. So, with that, we'll be able to get, like,
Anshal:A complete, we'll be able to get, like, a complete,
Anshal:like, I'll add a few helper functions on our end as well, on the spec side and the client implementation side, but as far as, like, the implementation work for Tevnet 5, for a single block proof is considered, we are, almost there, so maybe I can start speccing on that side.
Anshal:And on the parallel, we can, discuss and finalize upon the consensus mechanism.
Thomas Coratger:I think that, top priority right now would be to have this,
Thomas Coratger:PR, at least, open, on lean spec.
Thomas Coratger:For, for, for the mechanism of, like,
Thomas Coratger:Recursive proof all aggregation with type 1 and type 2, stuff.
Thomas Coratger:Because, yeah, for me to test PQ, this is the highest prior that we could have right now.
Thomas Coratger:So yeah, for me, the top priority is to have, like, a PR like this.
Thomas Coratger:Only inspect, do you think that you have, like, all of the tooling to do that right now, or is there something missing? Do you need help from my side? Anything, please let me know.
Anshal:Yeah, sure. There isn't anything missing as such, I can definitely start working towards it, because, like, APA signature is mostly finalized, which I'll, need to create a PR. Right now, like, there's still some work left for
Anshal:MIL to do on integration with the current LeanSig and integration of, like, the prod and test config signatures, using which I can update my bindings and have, like, a proper aggregation of stuff, working on LeanSec side of things. But since, like, we have already finalized upon… mostly finalized upon the
Anshal:API side of things, we can, move forward and, I can move forward and raise our spec PR.
Thomas Coratger:Yeah, that would be great. Yeah, agree.
Thomas Coratger:So, yeah, and my idea, like, was, yeah, if possible, to put goldfish. If not possible, we can wait a couple of weeks.
Thomas Coratger:That's not a problem for me, even the fact that we have to mock, kind of, the finalization, because for now we are… we don't have, like, a finalization gadget if we put on goldfish.
Thomas Coratger:Because we will drop, basically, everything.
Thomas Coratger:from 3SF Miniso.
Thomas Coratger:But for me, all of that is not a problem, since what we need is to test PQ, and for now, we haven't run, like, a real devnet that is kind of full stable with, for example, 10K client… 10K validators or stuff like that, so…
Thomas Coratger:This is, for us, what is really important, as part of the job.
Gajinder Singh:Yeah, just to add to that, Tom, I think what we actually need is, not really a something naive finalization mechanism, let's say some KD pool, because we also want that, for example, if, if the nodes are basically restarting and,
Gajinder Singh:they are rebooting from a checkpoint, then they should not produce blocks that violate the finalization, because in KDeep, it is possible to violate this finalization by just creating a branch out of
Gajinder Singh:for example, child block out of the finalized block itself. So, basically, we need to think a little bit more onto this, and what I propose
Gajinder Singh:is that we have a breakout on this. Jan has shared a very useful link on
Gajinder Singh:on this thing where it talks about goldfish plus stabilization gadget plus finality gadget, and I'm basically digging through it as of now to understand the interaction between them.
Gajinder Singh:And what we could… we could have is maybe, some sort of a finalization rule that can also be coded in straight transition function as well. I mean, which is straight transition function frontier as well, so it is not just…
Gajinder Singh:poke choice that we are concerned with. We are also concerned with straight dungeon function, because, again, you know, if you get a block.
Gajinder Singh:you will have to run it, and you will basically, have to get that finalized with whatever way you are treating your fork choice with. So, in that sense, I think we need this…
Gajinder Singh:We need a breakout call, and can we sort of come up with a time to have a breakout call before the next…
Gajinder Singh:Wednesday call, so that we are a bit clearer on it in terms of
Gajinder Singh:What is the very rough mechanism?
Gajinder Singh:That we… we can come on page with, with regard to the entire consensus.
Thomas Coratger:Yeah, so, two points. For me, like, this is not a problem. Even, like, if we need a bit more time to think about this, the important point is to have, like, the PQ things done, that is very important.
Thomas Coratger:For the goldfish part, even if we have to wait, like, 2 weeks, 3 weeks, or 4 weeks, for me, this is totally okay.
Thomas Coratger:We have the… we have the time for this. We can still, with the 3SF Mini, for the time being, this is totally okay, and continue to experiment with PQ, that is fine.
Thomas Coratger:Regarding ourself.
Thomas Coratger:as EF people, at least Emil and myself, for the next week, I guess this will be complicated, because we are in Rome.
Thomas Coratger:for, like, a ZK event, or…
Thomas Coratger:We are in Rome from today to…
Thomas Coratger:next Wednesday, I think, so it will be probably complex, but feel free to do that even without us, like,
Thomas Coratger:If this is just, like, to discuss, like, these contentious things, like, feel free to do so.
Gajinder Singh:Okay, so maybe… Next Thursday or Friday, we can have this call, I guess.
Thomas Coratger:Yeah, totally, yeah.
Gajinder Singh:And yes, I agree, right now, Anchil can just drop in the PR with regard to peak changes, and aggregation changes, and…
Gajinder Singh:then we can follow up with the separate PR regarding consensus changes, and if we are not able to finalize, we can also do devnet 5 without consensus changes. So, that is entirely possible, so definitely on page…
Gajinder Singh:With you over there. But yes, I mean, it's… it's good that if we… before we free spec, basically, we also put in the heartbeat and use this consensus so that this is something valuable. We can also
Gajinder Singh:Sort of upstream, in the sense that we can also contribute back to the main, to the mainnet.
Gajinder Singh:By basically saying that, okay, you know, we have tried it out, we have sort of figured out what the issues are, or we have solved what the issues are, and we can basically propose a cleaner scheme.
Gajinder Singh:After running the devnets on it. So in that sense, before we freeze the spec, basically, we should definitely, have this as part of
Gajinder Singh:our lean Ethereum roadmap this year.
Thomas Coratger:Yeah, perfect. And as mentioned by Yanni in the chat, indeed, like,
Thomas Coratger:I always prefer, like, Goldfish rather than 3SF Mini, because we know right now that 3SF Mini will never be shipped in production to mainnets. This is a certain fact. And goldfish can probably be, so I always prefer to test something that can potentially reach production.
Thomas Coratger:But indeed, like, the goal right now is to test PQ, and I want to have PQ, especially given the fact that internally, within the EF and
Thomas Coratger:external discussions that we had, is that we… we will potentially, this is a possibility, not sure for… but potentially, like, accelerate the PQ roadmap.
Thomas Coratger:This means that we have to test it, like, in real-case scenario as soon as possible, and that's why I,
Thomas Coratger:I want to test PQ, I want to test with a growing number of validators, I want to test the transport layer and all of this, so I think that we should work on this. For me, the pretty good news is that once we have, like, a single proof for each block and chain after the PR of Anchal.
Thomas Coratger:I think that we will have basically all of the features that we need to test PQ. The rest is just, like, MISC or stuff like this that we could add later on, and this is not a problem. We will be able to test PQ.
Thomas Coratger:So, yeah.
Thomas Coratger:I think we can do that way. We first do the PR of PQ,
Thomas Coratger:With minimal change. We stay with 3SF Mini.
Thomas Coratger:Then we can have this breakout call session next week, end of next week, about goldfish.
Thomas Coratger:We can see, how using Goldfish could simplify our consensus right now, while keeping a kind of finality, or mocked finality, or whatever we think is the best.
Thomas Coratger:And then we try to implement it, it is simplified.
Thomas Coratger:In the meantime, I think that after that, we can just, like,
Thomas Coratger:Focus on stability, freeze a bit the change that we had to the clients or to the spec.
Thomas Coratger:and do, like, an iterative process on, like, trying to reach, like, better deveness, better stability, better observation tooling, and all of this, better visualization, to share what we are doing with the community. I think this is the best, for me, I have thoughts about this.
Thomas Coratger:Quite a bit recently, and I think that this is maybe the best way to go right now.
Gajinder Singh:Something's like a plan.
Thomas Coratger:Perfect, so… we are pretty much on time. In general, we have, like, a couple of minutes about, like, metrics and stuff. Maybe if, Katya, you want to add something, if you had something in mind, feel free.
Katya:Oh, well, just briefly, these two PRs for latest metrics has been… have been merged. They are pinned in… all the .NET4 metrics are pinned in the tooling channel. Thank you.
Thomas Coratger:Great, that's perfect. Anything else that anyone wants to hide, or like, yeah, any comments, questions, stuff?
Thomas Coratger:Otherwise, we have exactly at time, so, yeah.
Thomas Coratger:Grace, thanks a lot, guys. See you next week.
T. Wambsgans:Bye.
Gajinder Singh:We do.
Pablo Deymo | Lambda:You, my wife.
Shariq Naiyer:Okay.
Pablo Deymo | Lambda:Why would they be wrong.
Chat Logs
00:25:04
Mega | Lambda:Looks great!!
00:34:56
T. Wambsgans:thats interesting
00:35:03
T. Wambsgans:idk why
00:35:23
T. Wambsgans:good info, thanks a lot
00:37:33
T. Wambsgans:correct
00:39:05
Parthasarathy Ramanujam:I think 200ms is optimistic as we are not in the same datacenter.
00:39:17
Parthasarathy Ramanujam:Cool benchmarks.
00:43:48
Yann Vonlanthen:How are the budget constants chosen? Or are they just placeholders right now?
00:45:24
Shariq Naiyer:https://bench.leanroadmap.org/
00:45:59
T. Wambsgans:for info: it works nice on my safari, but not on my firefox (macos)
00:46:14
unnawut:Replying to "How are the budget c..."
Just placeholders, based on the values we use in the past devnets
00:47:18
Yann Vonlanthen:Replying to "How are the budget c..."
Thanks!
00:47:47
unnawut:Replying to "for info: it works n..."
Bench right? Thanks! I’ll have a look
00:48:27
T. Wambsgans:Replying to "for info: it works n..."
will send screenshot on TG
00:51:00
Parthasarathy Ramanujam:The script does that.
00:51:10
Gajinder Singh:ansible is deployed from a master
00:51:22
Parthasarathy Ramanujam:lean-quickstart had support for k8s as well. We just deferred full implementation for later
00:56:20
Gajinder Singh:problem with k8s is that it runs its own network layer if you wanna run it in multiple nodes, which might be prohibitive to have closer to real network conditions even if nodes are across data centers for k8s
its priamarily ansible vs k8 tradeoff and ansible might allow a per node management better than k8 although it can also be done in k8s
so in the end its more about what people are well versed with
00:57:23
Gajinder Singh:Replying to "problem with k8s is ..."
(disclaimer: I run multiple node k8s for personal stuff, so I do know a bit about its admininstration)
00:58:03
Gajinder Singh:Replying to "problem with k8s is ..."
nodes = machines in k8s lingo
01:01:41
Yann Vonlanthen:100% agree!
01:02:00
Shariq Naiyer:Replying to "problem with k8s is ..."
interesting I am going to look into this tradeoff further I haven’t used ansible as much yet
01:02:46
Gajinder Singh:Replying to "problem with k8s is ..."
your work is quite redundant in that regard to be honest, we should coordinate on how to maximise useful work
01:02:58
Gajinder Singh:Replying to "problem with k8s is ..."
although good work
01:04:00
T. Wambsgans:I am going to send a msg in TG @Anshal
01:10:42
Yann Vonlanthen:I think switching to Goldfish makes sense only if it simplifies things, if you feel like it doesn’t, I wouldn’t make it a priority at all
01:12:30
Gajinder Singh:Replying to "I think switching to..."
we wanna devnet the PQ heartbeat for mainnet, so that is the target we should go for in terms of consensus
Summary
16 highlights
· 2 action itemsExperimental
Summary
16 highlights · 2 action itemsExperimentaltesting infrastructure
performance benchmarks
- Lean signature performance flat across modern 4+ core CPUs00:32:04
- Aggregation scales linearly to 16 cores; 32+ cores show diminishing returns00:33:08
- XMSS signing optimization dropped time from 200ms to 5ms00:36:23
- 1,000 validators achievable in current slot timing; 10,000 possible with adjustments00:42:09
client progress
devnet 5 scope
- Devnet 5 goal: single recursive proof per block via Type 1/2 aggregation00:58:37
- PQ testing prioritized over consensus mechanism changes01:00:08
- Type 1/2 signature APIs mostly finalized; helper function needed from LeanVM01:05:04
- PQ roadmap acceleration possible; real-world testing urgent priority01:12:23