Jihoon:start this call. Alright, hello and welcome, everyone, to Fossil Breakout 20.
Transcript
Jihoon:Same as before, we can jump right into status updates and have some discussions, if any.
Jihoon:I guess.
Jihoon:We already have.
Jihoon:Everyone who are active on fossil development these days, but do we have any development metrics or testing updates?
Jihoon:Yes, Katya?
Katya:Hello, everyone. I updated the PR and added these new metrics implemented in Lodestar in this PR. Yeah, I left a couple of comments. I've seen your answer already to whom.
Katya:So, yeah, I'm not sure. My question is, are we going to have some simulation of failing inclusion lists, or equilocating inclusion lists? For example, for peer does,
Katya:prison team, made this… added the CLI flag, which… which caused kind of loss of columns, and so we could check a little bit.
Katya:This kind of thing, do… do you have something on your mind, something like this?
Katya:Are we gonna check it?
Katya:Somehow, or just through the dead nuts and…
Jihoon:Yeah, that's a good question. I think…
Jihoon:We can take our two prongs here. One is using some CLI flag, as you just mentioned, to produce some
Jihoon:Invalid or equivocating ancological leads.
Jihoon:And the second approach would be, setting up Author Tour to, you know, run some tests over invalid or duplicating culture lists.
Katya:about 4 weeks ago.
Katya:Yeah, for the tests, the metric, the metrics won't work, so we won't, see the simulations on the dashboard, I guess so. So, with CLI common, that would be great, would be interesting.
Katya:At least to check the metrics, and yeah.
Jihoon:Yeah, I agree. Once we,
Jihoon:decide to include fossil in Glamsterdam, we can add such flags so that we can test if every client is working as expected, and use your metrics dashboard to
Jihoon:See how it works.
Katya:So, currently.
Katya:My latest run of Lodestar was yesterday, and I had some issues, but we're trying… we're discussing it with the Lodestar team, so maybe, yeah, I think they made some updates, and they need to check again today.
Katya:That would be great, if we could have some interop,
Katya:I don't know, maybe you could try to implement these metrics as well on the prison side, or then…
Katya:We can launch some kind of interop, even through kurtosis, maybe some of these metrics that would be interesting to look into it.
Katya:Whenever you have some time.
Jihoon:Yeah,
Jihoon:Do you know what kind of work that most of people would do? Is it about rebasing, or just adding metrics?
Katya:The metrics are added, yeah, I think rebasing.
Jihoon:Oh, great. Yeah.
Jihoon:Yeah, right.
Jihoon:for Prism… Hmm.
Katya:We have kind of… we have the set of metrics, we have the dashboard, so it's technically possible if some other clients will build the metrics, it's technically possible to check, to compare it to clients, for example.
Jihoon:Yes, for PRISM, I think…
Jihoon:it'll take some time to rebase it on the flow. Yesterday, I…
Jihoon:I had opened a PR that enable
Jihoon:fossil genesis block to PRISM, and it was marched yesterday, so I think we can start working on replacing, on the PRISM side.
Jihoon:But…
Jihoon:the branch is a bit stale, so I think… not sure. I need to take a look into it to measure how much it would take. To be honest, I've been mostly working on something else, so I cannot promise, when that
Jihoon:when that I'm gonna start working on it. Yeah.
Katya:Okay, just let me know if this rebase happens. Just let me know, maybe I can take a look into it, how to add metrics.
Katya:Yeah, just… Pin me somewhere.
Jihoon:Okay, great. Maybe you can try interop with Lighthouse?
Jihoon:Not sure if Lighthouse is on Flow.
Katya:I, I will, I will ask.
Katya:In the channel.
Jihoon:Okay, great.
Katya:as soon as I can run Loadar.
Jihoon:Yeah, sure, yeah, take your time.
Katya:Thank you.
Jihoon:Thank you.
Jihoon:Do we have any other updates?
Jihoon:On my end, I've been mostly working on something else, and a little bit of spec tests.
Jihoon:I do hope I can finish it by this week, and I've been saying this over and over again for the past few weeks, but I…
Jihoon:I'm gonna aim to finish it by this week, and open up a PR against Constance's spec repo.
Jihoon:Yeah.
Jihoon:Hi, Doma. Hi, Nancy. We were just sharing some status updates.
Jihoon:Yeah. Welcome.
Jihoon:Do we have any… Updates on red side.
Pelle Krabbenhoeft:On the rest side, it's been pretty much just cleaning up some of the code and doing the interop testing with, Geth, which has all been going well. If you want, I can talk on the…
Pelle Krabbenhoeft:Lighthouse, issues I was finding and working on real quick.
Jihoon:Yeah, please share it.
Pelle Krabbenhoeft:Yeah, so… Lighthouse…
Pelle Krabbenhoeft:was only passing zero inclusion lists even back for Electra, which I thought was kind of weird. So I looked more into it, and it seems like, the inclusion list that it's trying to get from the cache is… it's trying to get from, slot N instead of slot N minus 1.
Pelle Krabbenhoeft:And when it's trying to get the inclusion list, there's obviously no inclusion list, because…
Pelle Krabbenhoeft:those inclusion lists for that slot haven't been sent out, so it's just grabbing one slot ahead. So I'm working on having it grabbed from one slot behind, and so far, it's been working in testing. So, still need to iron out a few things with that, but I'll probably open a PR for it.
Jihoon:Great, great. And, if I remember it correctly, there's, another issue that some blocks are missing.
Pelle Krabbenhoeft:Oh yeah, when I just put my naive, implementation together.
Pelle Krabbenhoeft:I was running Geth and REST together, and both clients were sometimes failing blocks, or not proposing blocks. I know on the REST side, my logs were saying that it was an inclusion-less issue, so…
Pelle Krabbenhoeft:one transaction didn't get included. I didn't quite check… I didn't quite figure out if it was the same issue on Geth, but I'm guessing it's probably the same. But still need to look into that.
Jihoon:Could you elaborate a little bit of how does the enclosure list transaction prevent some block from, being proposed?
Pelle Krabbenhoeft:Yeah, on my end, just the inclusion list transaction didn't get included. Sorry.
Pelle Krabbenhoeft:And that's why it was failing the new payload V5.
Jihoon:Got it, got it. So… the CL is reorging out because it found
Jihoon:Some trans… English transactions are missing.
Jihoon:For some reason.
Pelle Krabbenhoeft:Yeah, yeah, I haven't quite gotten to the bottom of it, but that's what I've seen so far.
Jihoon:Got it, got it.
Jihoon:Right.
Jihoon:Alright, cool. So, it's really nice to see that we are having some… Debugging and progress on the…
Jihoon:On the first EL interrupt, there might be some bug in GAT, so…
Jihoon:Just, yeah, feel free to just…
Jihoon:find the bug and open up a PR or report it somewhere.
Jihoon:And, yeah, looking forward to it.
Pelle Krabbenhoeft:Sounds good. I'll let you know. And then, also, Lodestar, I don't know what's causing it, but there's also zero inclusion lists, and the logs of Lodestar say that it's successfully adding the inclusion list to the cache, so…
Pelle Krabbenhoeft:I don't know what's going on there. It could be a similar thing, but something to think about.
Jihoon:Right. But you mean Lighthouse, right? That low star.
Pelle Krabbenhoeft:No, Lodestar as well. I haven't looked into the Lodestar code, so it could be something else, but just saying it might be the same thing as Lighthouse. Not sure, though.
Jihoon:Oh, I thought that Lowestar is not working today, but… Probably, I was wrong.
Pelle Krabbenhoeft:I… it was… it wasn't working perfectly when I was checking on Fulu, but I was able… the build for a while, there were… there is another issue, but I was also getting a zero inclusion list.
Pelle Krabbenhoeft:It was a bit of a shaky testnet.
Jihoon:Got it, got it.
Jihoon:Hmm.
nflaig:What value do you use for seconds per slot?
Pelle Krabbenhoeft:I was using 6.
nflaig:Yeah.
Pelle Krabbenhoeft:Do I need to move that to 12 for…
nflaig:Yeah, I think so, because we didn't move the code to the BPS yet, so to use basis points.
nflaig:So I think it uses absolute values, and then you get to the… yeah, it submits the wrong inclusion lists, because the freeze deadline or the submit deadline is 9 seconds, so it's outside of the slot.
nflaig:I think that might be an issue, yeah.
Pelle Krabbenhoeft:Yeah, that sounds like it probably is that. I'll have to get that a look.
Jihoon:Yeah, remember that we had this issue…
Jihoon:I don't know, like, many months ago. Right, yeah.
Jihoon:Makes sense.
nflaig:Yeah, I don't think we have fixed anything in that regard, because the spec was just like that, and was just recently changed.
Jihoon:Yeah, you're right. So, I guess we can try interop between, Lostar Lighthouse, GED and Rev.
Pelle Krabbenhoeft:Yeah, we can probably do that.
Pelle Krabbenhoeft:Especially once I get Lighthouse fixed up.
Jihoon:Yeah, sounds great. With that, would it be enough for…
Jihoon:testing the metrics dashboard, Katya, if we have, like, at least two CL clients.
Jihoon:Even if we don't have metrics implemented in Lighthouse, it'll be implemented in Lowestar, so…
Katya:then it doesn't make sense if the other client… I mean, for metrics interop, if the other client doesn't have… so it will be just…
Katya:door, and that's it, so I won't see anything.
Katya:on the Lighthouse, on the dashboard.
Jihoon:Right, right.
Jihoon:Like, could you remind us what would be the…
Jihoon:Goals of this metrics, testing, like, how, how…
Jihoon:Like, let's say that we implemented the metrics correctly in all clients, then how do we tell if they are implemented correctly?
Katya:I mean, first of all, we are just observing the values on the dashboard, and I mean, it's more for…
Katya:The final goal is to observe this metric in the long-term period, so when we already have DevNets, so normally DevOps teams just observe if something's going wrong.
Katya:During the long period of time, it's… it's… It's good to use dashboards.
Katya:And at the same time, we can… we can just compare, different values.
Katya:On different clients, so… Which also sometimes can be helpful for the clients itself, like…
Katya:How long is, like, purification going, etc, so… and to find some issues, maybe?
Jihoon:Got it, got it. So… even if you just run a local dev net with Lowestar.
Jihoon:then you can see the dashboard bit, but it doesn't tell you anything other than you just see the metrics measured in certain ways. So you want to use that dashboard to see if others say, clients are seeing the same thing in the expected ways.
Katya:Yeah, that would be great, because this is the goal of the metric specs, so, the clients have, like, more or less,
Katya:Similar implementation in the important points, like.
Katya:You wanna see, like, how verification goes, like, how many inclusion leads are produced, etc.
Katya:So, just a… Some kind of evaluation and observation.
Jihoon:Got it, got it. Cool.
Jihoon:Right, thank you.
Katya:I would also like to collab with, EL team, maybe Reth, just to look into, what metrics could be added on the EL side.
Pelle Krabbenhoeft:Yeah, I'd love to work on that as well, so, just DM me on Discord.
Katya:Okay, thanks.
Jihoon:Very nice. Do we have any other updates, or…
Jihoon:Something that we want to discuss?
Jihoon:Otherwise, I think… We can go ahead and wrap up this call.
Jihoon:I was thinking about…
Jihoon:Yeah, I just wanted to ask, what do people think about changing falsehood breakout to monthly until
Jihoon:We have client devs come back, because people are busy with Fusaka, and then we also need to wait until a PBS spec and implementation are finalized, so…
Jihoon:On one hand, we can just skip this one bi-weekly, so that we can
Jihoon:Keep the momentum or something.
Jihoon:On the other hand, I don't know, maybe we are… We could use…
Jihoon:Our time more efficiently by making this monthly?
Jihoon:I don't have a strong opinion, I just wanted to…
Jihoon:Ask what people here think about it.
Katya:My opinion, bi-weekly would be great, not to lose focus at all, like…
Katya:One… once a month is kind of… Too rare.
Jihoon:Okay.
Jihoon:Oh, yeah.
Jihoon:Yeah, I think for now we can just…
Jihoon:Keep it as is, maybe bi-weekly.
Jihoon:Because monthly is too long, I think that's a valid point.
Jihoon:And… yeah, thank you, Thuma. There's a… there was a… there's a proposal to… use
Jihoon:fossil in the context of native roll-ups. Luca…
Jihoon:Yeah, Ruka reached out to me, like, I don't know, a few months ago, and we…
Jihoon:doubled in about how to implement this, in the native roll-ups, I mean, how to implement it using the… inheriting, the existing execution spec.
Jihoon:I think what's interesting…
Jihoon:to see this idea, maybe we can ask Luca to share his idea next breakout.
Jihoon:Yes, it's a… it's a nice summary.
Jihoon:There was a tweet thread, that Luca shared. Let me pull that link real quick, and then we can wait for more discussion, and then wrap up this call.
Jihoon:Yeah, here's the thread. Please take a look into it if you're interested.
Jihoon:And then… Yeah.
Jihoon:Maybe we can have a look here to share a little bit more about it next breakout.
Jihoon:In two weeks.
Jihoon:Does anyone have any more comments?
Jihoon:Or… anything to discuss?
Katya:Last quick question, should we start from Elektra in ketosis, or maybe you can start from Fullo?
Jihoon:I think they're starting… no.
Katya:My latest config is, like, Electrod and Foolos, and…
Jihoon:Yeah, we are still on Electra, right? Pelly, do you start on Flu or Electra?
Pelle Krabbenhoeft:For Lodestar, or…
Jihoon:For Lostar and Lighthouse.
Pelle Krabbenhoeft:For Lighthouse, I've been doing, Glow Ass and Electra, just testing both of those, and then Lodestar, I believe, is only up to Fulu right now.
Jihoon:up to Flu, so you start, you start from Electra, and then transition into Flu, and then Gloas.
Pelle Krabbenhoeft:I… I think I was just starting from Fulu for Lodestar.
Jihoon:Got it.
Katya:Okay, we'll try. Thank you.
Jihoon:Does that mean the load star already repased on the flow?
nflaig:Yeah, we have the fork between claws and fool, so in between those two.
Jihoon:Oh, cool.
Jihoon:Alright, I think… We can go ahead and wrap up this call, and… We can get their…
Jihoon:in two weeks, and share another update. And in the meantime, I will ask Luca if he can join us to share his idea on fossil
Jihoon:In negative roll-ups next time.
Jihoon:Alright, guys?
Jihoon:Thank you for joining, and… See you in 2 weeks!
Katya:Thank you, everyone. Bye.
nflaig:Yep.
Chat Logs
00:12:44
soispoke:Hi hi! Can’t speak but nice to be here 🙂
00:16:36
Katya:Where can I find the latest reth focil image?
00:16:59
Jihoon:Replying to "Where can I find the..."
https://github.com/PelleKrab/reth/tree/focil_impl
00:24:24
soispoke:New idea around why FOCIL is important for native rollups: https://native-rollups.l2beat.com/forced_transactions.html
Might ask Luca to come present soonish
00:25:27
soispoke:tldr: FOCIL infra on the L1 can be reused to constrain centralized sequencers on native rollups in the future
00:26:06
Jihoon:https://x.com/donnoh_eth/status/1961368203566739658
00:28:56
soispoke:Thanks!
00:29:06
Pelle Krabbenhoeft:Thank you
Summary
11 highlights
· 3 action itemsExperimental
Summary
11 highlights · 3 action itemsExperimentalclient implementation progress
- Lighthouse passing zero inclusion lists; identified slot N vs N-1 cache bug00:13:49
- Block proposal failures in Geth/Reth due to missing inclusion list transactions00:15:13
- Lodestar zero inclusion lists likely caused by 6s slot time vs 9s deadline00:16:43
- Lodestar rebased between Fulu and Gloas; metrics implementation active00:21:15
testing progress
specification and testing
Action Items
Targets
- Next FOCIL breakout in 2 weeks (October 7, 2025)00:28:23