Jihoon:4 minutes in, I can… I think we can start. So, hello everyone, welcome to FOCIL Breakout 30.
Transcript
Jihoon:Today, we can share development and testing updates. Perhaps I can go first with testing updates.
Jihoon:So we started EL and CL fossil testing.
Jihoon:Nothing much to share that, except that we started, taking a look into our testing. So we are in contact with Steel Team to discuss how we can go about BL testing and other CL testing.
Jihoon:I think we will mostly work on reference tests.
Jihoon:Yeah, that's actually pretty much it. If anyone is interested in helping us.
Jihoon:on testing, please let me know, and… Yeah.
Jihoon:So, does anyone have anything to share on development updates?
Jihoon:Maybe, Justin, I have a question for you as well. Last time, I heard that Besu might start implementing fossil on top of Amsterdam.
Jihoon:Do you know if… That work has started.
Justin Florentine (Besu):Yes, it has. I have a draft PR that I'm currently working on right now. It's very early phases. It's very rough, but, yeah, it started. I don't have too much to report, haven't really started testing or anything like that. It's literally just at the spec implementation phase.
Jihoon:Got it.
Jihoon:maybe you share the link to the PR?
Justin Florentine (Besu):Sure. It's in my own fork, but yes.
Jihoon:I'm… is current… spec okay for… EL implementation, I think, like.
Jihoon:It's a little bit outdated, especially when it comes to versions, like New Payload V6?
Jihoon:Ding.
Justin Florentine (Besu):Sorry, good question.
Justin Florentine (Besu):I'm basing it off of what's actually been published, not necessarily any PRs that are, outstanding.
Justin Florentine (Besu):So, it could be, it could be a little out of date, this implementation.
Jihoon:Gotcha. Let me update those PRs, by this week.
Justin Florentine (Besu):I mean, right now, this is just me and an agent kind of grinding on it.
Jihoon:Can I expect that this PR… will… Be promoted to actual implementation?
Justin Florentine (Besu):Yes.
Jihoon:Got it. Cool.
Jihoon:Nice.
Jihoon:Any other development updates?
Mehdi Aouadi:I can give a quick update for Teku.
Mehdi Aouadi:Basically, we started merging all the Heze Fork, boilerplate code.
Mehdi Aouadi:And, we're planning to add fossil on top of Haze, so…
Mehdi Aouadi:Yeah, hopefully that won't be an issue when we start interoping, because I see some people are here,
Mehdi Aouadi:adding, FOCIL on top of Glamsterdam.
Mehdi Aouadi:Oh yeah, but for now, I won't touch the Docker I published, which is based on top of Fulu, but at some point, everything will be based on top of Hazy.
Mehdi Aouadi:Oh, yeah.
Jihoon:Awesome.
Jihoon:Thanks for sharing.
Jihoon:Do we… do we have any more updates?
Jihoon:Alright, at least we have… one CL and one EL working on fossil on top of…
Jihoon:Amsterdam, so… yeah.
Jihoon:Looking forward to seeing more clients.
Jihoon:Working on FOCIL.
Jihoon:All right, do we have any other updates other than development and testing updates?
Matthew Keil:Oh, just as a note, we are planning on doing the updates, but we're just finishing up our Glamsterdam, now, so once we get DevNet Zero squared away, it's stable and there's no issues. We are intending to rebase everything on top.
Jihoon:Gotcha. Thank you.
Pelle:Yeah, I can… By leash.
Pelle:Where's their echo?
Jihoon:Yeah, please go ahead.
Pelle Krabbenhoeft:Sorry, there we go. Yeah, I can share… finally share some of the work I've been doing on the mock inclusion list real quick. Just need to have, like, 5 slides I'm just gonna run through.
Pelle Krabbenhoeft:Anyone see that?
Jihoon:Yep, you're good.
Pelle Krabbenhoeft:Perfect.
Pelle Krabbenhoeft:Alright.
Pelle Krabbenhoeft:So getting into it, ayy…
Pelle Krabbenhoeft:The main point of this research was to answer three main questions. We want to figure out what building inclusion list is going to look like in terms of redundancy. So, how many of these transactions that are getting added to the inclusion… the inclusion list, would have been included into the block without the inclusion list?
Pelle Krabbenhoeft:We also wanted to see how much, delay of
Pelle Krabbenhoeft:0 to 2 slots affects this redundancy. It's definitely going to go up, but by how much? And also see what those redundancy numbers look like.
Pelle Krabbenhoeft:for different strategies, in this case, just top fee and censorship-only I.O. So, top fee is how most of the clients, I believe, are building inclusion lists, where we just take the best transactions from our mempool, and then spend it up to the chain.
Pelle Krabbenhoeft:So, the way this was all implemented was using block-native mempool data from,
Pelle Krabbenhoeft:from PandaOps's Quick House server. I did try to use the X2 data, but it's super noisy, and there's a bunch of spam transactions in there, so about 70% of transactions in that data never even make it on-chain.
Pelle Krabbenhoeft:And it was very hard to parse for legitimate transactions, but the block native data is a lot
Pelle Krabbenhoeft:It's been cleaned, and it also has some other metadata that allows us to do this
Pelle Krabbenhoeft:analysis a lot quicker, as the X2 data,
Pelle Krabbenhoeft:it takes about 30 days on my hardware, so I kind of had the… I need to use my laptop every day, so, decided to just go with this block-native data, which took about 3 days to run. But basically, we're gonna grab that data.
Pelle Krabbenhoeft:Filter out the transactions, so no blob transactions, make sure we're reaching the base fee, and anything that makes it fossil valid.
Pelle Krabbenhoeft:And then we'll build a censorship
Pelle Krabbenhoeft:Inclusion list, which we'll get to in a second, and that top fee, and then we apply that to the…
Pelle Krabbenhoeft:N plus 1, N plus 2, and N plus 3 slots, and see how many transactions got included on-chain, and you'll see the day in just one second.
Pelle Krabbenhoeft:Here's a better picture of what it looks like for the censorship window. We're just looking at the last 12 seconds.
Pelle Krabbenhoeft:The reason for this is censorship is pretty low, so none of these censorship inclusion lists are ever getting to capacity, and also it solves the problem of
Pelle Krabbenhoeft:we're trying to build these mock inclusion lists on data where inclusion lists weren't a thing. So if we have a bigger window, then we're going to have a lot of overlap, and then it can kind of distort the numbers a little bit.
Pelle Krabbenhoeft:And then for the normal top fee, we're just doing, as the spec says, from negative 4 seconds to 8.
Pelle Krabbenhoeft:From our point of view after block N.
Pelle Krabbenhoeft:So, this is for the top fee inclusion list. As you can see, they're about full. This is, this is from the block native data, only covering about half the mempool, well, half of included transactions, and not the full mempool and being significantly filtered.
Pelle Krabbenhoeft:But…
Pelle Krabbenhoeft:We can assume it's about full, and as you can see, about 92% of those transactions are essentially wasted bandwidth, and it goes up by about 2 percentage points, and doesn't change much after the second spot, delay.
Pelle Krabbenhoeft:And then, this is what I found a little bit more surprising, with these censored transactions, so basically anything, that was in the mempool when the last block was being built but never made it, and met, all the requirements, and we're also checking to make sure that there was a room in the last block.
Pelle Krabbenhoeft:So, using this strategy, you can see we get a little bit less redundancy, so the inclusion lists are actually helping, with UX and censorship a lot more, but it is still a pretty significant amount of these transactions just aren't getting included. Those would probably fall if you're looking at a
Pelle Krabbenhoeft:further back window, but as mentioned before, I think it'd be kind of hard to backtest that.
Pelle Krabbenhoeft:So with all that said, the redundancy seems pretty inevitable, and as some people were talking before about, oh, we should only,
Pelle Krabbenhoeft:just…
Pelle Krabbenhoeft:just use censored transactions in the inclusion list. Obviously, there's no way to fleece that, so we can just assume all inclusion lists are going to be full anyway, but the best approach is likely some sort of hybrid approach, because it's not…
Pelle Krabbenhoeft:Doesn't require too much compute to find those, sensor transactions, but that is still up to the client.
Pelle Krabbenhoeft:And then, I would strongly suggest trying to keep us at that zero slot delay, but worst case, one slot delay, depending on EPBS and the implementation, shouldn't be too horrible.
Pelle Krabbenhoeft:And then, just as a… Just quick leaving notes.
Pelle Krabbenhoeft:It'd be interesting to see what happens to these numbers in a high censorship environment, so if it's a lot easier to see those censor transactions, can we see those redundancy numbers collapse, maybe testnet, future testnet and such, but, there's a lot of…
Pelle Krabbenhoeft:I don't know how important it's going to be with deciding how we end up building inclusion lists, but it'd be… it'd definitely be interesting if anyone has more, like, block-building experience can come up with a more sophisticated way to do this, but…
Pelle Krabbenhoeft:With that all said, let me know if you have any questions, and I can drop the full report in the… in the group… in the chat.
Jihoon:Amazing. Thanks for sharing.
Jihoon:It's good to see that different I.O. building algorithms
Jihoon:Already have different outcomes, even based on the past dataset.
Jihoon:I'd encourage client teams to adopt different strategies. I guess it's more about coordination that we
Jihoon:probably need to handle down the road. I have a question, so… When we… use…
Jihoon:half the algorithm, it seems 6-8% of transactions Are not timely included.
Jihoon:Do you know… Which delay after those transactions are included?
Pelle Krabbenhoeft:So the transactions that aren't here, I can pull it up again.
Pelle Krabbenhoeft:You're talking about… These transactions.
Pelle Krabbenhoeft:So… Yes.
Jihoon:So…
Pelle Krabbenhoeft:Yeah.
Pelle Krabbenhoeft:I mean, there's no specific, reason for it all the time, but,
Pelle Krabbenhoeft:I couldn't find a reason for why they're definitely not getting included. It's definitely another good thing to look into. I still have most of the data, so I could possibly take a look, but my best guess would just be,
Pelle Krabbenhoeft:And not making it into the mempool for, whoever the builder is.
Pelle Krabbenhoeft:It seems like most stuff does get included eventually, but some stuff that we can see from the block native data that sits for a while, or eventually gets replaced if it sits too long.
Pelle Krabbenhoeft:But we are checking for fees and everything, so, there's definitely no clean answer to that.
Jihoon:Right, so are there… are there transactions that are not included forever, and they would… I mean.
Jihoon:I'm talking about those transactions that are not included.
Jihoon:In this chart, like, those who are part of this, 6-8%. So, is it correct to understand that
Jihoon:Those transactions are either included, And after some delay, Or… not delayed forever?
Jihoon:For whatever reason.
Pelle Krabbenhoeft:Yeah, the vast majority… Lock Native did some… they had a nice chart of this, but it's a very…
Pelle Krabbenhoeft:And they also have a stale tag on the data, but none of the transactions I could find were, like, explicitly censored. They either made it in, or they were in the mempool too long and eventually became invalid.
Jihoon:Right, right.
Pelle Krabbenhoeft:Yeah.
Pelle Krabbenhoeft:Sorry.
Jihoon:And… yeah.
Pelle Krabbenhoeft:Yeah, do you also…
Jihoon:have…
Pelle Krabbenhoeft:Yeah, go ahead, sir.
Jihoon:Yeah, so, yeah, I just wanted to ask you about, like, do you have, like, rough data
Jihoon:Indicate that, Like, on average, like, they get included after, like.
Jihoon:For example, like, 6 sulas, for example.
Jihoon:did I… Yeah, so… My question is, like.
Jihoon:If they are included after a certain delay, did they get included, like… I don't know, like, maybe…
Jihoon:Like, 6 or, like, 10 slots, or, like, do you also have any insight on that delay?
Jihoon:Maybe you're on mute?
Pelle Krabbenhoeft:Yeah, my bad. Yeah, I did some per limit.
Pelle Krabbenhoeft:I did some, like, smaller queries when I was looking into it, and I didn't see anything alarming. Most transactions, like, 98% or so, maybe even higher, were included in the next 5 or so blocks, or became invalid.
Pelle Krabbenhoeft:So, that's my best answer for that.
Pelle Krabbenhoeft:But I didn't see anything in those investigations that led me to look further into it.
Jihoon:Right.
Jihoon:Maybe it is also worth to see, like, If there is any common… Characteristic of those transactions, like.
Jihoon:Maybe those transactions are the ones who… touch Tornado Cache, for example.
Jihoon:Yeah, it's just an idea.
Pelle Krabbenhoeft:Okay, yeah. And just another note on this, too.
Pelle Krabbenhoeft:It would be a lot better for, like, further expe…
Pelle Krabbenhoeft:data collecting and such would probably do a sidecar method, but you'd probably want nodes across the world, just because it's… it's tricky to figure out if something is actually valid without this state. You can do some stuff and
Pelle Krabbenhoeft:If a transaction makes it into the mempool, it's supposed to be valid in most cases, but it's definitely…
Pelle Krabbenhoeft:A lot of this stuff is not clean-cut to figure out if it's actually being censored based on the historical data.
Jihoon:Got it, makes sense.
Justin Florentine (Besu):Do you need access to an archive node?
Pelle Krabbenhoeft:Yeah, that could be useful.
Pelle Krabbenhoeft:Yeah.
Pelle Krabbenhoeft:Yeah, if anyone has…
Pelle Krabbenhoeft:Any ideas of other stuff to look into, or resources, yeah, feel free to reach out to me.
Jihoon:Do we get a different dataset if we have an access to some archive node?
Pelle Krabbenhoeft:I haven't looked into…
Pelle Krabbenhoeft:I looked into it a little bit with my… oh no, not a… or…
Pelle Krabbenhoeft:running a node and just following the chain node archive node. I haven't done any work with archive node, sorry. It would be pretty simple to just follow the chain, and but if anyone has any
Pelle Krabbenhoeft:experience working with archive nodes. I'm not quite sure.
Jihoon:Okay, I can, I can take a look into it. I mean, I can ask around, like.
Jihoon:If I can provide you any access to some archive node.
Jihoon:I mean, if you are up for it.
Pelle Krabbenhoeft:Yeah, that'd be great.
Jihoon:Yeah, okay.
Jihoon:about, I mean, when it comes to same-slot, enclosure list.
Jihoon:post a PBS.
Jihoon:It's still an idea, but there's an idea… To make… Balls.
Jihoon:Be propagated separately to payload.
Jihoon:And set ball's deadline much earlier than payload deadline, such as 6 seconds into the slot.
Jihoon:And… With this, we can allow… Intellectualist committee members to…
Jihoon:Update their state earlier, and so that they can build Fresh IELTs.
Jihoon:Even when they receive payload late.
Jihoon:And don't have enough time to… Executed… To have the latest state.
Jihoon:So, I think… we would be possible to have same-slot IELTS even after EPBS.
Pelle Krabbenhoeft:Sounds good.
Jihoon:Alright, yeah, so another good thing is, I think.
Jihoon:IL building is not specified, and it's not part of consensus.
Jihoon:So once we have FOCIL on mainnet, maybe we can also analyze real-world data to fine-tune IL-building mechanism.
Jihoon:For instance.
Jihoon:So in this simulation, currently censorship window is set to test 12 seconds, and, you know, this is because it's a,
Jihoon:IATs are hypothetical, but…
Jihoon:It'd be really nice if he can have, if he, if he can have this effort.
Jihoon:be continued after we ship FOCIL, then it will be useful for people who are interested in
Jihoon:you know, elaborating their IL billing.
Jihoon:algorithms, whether it be researchers or client teams. So, yeah, it's great work.
Pelle Krabbenhoeft:Yeah, or if there's any MEV potential for your… I can't think of anything off the top of my head, but I wouldn't be surprised.
Jihoon:Yep.
Jihoon:Yeah, makes sense.
Jihoon:Alright, cool, thanks. Do we have any more updates?
Jihoon:Yes, Katya?
Katya:Hello, everyone. Not an update, but I have a question. Do we currently have a single source of truth, like the Cortosis config and,
Katya:The client's images, whenever they're ready.
Katya:Just to quickly spin up.
Jihoon:We still have some Kurtosis config for… On top of Fallu.
Jihoon:But I'm not sure if it's still working.
Jihoon:And… Nowadays, client teams are working on FOCIL on top of Amsterdam.
Jihoon:I don't… yeah, we don't have cartosis config for post-Chall Amsterdam.
Katya:Okay, got it. Maybe later in the channel.
Jihoon:Yep.
Jihoon:For sure.
Jihoon:Any questions or updates?
Jihoon:Alright, I think we can go ahead and…
Jihoon:wrap up this call. Thanks for joining, and hope you guys have a…
Jihoon:Great rest of your day, and…
Jihoon:Let's catch up in two weeks. Thank you.
Katya:Goodbye.
fdifabio:Excuse me.
Jihoon:Love it.
Chat Logs
00:07:20
Justin Florentine (Besu):https://github.com/jflo/besu/pull/120
00:12:53
Jihoon:@Mehdi Aouadi Could you share a link to your work, if it's public?
00:17:43
Pelle:https://hackmd.io/@pellekrab/S1YPhS9OWl
00:19:41
Mehdi Aouadi:Replying to "@Mehdi Aouadi Could ..."
It’s already merged in our master branch already. Part of this PR: https://github.com/Consensys/teku/pull/10398 where we added all the Heze boilerplate (includes InclusionList container for example). So we will be merging FOCIL into master prgressively in the future
Summary
8 highlights
· 2 action itemsExperimental
Summary
8 highlights · 2 action itemsExperimentaldevelopment updates
testing progress
- EL and CL FOCIL testing initiated; coordinating with Steel Team on reference tests00:05:09
research updates
- Mock IL analysis: 92% redundancy for top-fee strategy; 85-89% for censorship-only00:11:25
- Redundancy increases 2% per slot delay; minimal change after second slot00:15:16
- Hybrid IL building approach recommended; zero-slot delay strongly preferred00:16:13
- Same-slot ILs post-EPBS possible via early blob propagation (6s deadline)00:25:52