Parithosh Jayanthi:Okay, let's get started. First on the agenda item is, how we're gonna structure the call today.
AllCoreDevs - Testing #074 — BAL Breakout
AllCoreDevs - Testing #074 — ePBS Breakout
Transcript
Parithosh Jayanthi:So we're gonna do our usual updates, as well as going through any immediate blockers, as long as they're shared blockers. Once that's done, we're gonna experiment with breakout rooms today, so we're going to split the call into two breakout rooms, block-level access lists and ePBS.
Parithosh Jayanthi:I'm choosing the option to self-select, so each person can choose which breakout room they're gonna go to.
Parithosh Jayanthi:This might be exclusionary for some people who do want to be in both breakout rooms.
Parithosh Jayanthi:The rooms themselves won't be recorded, so for now, we're going to have to just experiment with this approach, and in the future, we can still see how to merge it better. The breakout room approach is also
Parithosh Jayanthi:Temporary, as in we're not fixing these as the two breakout rooms, but instead, on a week-by-week basis, we can decide what are the most valuable breakout rooms to choose, and who's going to lead them.
Parithosh Jayanthi:That being said, let's, start off with the usual bits. Devnet updates for block-level access lists, do we have… sorry, for blob-devnet-0, do we have Barnabas?
Barnabas:Yeah, blob-devnet-0 has been seen some issues in the past week. I'm not sure if we have anyone from Prysm here that could, speak up regarding the Prysm bugs that,
Barnabas:For the… regarding the partial cell stuff.
Barnabas:Other than that, the spec is finally…
Barnabas:Pretty much firms, so everyone should be able to implement,
Barnabas:Everything that they need for partial size stuff.
Barnabas:If you have updated your partial self, self,
Barnabas:logic, then please let me know, and I can, add you to develop the list on blob-devnet-0.
Parithosh Jayanthi:Is there a rollout plan for this one? I know that there was some talk about deploying it on testnets.
Barnabas:Yeah, so ideally, we can start deploying it on testnets once, the clients feel very optimistic about it.
Barnabas:Right now, we… the alternative would be to launch a very big network, but, realistically, we're not gonna be able to see,
Barnabas:The network effect of… Of this change, unless we do, like, another 1,000-node network or something like that.
Barnabas:So, I think, what, we were thinking is a few validators will, have it enabled on,
Barnabas:on Sepolia, and on Hudi, and then on Hudi, we can investigate further if… in case we do see some kind of clustering. And, once we are confident, then,
Barnabas:Clients can decide to, have it, default enabled in their, in their logic.
Parithosh Jayanthi:Okay, sounds good. Next on the list is block-level access list, devnet 2, as well as, any blockers for benchmarking. Either Stefan or Tuni, whoever wants to go?
Stefan Starflinger:Yeah, I can start quickly on just a general update.
Stefan Starflinger:So, we finally have reached pretty much 100% participation on devnet 2. Erigon has implemented and fixed the issue today regarding proposals, so not… a few blocks get proposed, but most blocks now get proposed by all the clients.
Stefan Starflinger:There are still some small improvements that can be made in general. For example, I reported one issue to Besu, where sometimes when it falls behind syncing, for some reason, it has a little bit of trouble catching up.
Stefan Starflinger:But otherwise, I think we are at a good place with devnet 2 at the moment, which is great.
Stefan Starflinger:But just, the next thing would be the benchmarking, and it would be great to get an update on how far clients are that haven't added the benchmarking flags.
Stefan Starflinger:And if any clients have maybe also iterated on any of the benchmarking code that they have, maybe we could get an update from the clients that don't have the flags yet, Nethermind.
Stefan Starflinger:Or Erigon, or Reth.
Stefan Starflinger:That would be great. Maybe start with Nethermind.
Marc:Yeah, there's good progress on the parallel execution. It seems to be running, like, pretty stable in the devnet, like, it worked for about 12 hours, but then it broke, so… I'm just gonna, investigate that a bit more, and then hopefully we can get that merged soon.
Marc:And I think Alexi's still investigating on the, parallel, state, ex…
Marc:Sorry, a parallel, state loading.
Marc:Yeah.
Stefan Starflinger:Yeah, I mean, we can also do that in the breakout call, would also make sense.
Stefan Starflinger:And what Barnabas said.
Stefan Starflinger:But I think that's then pretty much it, unless someone else has something regarding devnet 2.
Parithosh Jayanthi:Yeah, I think we'd structure it such that general updates can happen in the main call, but if there's anything specific to dive deep into, then that can happen in the breakout rooms.
Parithosh Jayanthi:So yeah, it would still be interesting to get an update from, from the other ELs as to where they are.
Stefan Starflinger:Then can we continue with Reth quickly?
emma:Hey, I just joined… late. What are we discussing? Sorry.
Stefan Starflinger:Just the flags on the benchmarking, so the flag to toggle the prefetching, the flag to enable all the optimizations, and the flag to disable all the optimization regarding parallel processing and prefetching state and state root calculation.
emma:I think we have all three flags on our devnet 2 and devnet 3.
emma:Brunch.
Stefan Starflinger:Okay, could you, could you share all of the configuration? Because so far, I think I only have one of them.
emma:Yeah, I'll ping it through in the chat.
emma:No worries.
Stefan Starflinger:Thank you, and Erigon would be great, if you have any update on that.
Andrew Ashikhmin:Right, so, yeah, we are working on,
Andrew Ashikhmin:We are finishing parallel execution, I'll check about the flags,
Andrew Ashikhmin:So… I see that, where are they spec'd? On the bal-devnet-3 page. I see some flags for Basil…
Stefan Starflinger:Yeah, both. bal-devnet-2 and bal-devnet-3, but bal-devnet-3 is not ready yet, so it's good to have it on 2 right now, so that we can continue the progress on the gas repricing.
Andrew Ashikhmin:Alright, okay, yeah, I'll check the implementation status, and if they are not implemented, I'll try to add them.
Andrew Ashikhmin:soonish.
Stefan Starflinger:Okay, super.
Stefan Starflinger:I think that's it for devnet 2, otherwise we can move to devnet 3, if there's nothing.
Parithosh Jayanthi:Let's move to that 3, then. What's the blocker or timeline there? I know that Tony shared an agenda, so maybe besides those discussion points, if there's any blockers?
Stefan Starflinger:Yeah, maybe I can give a quick overview. I've been doing some testing, regarding devnet 3 and,
Stefan Starflinger:In a single testnet, most of the clients are doing pretty well with just normal transactions, but as soon as I start with EVM fuzzing scenario, it breaks apart pretty quickly, maybe…
Stefan Starflinger:Some updates from the client teams and testing would be great.
Parithosh Jayanthi:Have you already shared all the issues with the client teams?
Parithosh Jayanthi:Are there anything in specific you want to dig into?
Stefan Starflinger:It's quite general still, so…
Stefan Starflinger:There's nothing in particular right now, that I would want to share at the moment.
Stefan Starflinger:Okay, so…
Parithosh Jayanthi:So it sounds like we're still going to continue fuzzing locally. We'll probably have some minor spec change before we do a devnet launch, right?
Stefan Starflinger:Yeah, we will…
Stefan Starflinger:I think, talk about it more in the breakout call. I think it's… yeah, I think we should…
Stefan Starflinger:Not go into too much detail here now.
Parithosh Jayanthi:Okay, yeah, drawback?
Dragan Rakita:I just mentioned that Friday I found some existing inspectors that needs to be fixed.
Dragan Rakita:And I think the test feature was just created a few times previous week, so I don't think that anybody's passing them.
Dragan Rakita:So there is still, like, work that needs to be done.
Parithosh Jayanthi:Okay, cool, thank you. Anything else for bal-devnet-3?
Parithosh Jayanthi:Okay, I did want to announce that, Justin's made a release on the consensus specs for V170Alpha 3.
Parithosh Jayanthi:And to continue that point, do we want to get some updates on epbs-devnet-0, as well as where Alpha 3 will be used?
Barnabas:Yeah, I can give a quick update. So, we currently have Prysm, Lighthouse, and also working, correctly on the devnet.
Barnabas:We have Nimbus proposing blocks here and there. It's not completely working yet.
Barnabas:And, Teku has not been able to sync or propose at all yet.
Barnabas:local groups that also weren't able to work with Teku.
Barnabas:we added Checkpoint Sync… we added Lodestar to the Checkpoint Sync instance last week, and that seemed to have broken some stuff, so we now reverted back into only using Prysm.
Barnabas:And,
Barnabas:Yeah, I tested exits last week, and deposits, deposits worked, exits do not work yet. I can run another round,
Barnabas:Later today, to see if there has been any changes that,
Barnabas:Would make sure that we can do the exits as well.
Parithosh Jayanthi:Sounds good. To carry on there, I also posted a message about disabling block production and then re-enabling it, and that's also leading to some
Parithosh Jayanthi:Some issues with the network, healing afterwards, but yeah, I guess there's still a lot of fixes for epbs-devnet-0 before we go to devnet 1, right? And I guess Alpha 3 will be what we use for devnet 1?
Barnabas:Yes, most likely.
Potuz:Don't we want the PTC… I mean, we've discussed this in the breakout, but I think we want the PTC loop behind, or look ahead, one way of it, for the next advent.
Parithosh Jayanthi:Okay, so there's probably going to be another release, before that. Okay.
Parithosh Jayanthi:Cool. The next point on the agenda was, update from the gas repricing, effort. Do we have any more insight into numbers, where we are in terms of getting them, and when we want to sequence these into devnets?
Parithosh Jayanthi:Okay, I think the relevant people might just not be here today. Okay, we can do that on async.
Parithosh Jayanthi:There was maybe one other agenda item before we switch to breakout rooms. The item was from Spencer talking about, SFI and EIPs.
Parithosh Jayanthi:Spencer, do you want to first bring up the point, and then I think Tony and I had some follow-up points over there?
spencer:Yeah, sure. I guess, just some context, so… And personally, I would, like.
spencer:us to SFI EIPs a bit faster,
spencer:Kind of aligning more with the network upgrades.
spencer:Including EIP, so EIP7723.
spencer:And selfishly, on the steel side on eels, we only include
spencer:EIPs, or we only merge EIPs into our main…
spencer:Our main branch, Forks Amsterdam, wants to recified.
spencer:it's a process that we like to keep, but we can also change that. But, I guess right now.
spencer:For Amsterdam, we've been having, like, a lot of, a lot of issues with, consistently having to have all these, conflicts between EIPs when creating the devnet branches, because we don't…
spencer:yet have these EIPs specified, so we can't merge them in earlier into our works Amsterdam.
spencer:branch. So that's the kind of the selfish side.
spencer:From steel.
spencer:I guess the other benefit of SFI earlier is the downstream effects it would have.
spencer:For the community.
spencer:And… but yeah, I guess,
spencer:the main reason I brought this up is because…
spencer:In the network upgrade EIP, 7723, we staked
spencer:When client teams agree to implement a core EIP in the next upgrade devnet, the EIP should move to
spencer:The SFI stage and the upgrade EIP should be updated to reflect this. We also state, SFI signals that implementation and testing work are underway.
spencer:the EIP should be included in the network upgrades if no issues arise. Now, I know, like, the key word here is should, doesn't say must.
spencer:But my gut feeling is that should apply to the majority of EIPs, and then the must should, I guess, negate that. So, my feeling is that maybe, like, 70-80% of the EIPs should
spencer:She'd follow this,
spencer:And I, yeah, I do… I do… I don't think it's the best idea to just say the EIP has been added to a devnet, therefore we must SFI it. I think we could maybe change the terminology a bit to
spencer:state that, EIPs can be SFI'd once it's successful in a devnet. So, for example, all the EIPs that have been successful on devnet 2,
spencer:I feel like she'd be SFI'd by now,
spencer:But yeah, I guess I just wanted to bring this up as a general process question.
Parithosh Jayanthi:Yeah, maybe, sharing… I already posted my thoughts on the, on the topic in the issue as well, but, thanks for the context, I think that helps a ton. Probably…
Parithosh Jayanthi:The biggest fear for me is that we SFI something. It has a relation with another EIP that we only haven't included in a devnet.
Parithosh Jayanthi:Because of differing complexity, and then…
Parithosh Jayanthi:we have to kind of backtrack the SFI-ing, if we SFI too early. There's probably a middle path over there where if an EIP doesn't have any interactions with other EIPs, or any foreseeable interactions with other EIPs, as well as is deemed
Parithosh Jayanthi:well implemented and testing coverage is good, then we can SSI it. But at least from my perspective, I think this makes more sense to do one ACD rather than ACDT until ACD explicitly says ACDT can take a call on this.
Parithosh Jayanthi:Yeah, that was, I guess, high-level thoughts, but…
Parithosh Jayanthi:yeah, open to other opinions or decisions. Do we want to SFI stuff on this call, or do we want to defer that to ACD?
Parithosh Jayanthi:Yeah, Tony?
Toni Wahrstätter:Yeah, I agree with you. I think we should do that on ACD.
Toni Wahrstätter:And…
Toni Wahrstätter:Yeah, so far, there haven't been many definites, so not many EAPs were already on devnets, but maybe this week on ACD, we can already take a decision on a few of them that were already on a devnet.
Toni Wahrstätter:Two, and that don't interact with,
Toni Wahrstätter:many other EIPs, and we can already SFI them.
Toni Wahrstätter:Especially, like Spencer said, to give the community, like, more certainty about what is going to… Ship, or whatnot.
Parithosh Jayanthi:Yeah, that makes sense. I think the list that Spencer shared already makes sense. I think he also has a PR to update them, right?
Parithosh Jayanthi:Yes, he does.
Parithosh Jayanthi:So, in general, is anyone opposed to Spencer presenting this list on Thursday, and then getting ACD's approval for SFI?
spencer:And note, this list contains all the EIPs in devnet 3, so… I think we might need to filter it down.
Parithosh Jayanthi:Sorry, okay. I guess you can post a list on ETH R&D and then solicit feedback there?
spencer:Sure, sure, sounds good.
Toni Wahrstätter:And we could also… we could start with only the EIPs from devnet 2,
Toni Wahrstätter:Because we don't even know if they work yet, we haven't launched devnet 3 yet, so maybe it would make sense to do that, like…
Toni Wahrstätter:One after another.
Toni Wahrstätter:But I have no strong opinion on that.
spencer:I guess, like, the main reason I brought this up was not necessarily to SFI the EIPs, but, like, should we change what we state in the network upgrade inclusion EIP to, kind of.
spencer:Align more with, a slightly different process of, like, okay, the…
spencer:the EIPs in the previous devnet were stable, we should SFI them
spencer:Assuming that, yeah, there's no future conflicts with other EIPs that we may see in the future. I guess, like, that's…
spencer:That was kind of the main question. I don't want to take up too much more time here.
Parithosh Jayanthi:Yeah, maybe vulnerables for last points, and then we can move on.
Barnabas:Yeah, just one point that, for example, EIP-7637, the state creation gas cost increase, the IP, is not really finalized yet, so I would actually make sure that
Barnabas:We're not gonna SFI a half-complete EIP, because, yeah, right now it's just using some hard-coded version for devnet 3.
Barnabas:So, I would not feel comfortable specifying that EIP in that… its current form. So, unless we're actually gonna be doing the seed creation gas cost increase properly, I would keep that away from the list, even if we do end up including it in the mystery.
Parithosh Jayanthi:Yeah, for sure. I think the same should apply for us putting in a change that requires a repricing later on. I think we should SSI EIPs
Parithosh Jayanthi:That we just see having very few interactions, and has already been tested adequately, and has implementation that's already in the final stage.
Parithosh Jayanthi:Okay, cool.
Parithosh Jayanthi:That's, I think, everything on the agenda. Are there any other topics people wanted to bring up before we head to breakout rooms?
Parithosh Jayanthi:Okay, if that's the case, I will… Create the rooms now.
Parithosh Jayanthi:Please note that the rooms won't be recorded, at least not right now. There also won't be a transcript, so if you're participating in one of the two rooms and make, make notes, please do post them on EthR&D so that we can, collect them later on.
Parithosh Jayanthi:And yes.
Parithosh Jayanthi:You should be able to choose a breakout room now.
Toni Wahrstätter:Is it only me, or is this…
Toni Wahrstätter:button I should click, like, outside my window, I can't click it.
Stefan Starflinger:I can't even see one.
Toni Wahrstätter:It's amazing.
Karim T. (matkt):students.
Parithosh Jayanthi:Is no one able to see the results?
Milos:And it should give a pop-up.
Parithosh Jayanthi:Yeah.
Parithosh Jayanthi:There's people joining the rooms.
Stefan Starflinger:Where should we see them?
Pooja Ranjan:Click on More, and then you will see breakout room in progress.
Parithosh Jayanthi:Thanks. Okay, let me just manually assign you, if I can, Rakan.
Dragan Rakita:Appreciate it.
Dragan Rakita:Shit, no.
Parithosh Jayanthi:Yep. Thanks, everyone, for coming. Bye.
Toni Wahrstätter:Thank you. Bye-bye.
Louis:Thank you.
Transcript
Transcript pending
Transcripts for breakouts are not yet processed
Transcript
Transcript pending
Transcripts for breakouts are not yet processed
Chat Logs
00:08:20
Barnabas:wouldn’t this be better to do in the breakout room?
00:10:34
Stefan Starflinger:https://notes.ethereum.org/@ethpandaops/bal-devnet-2
00:11:23
FLCL:Are new tests planned?
00:12:16
spencer:Replying to "Are new tests plan..."
We'll have another release but it wont change much. Just a single test case.
00:12:22
jochem-brouwer:If fuzzing and then network breaks then it means testing coverage is missing?
00:14:00
spencer:Replying to "Are new tests plan..."
This is the test case mentioned by dragan: `tests/amsterdam/eip7954_increase_max_contract_size/test_max_initcode_size.py::test_max_initcode_size_gas_metering_via_create[fork_Amsterdam-create_opcode_CREATE-state_test-short_one_gas]`
00:14:34
Toni Wahrstätter:To add to jochem's qu., any specific EIPs that break?
00:14:39
Stefan Starflinger:Replying to "If fuzzing and then ..."
yes and no, not all tests are being passed yet
00:14:57
terence:Prysm is not ready yet, we need to merge a PR, give us 2 days
00:14:59
terence:For exit
00:15:04
felipe:Replying to "If fuzzing and the..."
https://hive.ethpandaops.io/ should we add a run for bal-devnet-3 here to track client status?
00:15:38
terence:Yes, can we do alpha4
00:15:53
nflaig:did some updates to our checkpoint cache serving @Barnabas should be fine to add to checkpoint-z again
00:16:38
Stefan Starflinger:Replying to "To add to jochem's q..."
8037 gas accounting causing BAL hash missmatch
00:17:56
Barnabas:I’d propose SFI-ing:
7708
7778
7843
8024
00:20:07
Stefan Starflinger:Replying to "To add to jochem's q..."
For example: CREATE, CALL-with-value, or precompile calls. Nimbus records different storage/account accesses than geth for these operations. Simple eoatx passes. Only evm-fuzz triggers the divergence.
00:21:06
nixo:i think if ACDE/C is responsible, we can be a bit more proactive there
00:21:36
FLCL:Replying to "Are new tests planne..."
Ty!
00:21:41
Stefan Starflinger:Replying to "To add to jochem's q..."
Erigon — EIP-8037 Block Gas Overflow (Single-Dimensional Pool)
00:22:33
Barnabas:I’d do SFI devnet 3 maybe a week after its been stable
00:22:34
Barnabas:also I
00:23:48
Stefan Starflinger:Replying to "To add to jochem's q..."
reth: Bug likely in how CREATE/CREATE2 tracks code changes or how storage reads vs writes are classified during contract deployment. The hash is computed by alloy compute_block_access_list_hash() which is shared, so the divergence is in the BAL
00:24:25
spencer:Will aim to add draft PR to update 7723 aligning on what we have spoke about. Few interactions and in passing devnet etc
00:24:28
Barnabas:"Fully baked”
00:24:52
Barnabas:Replying to "I’d propose SFI-ing:..."
7773 you mean?
00:25:14
Potuz:how?
00:25:17
spencer:Replying to "I’d propose SFI-i..."
both :)
00:25:21
terence:Too high tech for me
00:25:22
wolovim:I think hosts can record locally - plz do that if you have the UI for it 🙂
00:25:31
felix (eest):frick epbs all my homies join bal
00:25:50
Dragan Rakita:Room not detected, what is room code?
00:26:03
Louis:Replying to "how?"
There is a “More” button, select it and there are two options
00:26:11
Kevaundray Wedderburn:More > Breakout rooms
00:26:19
Dragan Rakita:I see Transfer to room, but it requires room code
00:26:34
Enrico Del Fante (tbenr):More?
BAL Chat
15:29:08
FLCL:state reads are sorted right? and not by access?
15:29:41
jochem-brouwer:Antwoord verzenden naar "state reads are so..."
sorted lexicographic, so not by time of access
15:30:10
FLCL:why access based ordering would be worse?
15:30:41
Parithosh Jayanthi:https://github.com/ethpandaops/observoor
15:30:42
Stefan Starflinger:pari was talking about this project: https://github.com/ethpandaops/observoor
15:30:45
Parithosh Jayanthi:For those interested
15:30:47
Stefan Starflinger:damn
15:30:51
Stefan Starflinger:you were faster
15:32:23
Parithosh Jayanthi:github.com/ethereum/client-traces
15:33:14
Stefan Starflinger:the bal specs https://github.com/ethereum/client-traces/blob/main/bal-traces.md feel free to update them
15:33:28
FLCL:it's hard to speed up db access even if you know keys in advance and read then in order of index. But if we could know access order, we could load it for the next transaction and so on. My concern is if execution parallelisation can do a lot if most of the time is spend on state access
15:38:28
Karim T. (matkt):When will be able to run a shadow fork of mainnet
15:39:50
Łukasz Rozmej:Replying to "state reads are sort..."
you could potentially process transactions in lexicological access order, we can do them in any order with BALs
15:44:38
jochem-brouwer:Ah right good point, yes this would indeed be the releases of benchmarks
15:45:25
Louis:I think the difference is:
If we use benchmark release via EELS, it would be limited to stateless benchmark with compute scenario
If we are generating payloads via gas-benchmarks, it could support stateful benchmark.
Is this correct? @Kamil Chodoła
15:46:24
Parithosh Jayanthi:This will all change with the testing endpoint right?
15:46:51
Parithosh Jayanthi:Cause then eels can just specify txs and outcomes, the rest benchmarkoor/hive can do
15:47:38
Kamil Chodoła:Yeah this is already being specd on hwo to make EELS stateful with testing endpoint.
But until then we have the old battletested gas-benchmark mechanism
15:49:50
FLCL:Depending on what is goal. Do we need faster execution?
15:53:57
Łukasz Rozmej:In terms of BAL size this would be just a simple list of transaction indexes for optimal execution order
15:54:45
Parithosh Jayanthi:Last 5 mins btw
15:54:52
Parithosh Jayanthi:cc @Toni Wahrstätter
15:55:06
Stefan Starflinger:I thought the benchmarks are executed against the state of either mainnet or bloatnet not against the actual shadowfork?
15:55:20
Stefan Starflinger:otherwise we have the mempool?
15:55:32
Łukasz Rozmej:Replying to "In terms of BAL size..."
so basically if we added this we can focusing on benchmarking best cases, as worst cases and best cases would be more or less the same
15:56:28
jochem-brouwer:Antwoord verzenden naar "I thought the benc..."
It's not a shadowfork, it runs on top of bloatnet which is stalled, so no networking
15:56:59
Stefan Starflinger:Replying to "I thought the benchm..."
stalled?
15:57:12
jochem-brouwer:Antwoord verzenden naar "I thought the benc..."
Yes all benchs run on top of the snapshot block
15:57:18
jochem-brouwer:Antwoord verzenden naar "I thought the benc..."
So there is no networking
15:57:52
Stefan Starflinger:Replying to "I thought the benchm..."
So they run without a shadow fork?
15:57:55
Stefan Starflinger:Replying to "I thought the benchm..."
just with a snapshot
15:58:16
jochem-brouwer:Antwoord verzenden naar "I thought the benc..."
Yes, sorry for using the term shadowfork because it is confusing. In my mind shadowfork = fork of network, but in this case it is "just" a snapshot indeed
16:00:31
Łukasz Rozmej:can you write eip number?
16:00:45
jochem-brouwer:https://eips.ethereum.org/EIPS/eip-8189
ePBS Chat
Breakout chat log unavailable.
Summary
11 highlights
· 3 decisions · 1 action itemsExperimental
Summary
11 highlights · 3 decisions · 1 action itemsExperimentalblob devnet updates
bal devnet progress
epbs status
Decisions
Action Items
- Spencer: Draft PR to update EIP-7723 SFI criteria: stable devnet + minimal interactions00:24:25