Parithosh Jayanthi:So, the first item on the agenda today, is Fusaka updates. So, do we have Barnabas on the call for DevNet updates?
Transcript
Barnabas:Yes, we did a non-finality event on WN3 throughout the weekend, and this morning I have restarted all the machines that were producing bell blocks, and they're slowly syncing up back to head.
Barnabas:Currently, we see a lower participation than before, which is quite interesting.
Barnabas:…
Barnabas:Right now, the major nodes that are offline still are, I guess, another mined ones, because those were pretty, in the images before. I have, here their database, and they should be able to sync back to HUD. Currently, we see about 36% participation on the network, though.
Barnabas:Which is rather low, and it's a bit surprising that, about 20% of extra nodes have decided to
Barnabas:Also, stop thinking.
Parithosh Jayanthi:So, does this mean that during the non-finality period, we had about 50-ish percent participation, and then once you started all of the nodes back on, we reduced participation again?
Barnabas:Yes.
Parithosh Jayanthi:Okay, that sounds like they're not able to keep up with the sync load, right? Or is there some other prevailing theory?
Barnabas:Mmm, yeah, that's exactly what I'm thinking.
Barnabas:We have rate limiting enabled on all the machines, and I think, some of them are hitting that.
Barnabas:Especially for notes.
Parithosh Jayanthi:What's very interesting is, none of the R machines are out of sync.
Barnabas:So, basically, right now, the network is, held up by mostly our machines, so it might also be some cloud-related issue.
Parithosh Jayanthi:And at least according to logs, how does it look like it's going? Are the nodes progressing, or are they just struggling to find healthy peels?
Barnabas:So, there's a bunch of nodes that are progressing still. Most of them have caught up, especially in the ES side.
Barnabas:there's a few nodes, like Lighthouse GAF1 that have not able to get started thinking, and I think it's still the same syncing bug that Lighthouse previously reported. I'm not sure if anyone from Lighthouse is around.
pawan:Yeah, I'm here. Yeah, I think it's the… I'm not sure if you're using the…
pawan:other brands that I gave you on Friday, but, like, that… we found, like, a couple of issues over the weekend on that one as well, which was related to…
pawan:like, a few bugs on our unstable branch, so we have a few fixes on unstable that we are going to merge into that branch today, and again, test if
pawan:… stuff should work as expected, and yeah, I'll keep you in the loop.
pawan:To… as to which branch to use.
Barnabas:Yeah, okay. So, right now we have the blocks and columns on Lighthouse Gas 1, which is… which has stopped syncing, and we have Unstable on everything else.
Barnabas:But you can update it to whatever you want.
Barnabas:Yeah, so slowly the CLs are, forward syncing, but because we had a quite long non-finity, and because the forward sync is very, very slow on every single CL, it might take another few hours before we see any improvement.
Barnabas:I… I would actually like to keep it as is right now, not really touch anything, unless we see something definitely wrong, and see if we can recover back to finally in, like, a day or two.
Parithosh Jayanthi:Yeah, that sounds like a good plan, and yeah, if there's any fixed images, then we should roll those out, but otherwise, I agree, I think we should…
Parithosh Jayanthi:monitor how the nodes are going and take that data into account. What blob count are we at right now?
Barnabas:We are on 20, I think?
Barnabas:whatever BPO5 was, it's max blobs per block 20.
Barnabas:But Spammer had a bug, and we are not spamming block transactions right now, but because debugging.
Parithosh Jayanthi:Yeah, but during the non-finality event, were we able to hit 20 blobs?
Barnabas:I mean, we are still at the non-finity event.
Barnabas:Yeah.
Parithosh Jayanthi:I meant the core non-finalty went over the weekend, at least.
Barnabas:Yeah, we can check that, but I think so.
Parithosh Jayanthi:Okay, cool. Because those are the blobs that everyone would need… would sort of be struggling to fetch columns for.
Barnabas:Yeah, so, like, even an hour ago, we had, consistent 20 blocks in the blocks that were proposed.
Parithosh Jayanthi:Okay, that's a really nice test. Thank you.
Parithosh Jayanthi:Anything else you wanted to add on that, or anything the client teams wanna talk about? Yeah, Enrico?
Enrico Del Fante (tbenr):Yeah, I just want to mention that,
Enrico Del Fante (tbenr):Couple of hours ago, we've discovered a bug in our, state upgrade from Elektra to Fulu.
Enrico Del Fante (tbenr):And, I share the link of the PR that fixes this.
Enrico Del Fante (tbenr):It contains a couple of things, but this is the main thing. So, essentially, we missed, the…
Enrico Del Fante (tbenr):The, porting of the pending consolidation from the state of, from the later state to
Enrico Del Fante (tbenr):the Fulu State.
Enrico Del Fante (tbenr):So…
Enrico Del Fante (tbenr):I think that we never tested on any testnet that across the FARC, there was actually a consolidation pending.
Enrico Del Fante (tbenr):So we always started with an empty consolidation, set.
Enrico Del Fante (tbenr):So maybe there is something on the spec test that needs to be addressed, or…
Enrico Del Fante (tbenr):But seems, like, serious, so we fixed it there.
Parithosh Jayanthi:Yeah, this definitely sounds like a spec test, right?
Justin Traglia:Yeah, I can… I can look into this.
Enrico Del Fante (tbenr):Yeah, let me know if you need more context on that, but the link should be enough.
Parithosh Jayanthi:And I guess….
Barnabas:So we have….
Parithosh Jayanthi:make sure that we have at least one spec test for every one of these special requests.
Enrico Del Fante (tbenr):I was thinking, like, we should constantly… whenever there is something that moves from the status, from a state to another state, we have to always constantly testing it. Even previous introduced
Enrico Del Fante (tbenr):stuff. So not just testing what the new status is introduced in by the fork, but also all the regression possible from the previous forks.
Justin Traglia:So, we do have fork, like, at fork testing. I'm thinking of, like, deposit requests and consolidation requests, but I'm not sure if we have, like, pending consolidation.
Justin Traglia:At the fork, so that would be good for me to look into.
Enrico Del Fante (tbenr):Probably the missing thing.
Enrico Del Fante (tbenr):Yep.
Parithosh Jayanthi:Awesome. Thank you.
Parithosh Jayanthi:Any other updates for DevNet 3 or Spec?
Parithosh Jayanthi:And I'm assuming there'd be another spec release, right, Justin?
Justin Traglia:Yeah, those.
Justin Traglia:There'll be another spec release, probably.
Justin Traglia:I don't know, a week or so.
Barnabas:Is that gonna be an offer fix?
Justin Traglia:I was actually thinking of doing Beta Zero.
Justin Traglia:If this would be the one that lines up with the first testnet, … Release.
Barnabas:Is the idea that the EPBS and net zero is gonna be based on that, too?
Justin Traglia:Say it again?
Barnabas:Is… is, APB as then at zero also going to be based on top of that?
Justin Traglia:Oh, I don't know.
Justin Traglia:I mean, could be.
Parithosh Jayanthi:I guess it doesn't matter too much, because this is just adding tests, right? So, from a specification perspective, alpha 5 and beta 0 would be the same.
Justin Traglia:Yeah, it doesn't really matter much.
Parithosh Jayanthi:Yep.
Parithosh Jayanthi:Any other updates relating to DevNets? We do have the MEV workflow working? Is there anything to report over there? Anything we're investigating?
Bharath:Yeah, so, in terms of the workflow, yeah, last week, we were able to get the
Bharath:Private order flow thing, we, just kind of… we have… we're able to get a resolution with that, like…
Bharath:Basically on a high level, it looks like our builder was…
Bharath:only retaining, like, transactions for, like, 5 blocks by default. We somehow bumped that up to, like, 10,000 blocks, and things seem to generally be fine.
Bharath:I think the RBuilder folks also made that a configuration parameter, so…
Bharath:Future, we don't have to modify code for that.
Bharath:And yeah, the other good thing is, like, the RBuilder folks have launched their own builder, too, like, so we have, like, two builders submitting to the MethBoost Relay, which is nice, and generally, that's good testing.
Bharath:Yeah, outside that, there's one thing which, Fahil from the PBS Foundation is looking into, is that,
Bharath:So there's one issue, again, you know, it's probably worth… it's not a very serious issue, it's just that when we transition from Elektra to Fusaka, it… we are… I think there are cases where a block simulation fails with the relay. That's because I think our builder is including some…
Bharath:pre-EIP, pre, Fusaka transactions without cell proofs,
Bharath:This is basically an issue with the logic, like, it's filtering out blobs for certain transactions, but including the transaction… the IP4844 transactions. So, that is something, like, I think Fayel is looking into. It's not a really critical issue, because that
Bharath:The mempool, kind of slowly, like, drains itself out of, like.
Bharath:the, pre-Fusaka transactions without cell proofs.
Bharath:But yeah, I think File is looking to that. Maybe I'll also spend some time later on that. But outside of that, there's not much, with the MV workflow we release, like, builder specs. We merge, like, MevBoost to main. We're looking to merge MevBoost Relay to main, but we're waiting on, like, the attestant IOPRs, like.
Bharath:For GoBuilder Client.
Bharath:To get merged. I think somebody from Attestant.io has started looking into it and has raised some review comments, which I've addressed. So, I think…
Bharath:Yeah, we're looking to, like, merge everything to main, and there is that one issue which we'll… we'll see, like, how we can resolve it, or if that's even a problem at all, like, that's also not clear.
Parithosh Jayanthi:Okay, perfect. Thank you.
Parithosh Jayanthi:Besides this, what is the DevNet 5 plan? Do you… maybe you want to tell us what the blockers are there, Barnabas, and we can get an update from clients as to where they are?
Barnabas:Yeah, so the main blocker would be, trunk branches and, all major features merging today trunk branches. Ideally, we wanted to do that this week, but I think, giving an extra week would make the most sense for all, design teams.
Barnabas:And in this case, we would aim for, all major features merged into the trunk branches by this week, Friday, and then maybe we can do
Barnabas:Devnet launch on next week, Wednesday.
Parithosh Jayanthi:Do client teams want to chime in? Does this timeline work for you guys? Are we… do we have to change around dates? What is the… what's the general consensus?
pawan:Sorry, did you say next week, Wednesday?
Barnabas:Yeah, of course.
pawan:Yeah, that's… that seems good for us.
Parithosh Jayanthi:Yes, this is for DevNet 5.
Parithosh Jayanthi:Okay, it seems like most, if not all, clients have
Parithosh Jayanthi:agreed to that. If for any reason you guys feel like we're missing something, then…
Parithosh Jayanthi:Please reach out to us, and then we can still modify the timelines, but it takes a few days' worth of preparation to get these networks up, especially if it's over a couple thousand nodes.
Parithosh Jayanthi:So it would be nice if we know well in advance of time,
Parithosh Jayanthi:If we have to change our plans.
Parithosh Jayanthi:Okay, the next agenda item is, one from Alex. Do you want to talk about the temp check?
stokes:Yeah, sure. …
stokes:This ties in a bit with timing, for Sokka, and yeah, I guess one thing we could do here is couple the hoodie release with the mainnet release, and I just wanted to make sure that was okay with everyone.
stokes:Rather than have, like, a distinct mainnet release.
stokes:It would just help us pipeline things a bit, and compress.
stokes:Otherwise, what would just be some downtime?
stokes:Anyone have any issues with that?
stokes:like, we have precedent for having test next coupled, and I feel like we've done this before, where it's…
stokes:say a testnet and mainnet, but I just wanted to make sure.
pawan:what would be the gap between the actual hoodie folk time and mainnet folk time? Is that the same, or like.
stokes:Yeah, that's a good question. So…
stokes:I think the exact dates are gonna depend a bit on how, like, the next few weeks go with the DevNets and…
stokes:say, Sepulia and Holushki?
stokes:But yeah, I think I was thinking something…
stokes:Okay, so we have this invariant of having a month between the release and mainnet.
stokes:I think we could have hoodie somewhere in between that month.
stokes:So then, to answer your question, yeah, something like 2-3 weeks.
pawan:I think in that case, like… like you said, it would depend on how the other testnet folks go, like, if…
pawan:Yeah, if the other testnet folks go pretty smoothly, then I think we'd be a little more confident to have, like, a smaller gap, otherwise…
pawan:Yeah, we would maybe want to have a bigger gap.
stokes:Yeah. Okay.
Parithosh Jayanthi:Okay, yeah, I guess we're going to be discussing this more and more over the next couple of weeks and months, so… yeah, this was a September check.
Parithosh Jayanthi:Any other Fusaka topics?
Parithosh Jayanthi:Yeah, I think Roman's asking, do we have a tentative target date on when we decide testnet timestamps? Do we do this after DevNet 5, or do we want to do so, earlier than DevNet 5? What is our blocker here?
stokes:To me, it's mainly DevNet 5, so…
stokes:If DevNet5 goes well, we can also do this, like, BPO analysis. We wanted to do it with DevNet4.
stokes:That should give us numbers we feel comfortable with for mainnet, which we would then also use for all the testnets. So, assuming DevNet 5 goes well, then we can look to schedule Holushki and Sepolia.
stokes:I think pretty soon after that.
Parithosh Jayanthi:Okay, that sounds reasonable. I think we'll try and have the DevNet 5 analysis up as soon as possible after.
Parithosh Jayanthi:I think Sam's already working on a lot of the…
Parithosh Jayanthi:the Jupyter notebooks, etc, so that once the data is there, it should be quite straightforward to plug in the data and then get an analysis out.
Parithosh Jayanthi:Yeah, and we were also talking about reducing the amount of time between BPOs for DevNet 5, so it won't take a week-long testnet, but it might be a bit shorter.
Parithosh Jayanthi:But yeah, I'm… I think we'll release a full spec for DevNet 5 and so on over the course of the week.
Justin Traglia:Do you have an idea of what the time between BPO work should be? Like, 12 hours?
Parithosh Jayanthi:8 hours. I think about 12-ish hours, sounds reasonable, and then we do want to collect some data with MEV as well, so maybe 12 hours plus another few hours of MEV worth of data, MEV-style data, and then so on.
Justin Traglia:Sure. Yep, that sounds good.
Parithosh Jayanthi:Yeah, I think Casey's asking for potentially a written, DevNet and release timeline, but maybe that can be, for all core devs on ETH R&D, or somewhere where we can discuss it a bit more async?
stokes:Yeah, I can put something in the next few days somewhere, and definitely, yeah, we'll discuss by this week's ACD.
stokes:Well, yeah, let's discuss there. But I'll get something async before that.
Parithosh Jayanthi:Okay, thank you.
Parithosh Jayanthi:Any other Fusaka-related topics?
Parithosh Jayanthi:Were there any other bugs on the DevNet last spec clarifications we needed to get checked out?
Justin Traglia:There was a typo in the most recent release, one of the new configuration variable names. I posted a comment about it on Discord, but just know it'll be fixed in the next release.
Parithosh Jayanthi:Okay, perfect. Thank you.
Parithosh Jayanthi:Next topic on the agenda is, gas limit and bloatnet slash Performance DevNet updates.
Parithosh Jayanthi:So we have an update from the sync test. We're able to get sync tests running well on Geth as well as Bezu. Both of these clients also offer a flag similar to a required block, which has been very helpful on these smaller shadow folks in terms of finding peers.
Parithosh Jayanthi:The others are struggling to either find peers or maintain their peers, so it would be great if Nethermine, Wrath, and Aragon could provide us with a similar flag. We flagged this in the state floor channel as well, but if for some reason you're not on the channel, or you want to discuss this, then, …
Parithosh Jayanthi:yeah, please DM someone from the PandaOps team, and we'll put you in the right channels.
Parithosh Jayanthi:And is someone on the Bezu team here who wants to talk about 60 million gas update? I see that they had a comment on the PM repo about it.
Luis Pinto | Besu:Yeah, I can share some updates. So, we've noticed that we are highly impacted by the divisions, so Maude, Steve, and we are working on that. We've tried, …
Luis Pinto | Besu:couple things, to avoid, a big overhaul, but I think we are gonna have to, do the NUTS implementation from scratch in Java, avoiding all the libraries, so, yeah, we're working on it, right now.
Luis Pinto | Besu:And… I had some more updates from Simon.
Parithosh Jayanthi:Could you clarify which, library you're talking about?
Luis Pinto | Besu:The big integers, Java library.
Parithosh Jayanthi:Got it, okay.
Luis Pinto | Besu:Yep.
Luis Pinto | Besu:We… we have some kind of… we have a bit of, …
Luis Pinto | Besu:memory churn, object churn, because of the big integer, which we would like to avoid to… to make things faster. So we… we need to do, arithmetics directly on the… on the, on the bytes, or, or primitive arrays, basically.
Luis Pinto | Besu:N… Just give me a second, I have a… I had an update from Simon.
Luis Pinto | Besu:which I'm trying to… look for…
Luis Pinto | Besu:Yeah, so he says also that there's no EC recovery issues.
Luis Pinto | Besu:… Yeah, and so we're working on this, …
Luis Pinto | Besu:on these benchmarks that now run EAST opcodes, or, test cases.
Luis Pinto | Besu:Which we hadn't had a chance to run on our benchmarking.
Luis Pinto | Besu:… And next, unblocking 6 million gas, by improving EC ads.
Luis Pinto | Besu:We also need to work on that, but it's now less urgent given the other problems that we have with more basic opcodes, like division, and so on.
Luis Pinto | Besu:Yeah, that's pretty much it, what he passed along.
Parithosh Jayanthi:Okay, perfect. Thank you. And do we have someone from the Geth team who wants to talk about maybe their bottlenecks, or would they prefer to do that async later?
Marius van der Wijden:I can quickly talk about it. We haven't… like, the big one, the only big one is Modex.
Marius van der Wijden:And we haven't, resolved our strategy, how to move forward with it.
Marius van der Wijden:We have 3 different ideas. Either we go with, with, GMP, Or we, …
Marius van der Wijden:We make a proposal to update the standard library.
Marius van der Wijden:Or we pick some of the edge cases in our own code.
Marius van der Wijden:All of these three different ideas have different drawbacks, and we are still debating, how to go forward with it.
Marius van der Wijden:But it should… We should at least…
Marius van der Wijden:resolve which way to go forward, this week, and then we're… depending on the strategy chosen.
Marius van der Wijden:It would take, us a bit to… to get there. But it's also…
Marius van der Wijden:I think the general consensus is we don't feel like…
Marius van der Wijden:ModeX is something that will keep us from reaching 60 million.
Marius van der Wijden:At this point.
Parithosh Jayanthi:Yeah, thank you for the update.
Parithosh Jayanthi:And next on the list is, Carlos, do you have any updates or anything you want to talk about, relating to state growth? Or would you prefer to do that next week?
CPerezz:Yeah, not much to that from my… I just came back from out of office today, so yeah, not much to say now. Working on all of the test cases we already defined, and waiting on…
CPerezz:Ref to… to get back to us on the questions we have for client teams, such that we can complete all of the test cases, and we can move forward with implementing as much as possible.
Parithosh Jayanthi:Perfect. Thank you.
Parithosh Jayanthi:Okay, the next topic was, …
Parithosh Jayanthi:I wanted to get a temp check on clients' opinions on tools like OpenTelemetry.
Parithosh Jayanthi:So, one of the things we've been noticing is it's getting extremely hard to, sort of, deduce data just from pure Prometheus metrics, or from logs. Both of those are not very easy to do any data-driven analysis.
Parithosh Jayanthi:Especially if you're going down more on the performance optimization route, it might make sense for us to have a standard system, something similar to OpenTelemetry that clients implement, and we can ingest and use that as a comparison point.
Parithosh Jayanthi:It's completely opt-in. I would assume that it's even disabled by default, and we just run them on the performance images, as and when required, but it could also be something that's never merged into master, but lies in the performance branches of clients.
Parithosh Jayanthi:Yeah, we just wanted to get a temp check on how this is, how plants feel about this. It could also be very useful for the future when we're talking about gas repricing.
Parithosh Jayanthi:In terms of, …
Parithosh Jayanthi:Yeah, figuring out if we've actually priced the gas correctly according to all the clients. Like, benchmarks are sort of treating it like a black box, but OpenTelemetry would give us actual data from each client as to how the performance is.
Parithosh Jayanthi:Yeah, I know that Prism has OpenTelemetry already, and wanted to know if, yeah, other clients have opinions on this.
Parithosh Jayanthi:To Alexi's point, we can maybe compile some examples, or do you have something? Yeah.
Parithosh Jayanthi:Okay, I don't think we have an example directly right now, but we can share some on OrcoDevs and maybe…
Parithosh Jayanthi:Yeah, see how clients feel about it there.
Parithosh Jayanthi:Okay.
Parithosh Jayanthi:Yeah, we'll talk about that async then. The next topic on the agenda was, …
Parithosh Jayanthi:EPBS update? Do we have someone from the EPBS working group who can give us an update on how things are going?
terence:POTUS is not here. I guess I can give a quick update. So, …
terence:Let's see, so this Friday, first of all, this Friday, there's a breakout call, so highly encourage you people to join the breakout call if you're working on EPBS, or looking at Glemson Day in general.
terence:Other than that, …
terence:I have been mostly focused on, consensus spec tests and consensus spec update on EPBS side.
terence:And, the… currently, the SPAT tests have pretty good
terence:not pretty good coverage, but struggling to have coverage on just basic, like, SSC structures, so you can run SSE spat tests. There are a few, like, open questions where I'm still resolving that it's just, like, some of the renaming things.
terence:Making sure the fields are in the right position and stuff like that.
terence:I don't want to, like, go over them so much in detail, because, like, it's still relatively early-ish, but then I do think, like, this will be brought up in the EPBS breakout code, if you want to pay attention to… if you… if you want to follow those.
terence:But yeah, but currently, I think most of the focus on just getting this back in the right stage that, like, a client can start implementing it, and, which I do believe client can start experimenting it already and give feedback, because, like, Tagu and Prism already had a finalized DevNet earlier this year, and then, other than that, yeah, it's basically just fact changes, so yeah.
terence:Happy to take any questions, feedback.
Parithosh Jayanthi:Cool. Is there anything you guys feel like you need from a testing or tooling point of view, or is everything okay there?
terence:In terms of tuting, I think once we have some prototy stuff then running, we definitely need tooling just making sure DORA can show, like, the right page, making sure, like, for example, like, 4K, the four choice, also can consider, like, the notions of, like, an empty slot, for example.
terence:And I think there will be a lot of tuning, like, a lot of tuning, like, needs to be updated. I think it may be good to start thinking about this, like, what type of tooling may be affected in the EPBS land. But yeah, definitely a lot.
terence:And, yeah, I think it's probably good to start sooner or later.
Parithosh Jayanthi:Okay, cool. I think we can put that on our plate for once.
Parithosh Jayanthi:Yeah, probably in the next month, we should be able to start looking into it.
Parithosh Jayanthi:Do we have Tony or someone else for an update for block-level access lists?
Toni Wahrstaetter:Yes, I'm here.
Toni Wahrstaetter:Yeah, quick update, we have a breakout call on Wednesday, 2PM UTC, so…
Toni Wahrstaetter:For everyone interested, please show up. The main topics will be
Toni Wahrstaetter:some final discussions about the scope of the EAP, and also then getting into
Toni Wahrstaetter:testing, specs, and yeah, how to get to a definite.
Parithosh Jayanthi:Cool. I think we merged in support slash have support for Glamsterdam fork activation on our Genesis tools, so it should work on Kurtosis as well, in case you guys are ready for local testing.
Toni Wahrstaetter:Perfect, yakra.
Parithosh Jayanthi:Okay, Sunnyside Labs, do you guys have an update? Do you want to share anything?
Minhyuk Kim:Now, we don't have any reports this week, but we, our current work is, analyzing some issues with Perfect PureDesk Network, so we have a few issues that we have identified in a few clients and are communicating
Minhyuk Kim:directly with the client teams. And another thing that we are working on is we are building a chaos testing tool to try to stress the network in a more aggressive way, so…
Minhyuk Kim:Like, in addition to applying stress to, like, RAM or CPU, you're also adding features like, splitting the network into half.
Minhyuk Kim:And Trying to see if the network progresses as usual after merging the network back in.
Minhyuk Kim:So if you guys have any specific chaos testing scenarios or things you guys would like to test, you can
Minhyuk Kim:try to simulate it in our DevNets.
Parithosh Jayanthi:Awesome. Thank you, really looking forward to those PRs and awesome job with the chaos testing stuff.
Parithosh Jayanthi:We have an open floor now in some time, so are there still any open topics that people want to bring out?
Parithosh Jayanthi:Okay, last call… And… if not, then that's it for today. Thank you all for joining.
Justin Traglia:Bye, everyone.
Andrew Ashikhmin:Thank you.
Fredrik:Thanks, bud.
Mario Vega:Thank you so much.
Marius van der Wijden:Gosh, bye.
lightclient:too.
Chat Logs
00:03:47
Trent:Morning all
00:04:48
pk910:watchtower restarted a a few nodes due to image updates. might be related, but not sure
00:05:45
Barnabas:ethpandaops/lighthouse:pawanjay176-blocks-then-columns - lighthouse-geth-1
rest unstable
00:06:08
pawan:Sounds good, thanks
00:06:38
pawan:Yes please
00:06:52
Barnabas:https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=now-3h&to=now&timezone=browser&var-consensus_client=$__all&var-execution_client=$__all&var-network=fusaka-devnet-3&var-supernode=$__all&var-filter=ingress_user%7C%21~%7Csynctest.%2A&viewPanel=panel-34
00:07:12
pk910:blob spamming will resume shortly :)
00:08:28
Enrico Del Fante (tbenr):https://github.com/Consensys/teku/pull/9801/files#diff-2ccac92189e8015a51f5f55587e09706eaffa4ca949cbecae26e091f03eb5cdc
00:08:51
James He:Web3signer master branch has fulu changes ( even though the remote signer repo pr hasn’t been updated) so we can make sure to do some kurtosis testing with that soon. On remote signer support
00:08:55
Barnabas:ufff
00:08:59
Barnabas:yes we should have tested this
00:09:00
terence:can we add this to spec test please?
00:09:09
Barnabas:💯
00:09:33
pk910:we can fill up all the queues on devnet5 before fulu activation
00:09:42
Barnabas:Replying to "we can fill up all t..."
assertoor test?
00:10:19
Barnabas:we did consolidations before, but it wasn’t done during the fork boundry
00:13:12
Barnabas:https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=now-3h&to=now&timezone=browser&var-consensus_client=$__all&var-execution_client=$__all&var-network=fusaka-devnet-3&var-supernode=$__all&var-filter=ingress_user%7C%21~%7Csynctest.%2A&viewPanel=panel-39
Head slot not advancing for a while on some nodes. Client teams. Please take a look at your client pairs and let us know if you wanna do something about it.
00:15:40
stokes:For devnet-5?
00:15:42
Barnabas:assuming we find nothing crazy on devnet 3 non finality
00:15:58
stokes:Earlier the better 🙂
00:16:43
Barnabas:Replying to "https://grafana.obse..."
mainly lodestar/prysm nodes seem to be stuck
00:18:20
Fredrik:Three weeks sounds late in case something goes bad with hoodi
00:18:41
Roman:do we have a tentative target date when we’ll decide on testnet timestamps? or is it just after devnet 5?
00:20:26
kasey:@stokes could you please write out a concrete devnet and release timeline if you are proposing a change? We previously talked about mainnet release on 10/1, and presumably moving things back by a week would also move that date to 10/8.
00:20:56
kasey:Yup for ACD would be great
00:21:46
stokes:Replying to "@stokes could you p..."
I’ve always proposed mainnet for 5 Nov
00:22:01
stokes:Replying to "@stokes could you p..."
Oh, you are talking about releases
00:22:22
kasey:Replying to "@stokes could you p..."
Yeah so were you thinking mainnnet releases 3 weeks before 11/5?
00:22:29
stokes:Replying to "@stokes could you p..."
Yeah, I think assuming everything goes well along the way there 10/1 would be good for mainnet (and hood) release
00:22:53
Roman:can the same not be achieved with trusted peers only flag?
00:22:56
stokes:Replying to "@stokes could you p..."
Yeah, we have this constraint that we want a month before the fork for mainnet releases
00:23:09
Parithosh Jayanthi:Replying to "can the same not be ..."
Responded on the telegram chat why that’s hard
00:23:16
stokes:Replying to "@stokes could you p..."
So if we do 5 Nov for mainnet, we could do releases around 1-3 October
00:23:46
CPerezz:Replying to "can the same not be ..."
Yep. TLDR: Enodes change between CI runs. Thus need to be updated upfront every single time
00:24:21
kasey:Replying to "@stokes could you p..."
Understood, ty
00:24:34
Roman:Replying to "can the same not be ..."
let’s continue on tg
00:24:54
CPerezz:Replying to "can the same not be ..."
kk
00:26:41
CPerezz:Replying to "can the same not be ..."
Also @Roman . Would appreciate if you could reply a bit more exhaustively in: https://hackmd.io/WUe1KgwSS3iT8xoIbDg3Ng#Questions-for-Reth. Would be really useful and is the last client team missing.
If nothing to add, just LMK.
00:28:02
Justin Traglia:Opt-in or opt-out?
00:28:19
Roman:Replying to "can the same not be ..."
ammending
00:28:31
Preston Van Loon:prysm has support for opentelemetry today
00:28:53
James He:I think we are using open telemetry already for tracing needs , would be cool to see an option in kurtosis for tracing etc without the external setup
00:28:59
Alexey:specific examples of tracking using ot?
00:30:22
Parithosh Jayanthi:Replying to "I think we are using..."
Yeah it should be relatively straightforward for us to add support on kurtosis
00:32:00
Justin Florentine (Besu):Besu supports OTEL today also
00:32:42
pk910:dora should already be prepared :)
just need to update to latest gloas specs
00:32:56
terence:Replying to "dora should already ..."
that's awesome!!
00:33:28
Barnabas:Can confirm gloas is already in latest main for ethereum package
00:33:33
Mario Vega:Could you share the date/time of the BAL breakout here pls?
00:33:38
Toni Wahrstaetter:yes one sec
00:33:54
Toni Wahrstaetter:https://github.com/ethereum/pm/issues/1691
00:33:59
Toni Wahrstaetter:This is the BAL breakout