 Hey Clemens, hello, how are you doing? I'm doing well. That's good. Yeah, folks This isn't your first time on the call is it mine? No, yeah, okay. I didn't think so I have to say today I have lack of motivation in general. I think that was that is all COVID induced You know, what's funny is I was looking Earlier in the week I was looking at The agenda, you know trying to get a ready and stuff and I realized we don't have a whole lot to talk about especially when it comes to PRs and stuff and I'm trying to figure out if it's People are just really really busy with other things. They're with all the stuff going on. They're just exhausted As you said, you know, you have lack of motivation right now Or whether it's something a little more serious like people aren't as interested in these topics anymore I don't think it's that last one, but I I'm trying to wonder and worry a little I think I think this is just a weird weird situation that we're all in yeah And and of course this is this is not as This is probably not quite as exciting as public events per se. So It is I think those topics are becoming a little bit more more niche maybe Hey, Eric. Hey, Tommy Good morning Hey Yeah, so one of the things I was going to do when I I apologize. I actually meant to to do this to try to force some discussions is to Really go back and read both the latest specs and basically just identify all the holes that I think they're there and that will force either people to sign up to do work or You know, give myself work to then open up prs to put initial draft proposals out there to fill those holes and force some discussions It's they're going so But I didn't get a chance to apologize Yeah Hey, Kristoff. Hey, how are you? Good. How you doing? Uh, Heinz, are you there? Yes, sir. I am. Hello. And is that mr. Mitchell's? Yes, it is. Good morning. Good morning Hi, David Good morning Who is VV? Sorry, that's me. Doug. It's Vinay Venkatragoon Hey Hey, good morning Wait a minute. That's not Vlad, is it? Can you guys hear me? Yeah, can you hear me? Yeah, this yeah, I do. This is Vinay Venkatragoon. Oh Vinay. Sorry Yes, yeah, sorry I am so bad with names, but now when I can remember it just I just couldn't hear the first time for some reason No worries. Thanks. All right. Hey, Nick Hi, Doug We do team Doug, do you think we will have like five to ten minutes to date to do the demo of some educate testing? Sure, do me a favor stick it in the agenda We'll do we'll do thank you. All right. Hi, Dustin Dustin, are you there? Hey, sorry. I'm here. Hey, hello And Thomas are you there? Yeah Yeah Thomas you've been here before right? Yeah Okay, good fun to make sure Hello, everybody Hello Hey, Klaus Hey, Doug Hey, R.I. One of these times I'm just going to type R.I.P. and that's going to be bad, but No, my email address is R.I.P. I really don't mind Okay Might be the easiest way for me to remember it Hey, Mike Morning Hey, Colin Good morning Doug. How you doing? I'm good. I'm good in yourself. Pretty good. Pretty good. It's a nice rainy day here in Raleigh. I love rain I think better than it. This is just it was a really cool huge thunderstorm that just shakes the whole house. I love those I know foreign for you California folks Uh, hey Lance We have not had rain in germany for like three or four weeks Really? Yeah Did you guys ever go um in a situation where you have like droughts and you have to like think about water rationing? Yeah, well, not water rationing, but it's a drought. Yes Yeah interesting We have some fire danger here in switzerland as well. Yeah We're not supposed to do fire outside There's there's a there's a Pretty large forest burning in Holland just right, you know do geography kilometers away from here Oh, wow Okay, anyway, it's three after that's going to start at the last check christian. Are you there? Yep. I'm here And kent. Yep. Morning kent. Are you there? Yeah, I'm here. Excellent. All right, and I see scott. Yes. I hear you. Yeah All right, let's go and get this show on the road. Um, just a reminder clements You took an ai to write up your proposal for schema registry stuff And all right community time anything from the community people would like to bring up All right, not hearing any so Um, in terms of us becoming a real full-fledged sig I did put the formal proposal out there to not actually do that And rather have us just move as a working group under the sig app delivery Um I'm still waiting to hear back from the to see to see what their official position is on that I think I just put the request out there yesterday or the day before So, uh, it might take a while for the to see to notice that but I'll let you guys know how that goes I don't see any real problem with it other than I think some people may think Serverless is worthy enough of its own SIG but no one seems to be able to come up with a really good definition that won't be too contrived. So It doesn't really matter in the end All right, uh, no stk call this week. Does anybody from the stk's Teams want to mention anything in terms of what we did on last week's call because I cannot remember Okay, not hearing any nothing too exciting. Then we can move forward Um, is team are on the call No, I don't see him. Okay. So I don't think there's anything update from the workflow then So let's jump into our prs and stuff. Um Okay, first one I don't think this one is controversial But I wanted to bring into people's attention to see if they have any concern about them doing a protocol binding for pulsar Anybody have any concerns with that? We have since we have Kafka. We can have pulsar It seemed like a good thing to me. I just wanted to make sure that I didn't want to because if we had some concerns with it I didn't want to go off and do all the work just to have us come back and say no. That was a bad idea I think we had we had a discussion um a while ago about pulsar and uh, there wasn't so, um Well-developed community I would say but that has clearly changed. So yeah if If Kafka has a binding then pulsar should also have binding Okay, any other comments from anybody else? Why not? Say that again Oh, I said why not? Okay. Why not? I thought you said or not. I'm like, okay. Oh, no, sorry Okay, cool. All right. Just maybe a quick one. Yes, uh Okay, there is to be Kafka, but now also pulsar, uh, prevega and a few other systems So, uh, they do share common Aspect of having records key, uh partition and so on and so forth and headers as well So perhaps it can be changed to generic specifications for Kafka like systems And include Kafka, pulsar, uh, prevega and others Yeah, but yeah, but they're all different, right? Because because because I think if you talk to the pulsar people and they and you tell them what their Kafka like They will say no They will say we have a much better architecture, which is true so I would I would love I would love for everybody for all of those folks to go and adopt mqp And and then that probably would go away and that's actually a discussion that I'm I'm opening I'm about to have with the prevega folks, but Yeah, I think as long as as the the those parts are different And if there is if they are under the umbrella of an open source organization I think that's part of the criteria that we have Then yeah, they should So prevega would actually prevega doesn't meet the criteria yet But pulsar clearly does Okay, well if you if you want to make a comment and they're Mentioning that to them and maybe they can think of a way to make it more abstract I don't know anything about this stuff. So maybe not But if you want to at least put the idea in their heads feel free to add a comment like that to the issue Did you talk to me? No, I was actually talking to sir jay right Yeah, I know you don't want to do it clemens I would want them to go to go and use amgp. Well, you could but you could put that as a comment in there too if you want Yeah, yeah, there you go. All right. So, uh, next one. So I opened up a pr last night So obviously it's too soon to even think about approving it. However, I did want to draw people's attention to it Uh, there are two main things in there one actually I guess three one is I added the new specs to the To the checking tool that we have So now we'll check where things like the rfc keywords are capitalized and stuff like that As a result, I did find some cases where we're using the keywords in non uppercase And I believe all the cases were not meant to be Using the rfc keyword. So I changed those but please Look at my changes to make sure I did not change the semantic meaning if I did it was a mistake and let me know And then I think there was a one bad href in there. So these are all mainly syntactical type fixes but in particular if Everybody but in particular uh clements and mike if you guys could take a look at my changes and make sure that I didn't Change anything um by mistake I'd appreciate that review Okay All right, moving forward then multi part mime Where are we on this one francesco? Uh to be honest this week. I didn't any progress at all to be relaxed. I just look at the comments try to Reply but I didn't look at it at all this Would you like to have a discussion now or do you rather wait until you've had more time to talk about the comments and It's like that. It's up to you whether we talked about it now or not. Yeah, let's let's skip to the next week Okay, there enough if anybody has any concern Or just put in the comments also I also remember to people that uh, there is also another proposal I did which is far simpler to implement and it's and it's Based on jason's swimming. So send me multiple jasons They limited by two chapters Okay It clements, I think you had some some thoughts on this one, but I didn't see you make any comments in there You would be possible for you to give you get a chance to add some comments in here Yes, I I just have not been able to do any more Yep, that's fine. Okay. Just wanted to put a little nagging reminder out there. Thank you Thank you. And of course that applies to anybody else. I just come into the one I remember speaking up Okay, so that's it in terms of actual like prs and stuff. So last night I did find some issues that I thought might be worthy of discussions um Just to see what people thought about them so we can clean out some of the backlog Um, francesco, this one is yours as well And I apologize. I meant to actually reply back to you. So why don't you talk to what this issue is first and then we can have a discussion about it Sorry, you want to you want to bring everybody up to speed on what this issue is about so we can have a discussion Yeah, so basically started implementing the uh, the distributed tracing extension and I found and I found something that to be honest, it's not Really clear to me, uh, which that which says that The uh, when you send through htp you could potentially contain both the uh transparent as a cloud event extension and as an htp adder matching the trace context uh, w3 spec and It tells and and and the sentence after is not really clear because it tells that I should use the cloud event extension in case There are both in the same envelope and they don't match But this doesn't really make sense because if you go through a middle word that is not cloud events aware It will and but it's tracing aware. It will use the trace context that and not the cloud event extension We had a long debate about this Yeah Okay, I think what I what I tried to write yesterday probably came out all wrong but I was trying to say yesterday in this comment was They can both appear in the message and they can be different because the actual trace header may change as it goes from Hop to hop to hop, right? But at the final destination if they're different My recollection was that if you actually want the real tracing value Because you understand w3c tracing then you go to the tracing header This this the cloud event version one was in there More for historical purposes to say what it was at the beginning of the flow Yes, but it's not actually meant to be used for tracing per se Yeah, okay, so what? Which which of the two? Should a middle word use a middle word that is both trace tracing aware and cloud events aware Use it for what purpose Well, that's that's not I mean this reasoning is not clear on the spec I know but but but you said the middle word wants to use it, right? And if the answer to my question is well, they want to use it because they want to adhere to the to the tracing spec Then they should use the real tracing header. Yeah Okay, so if I need to commit the span a new span and and I need the Idea of the parent span. I need to use the one that is in trace in the trace context others correct Okay, okay. I the idea the idea the the thought here was for for those two to coexist is There is an end-to-end concern for tracing and then there's a concern for tracing when it comes to the infrastructure pieces, right? and You want both of those activities anchored in the same Origin context, but effectively all the the the offspring contexts form effectively a tree so they the cloud event is effectively that's giving you kind of this original context information But then if you're just routing Let's say you're routing in a cloud event through a chain of multiple htp Landers, right? And then the cloud event shows up on the other end the application might actually not care about any of those htp hops But it only it only cares about the end-to-end scenario So it takes the original cloud event The the original context that it sits in the cloud event and uses that To keep to keep going But then if you look at the htp side the htp side will actually have have traced all used that same route To go and trace that and it will have used the the htp Context information so so so the the tracing information basically just forms a tree And it has a shared root But we're effectively carrying both of those contexts with the cloud events Transport mappings Okay I to be honest. I don't have this big knowledge of tracing but my question here is That I don't really get is why In the end-to-end case should I care about the original trace parent when the tracing infrastructure already does the link with the Middle trace with the middle spans, right? So If I'm If the sender is already committed the span with the original transparent The the final the other end of the Of the flow Is able to link back to the original trace to the original span. Okay Yeah, so if if you're running if you're running in the so let's say you're running in a cloud pass infrastructure Right, and then there's the in the cloud pass infrastructure is A ton of stuff that you don't see and that you don't care about and that you also don't have any insight into um For for that cloud for for the support folks in that cloud infrastructure To be able to be able to help you with a particular case They need to have an anchor for that for the the trace and that is your original event However, for your own application level tracing You don't care about any of that stuff that just happens in that cloud infrastructure But you're effectively just calling you're just care about your own publisher and your receiver And those are linked through the information that sits in the cloud event I see that makes sense. Yeah. Yeah, that the completely makes sense Well, I but again, this is not clear in respect. Are you okay with uh, if I work here to Yeah, we had we had a super long discussion. There was a there should be a There should be a fairly lengthy issue thread or pr thread That talks about that Yeah That is closed Giant very long discussion that Occasionally got heated If you want to write some new text for that For that spec that'd be most appreciated Okay, because clearly it's not sufficient. Yeah Yeah, I mean to be honest, I understood the opposite I mean Maybe it's me but Well, I think it's probably this right here, right? Must use the cloud events attribute Yeah, we don't say what we don't say what you're using it for. I think that's part of the problem Yeah. Yeah, I mean must use must be used the cloud event attachment for what? Yeah, so so how about how about so now none of you know the the The explanation of how we thought about this Yeah, how about you make a proposal because you have no fresh understanding That will probably be better to clarify this than Than anybody else Okay Cool. That would be much appreciated All right. Thank you everybody. That was a good discussion All right, um Graphql, Mike, would you like to talk about the graph ql today or would you prefer to save that for another day? I mean, I haven't prepared anything to talk about Well, okay, let me ask the house there hasn't been any comments on the issue So like if people are if people want to just let that fall by the wayside and never discuss it That's you know, that's fine. Well at some point we either need to do something like a pr or close at one of the two But since and thank you for mentioning that no one's commented Maybe that's the next step is to Use this opportunity to nag people to comment on the issue I I know clemen's you had an opinion I want to say gem might have an opinion But I'll let's give it at least another week for people to add commentary to the issue and then it will circle back around next week and say Okay, and I'll make a note to remind everybody to I'll send that a note or it might be able to talk about it or to think about it Yeah, thank you. Okay All right next issue Francisco this might be yours. Oh, it's not never mind so I wanted to get somebody who's a Kafka expert To have an opinion on this one Anybody have any thoughts on this one? Yeah, I know it's not just you Francisco I haven't seen it but I can take a look Yeah, I can take a look too Okay, if somebody could just comment on this because I just I don't know Kafka ball enough to have an opinion. Yeah Well, this is right. This is this is I I agree with that comment here. Is it we only currently have we we have Jason as the as our canonical structure format But yeah, if we have other formats So if we're if we're coming back to protobuf or if we're adding seabor Then of course having a self-contained structured event in seabor would just just as well be applicable here, so So yeah, this is this is unnecessarily constraining So I what would be the change of anything in the spec? Um, can we can we look at the actual wording? Um, Kafka spec, okay Implementation made I assume it's probably here. Yes Um, so I think if we strike and encode the event in Jason Then we're I think that's the unnecessary restriction Is that a breaking change though? I'm interpreting the must to go with the end. It's not Because Huh, we're making something less strict well As a receiver if they're assuming it's going to be Jason because of the must are we now going to break them if it's not jason And I don't maybe it's not an issue if we just made it So I think technically I see technically yes, but um Um So technically yes, it's it's a breaking change because of that but I'm not sure how much that really matters to actual implementations if they are using You know if they are in the spirit of the rest of the in the rest of the The doc's ends, you know today mostly all implementations will probably have a jason bias anyways, even though they shouldn't so if we were to strike this and claim that Either it did not go with the must or we were just flat out mistaken Yeah, are people okay with that? I think it's a carrot Would everybody agree with that? Yeah Okay, I'm not hearing any objections Oh Heinz I just have a quick question isn't this Kind of intuitive if you can't support headers you can't do binary mode So you can only do structured mode and structured mode is jason So i'm not sure where the conflict is It's structured mode is not always jason structured mode can also be seabor or avro So we don't have an we don't have a seabor by do we have an avro by Okay, so you're worried about the fact that it's a jason structured mode versus just structured mode in general Yeah, because because kafka just carries binary and doesn't actually care about the the shape of the payload In its value, okay Now it makes sense Okay, um Does somebody want to open up a pr for this or do you want to push it back to the original author ask them for pr? Anybody itching to do a pr Okay, fine I will poke the original person And ask them to do a pr I just wanted to add that in kafka 0.10. There was content type support already which means that some level of content initiation can be done and jason will come with application jason and seabor and others will come with their own content types, so Even if it's a breaking change due to mass change to shoot The implementations that handle the content type already would just continue working Yeah Okay Not mentioned that uh going nor uh java SDK They don't support, uh Structured mode in kafka Not yet No, no, no. Yeah, go long go long sdk support it now Okay, maybe I can talk about it later A v2 should Yeah, and sdk java will support it too Okay, anything else on that one we need to discuss Okay Next one All right Um Do we have any avro Pseudo experts on the call who could comment on this one? I'll give you guys a second to read it But he no avro well enough to comment on this one semi Care to go any further Um, can we can we take a look at the the schema? Do I can get to it? Um There's neither that one Okay So I assume that's an example Is this that the uh Oh wait, let's see. Maybe it's this one. Hold on Is he talking about the fact that Attributes is under fields Yeah And that we only have them We have exactly attribute and then we have a map for the attributes Um, the reason the reason for that was if I remember correctly That um We We basically just want to mimic Jason Um in in avro rather than then make basically bolt down the event Um structure Into into avro per se. So we actually didn't want to make it as strongly tight as we could because of extensions because if we if we um So we could have special case the the fixed the the fixed Attributes And then created an extra bucket But that would not have been effectively in the spirit of Of the the rest of it to have a bucket So we to make it all even in all in one place. That's what we chose chosen that if I remember that all correctly and then data um Should also be able to carry Structured information and that's how that um comes about that you can basically go and take a jason object as we allow in jason and then you can go and basically just transcode that into the into the Data elements and of course you can also just put a put a byte array in there if you want to do so is it Is it not technically allowable in avro to just put all the map entries as top little things as siblings to data Um as still no, so I mean this is effectively this is a this is a schema that doesn't that doesn't represent the cloud event But it's a schema that allows you to go and sterilize the data So it's not it's not a validation schema that tells you whether the the event is correct Or whether the event has you know all the all the fields have all the right values This merely allows you to take a structured cloud event and just put it on the wire in in um in an appropriate way Because the avro serializer is driven by that schema just like protocol for this Ryan your hands up Yeah, I think it's an interesting distinction So your clements if i'm understanding you correctly This is really just the the encoding the the implementation of encoding for avro But doesn't represent the canonical schema itself. Is that correct? Yeah, and and is that is that the way that we think about the other representations as well? I don't remember um well So the goal was to keep this so things were all in motion um and we had um You know this notion of extend we have this notion of extension where you can go and add whatever you would like And the the the predefined attributes are also optional um it took two large part and so I think the argument and this is not me defending defending the structure just trying to recall how we got there um Is that you should be able to go and take an event similar to Jason and then just encode that into So it's it this is effectively the same tagged value approach if you will that have that jason has but express an avro Uh hinds your hands up Oh, yeah, just a quick question um having done some recent work with uh Avro serialization with both apache and confluence It makes sense on the serialization part on the deserialization though. I found What uh, it spits back after you get the consumer record is the actual jason Data from the deserialized avro message So you're saying this will spit out the correct jason for the consumer It's after dis deserialized No, it's good the actual part of it It's going to give you the right object graph Not jason but object graph The way that's the problem Huh? It the consumer records if you're doing it for example with spring cloud scream libraries against confluence It gives you back a consumer record and then the consumer record It gives you back the value out of the object And the value is always represented as the jason object That was deserialized from the Avro message so i'm just You know so as long as it will spit out the result of the standard libraries of your actual jason representation that you expect record events If that's doing that it should be fine, right? If it's not that should be a concern Yeah, but that's that's the Arguably that's a choice that's the that's the um that the serialization library makes Um, because what? This will yield So what this yields is effectively you get an object with two fields And one of the fields contains all the attributes and and the other field contains the data So it's slightly off from how the cloud event looks in in jason, but that was never the design goal Because what you do use both the If you want to do a custom serializer using the avro libraries for apache or When we did it with the confluent Avro serializers and deserializers um, you're correct the When you do the cogen to create the pojo from the avro schema It does allow you then to update The record automatically have it stored as a serialized avro object However, in both cases when you consume it And deserialize it It always gives you back a jason representation of the object If it's not giving you back a cloud event jason message on the deserialization And I'm wondering if that's going to be an issue because you're not getting back cloud events It's surprising that the thing even gives you a jason and not just an object graph Well, I was surprised too, which really actually bothered me because then to actually work with the large Result of the jason. I had to use the jackson libraries to re-serialize it against the jason schema Just to be able to manipulate it or I had to use some You know primitive uh jason libraries where I called it up and I had to pre-move the field Just not a big deal, but when you got an ide it's kind of silly Well, that's that's a good question to ask confluent Yeah, exactly Again, I'm sure it's easy enough to test for whoever created this is you know Take a jason message or a Take the avril serialize it and when you deserialize it what jason do you get back if it's not the Captive jays. Sorry if it's not the cloud event jason, then you're probably going to have to change the avril schema Uh, no because avril is avril and so there should not be if you're doing if you're sending if you're if you're sending avril and you're getting avril Then jason should never be in the picture So if if if confluent decides that they run everything that has been avril serialized always through jason, then that's their problem Yeah, this feels like an implementation detail. Sorry go ahead Yeah, I mean this feels like an implementation detail because avril actually has two different representations or two different encodings It has binary and has jason And like it avril is not jason avril is avril and then jason is a representation of It's specific encoding That you can use with avril, but it's not avril So I'm sure I understand. I sound like you guys are saying that there may be a bug in somebody's implementation of the avril to jason mapping No, what I found is if you have if you don't represent the Avril serialization or the avril schema When you deserialize the message it and again, you're supposed to be able to from the deserialized switch back and forth between avril and jason My default when I tried it with and literally did this last week Where I tried to use the deserialization using the avril libraries using apache and apache capca and I tried it again with the Implementation from console where they provide their own automatic Serializers deserializers If my schema did not match what I wanted the output to look like when I went to use it as jason it actually spits out Jason as the value from the Kafka record when it deserialized so that's a so that's the two problems here one is that console and gets jason into the picture, which is weird Um, and then the second is the way how the way how you should do trans coding And that's also something we had discussed is we have this we have in all in every sdk We have this generic representation of what a cloud event is in memory and The the formatters That are using the format specification They serialize into and from that generic representation. So if you come from avril You go to in your sdk to that object and then from that object down to jason but of course if you're expecting that the wire format Of avril maps directly to the wire format of jason. That's that's never been a design goal Well, the design goal of avril was just to compress the data when you serialize it to make it highly efficient for transporter storage But you're still when you take avril You will then create a pojo if you're doing this with java From the avril schema You will then apply the fields that you want to update just like you would if it was jason Pojos from the schema and store it or send it in the case of casca and when you receive it It expects to have that available as data but the data in both cases it spits out was jason So so but that's not so the way how we're doing the way how we're restructuring the sdk is Is we have a notion of what a cloud event looks like in memory And that that is that is independent of all serialization formats and that will work for seabor and that work will work for everything else and From you should be able to go and take an avril documents Deserialize that into into a cloud event that that's our representation And then you should be able to serialize that representation into jason and and there should be no mismatch whatsoever But there can't be an expectation that if you take avril an avril schema as this this year And then turn that into a serialization object and do a raw deserialization what you will get Is the immediate representation of the structure that we have here Which we chose in particular to put avril on the wire But that is not the cloud event the cloud event that is one further mapping step Which takes the attributes and puts them into the attributes and takes the data and puts them into the data In the abstract data model that we have So let me go to ryan his hands been up for a little while ryan I Yeah, I was just going to go back to like the original Um Just thinking about the trade-offs here It feels like we're trading off type safety for extensibility of cloud events right because if we were going to If if there is going to be a new extension for cloud events That means we would need to create a new avril schema if we were going to do this in a type safe way Um, I don't I don't know how I feel about that I'll stop my head Um, but it feels like that's the trade-off that we're that we're making here That that that was the motivation because every cloud event can have arbitrary extensions Yeah, and and and the thing that you said in the very beginning comments resonated with me, which was All cloud event attributes Spectified or extensions were meant to be Treated as siblings to each other Correct and that that that was a big motivation for us. So scott your hands up This is actually the exact same issue with proto buff Um, where You give up the type safety for extensibility So if if if it affects this thing So one idea for proto buff was that uh, what what if the as the thing gets versioned and specced and Extensions get promoted up into the proper spec they The the object gets migrated out of the extension bag and into the Strongly typed spec like like these pieces I wonder if if for, uh, strongly typed Specifications like afro and for proto buff it actually makes sense to keep it as bags For extensions and then have a little bit of middleware shim to help you pop the extensions out and make them top level At the last mile Anybody want to comment on that or are we opening a can of worms? there's there's pros and cons so the Just in terms of prior art um So nqp shows to do Have an approach like this where um There's a properties collection, which is all the commonly used fields to reply to from dates times them etc and they are Using a schema where the metadata is then the metadata fields Names are omitted So they go their serialized web position So that's similar to what you would do with proto buff or with afro with the schema and then there's an application properties bag Which where all the extensions go The reason why the reason why we and this this goes back to the original discussion that we had around extensions and whether we should whether we would have bags and the the discussion that we had was was If we have an extension And that extension ends up being so popular that everybody wants that everybody wants it and we then decide to elevate it up to a Proper attribute So it becomes part of the canonical attributes It's a breaking change because now that attribute needs to go from the extension bag To the to the top level set. Yeah I don't want to go down that again. It's already a breaking change, right? You're going to have to change the version of the the spec proto Yeah, but you don't so with with jason for instance with so with any tag format may that be jason or seabor or xml for which we don't have mappings um The event wouldn't that wouldn't change. Yeah, I Thinks that aren't version like jason it uh, it's easy to promote the extension to the spec and have the The on the wire format not change, but that's not really the case for version protocols Well, that there will be the case. So if this work with with what we're looking at right now, this will also work I think I think the point is you lose it the the um The type safety or the proper term is right the well the fun types of the protocol I mean you can still I mean the validation is something you can still do It's just you don't do it in the serializer. You just do it a step later Because ultimately the the in-memory objects that we have they know what the types are And if you give them if you give them a a type that's wrong they'll throw But doing that you lose the the reason why you're choosing something like protobuf Like you can't you're basically then just using Uh a jason stream over protobuf instead of the actual proto Well, you're still using the prototypes You can't if if you support extensions and you go in version every extension promotion. Well, you still you still use the the It's still protobuf serialized It's just that you have tagged it's just that you carry the stuff in the map. It doesn't work in practice though Why Because you can't you have to tag it and the tagging is the the cumbersome part What you want is to send the the interversion Of the tag not the the jason version of the tag I'm sorry. I know nothing about afro. So I can't Same here. It's the same here. So yes, your argument is the the the message will run long because you're including the tag Yeah, that's that's right. And but When you promote that extension into the core spec Then it's it's still a breaking change and it has to be a last mile Collapse or you can't use the the binary format of anything For the the proper spec So i'm gonna call time here because I have a feeling this is gonna this could technically go on forever um So I think I understand what's been said here to at least give an initial answer back to the person Open the issue I'll take a stab at it and then you guys may be correct me where I got it wrong But I do feel like we should give this guy at least some kind of response because I get based upon his Based upon his issue It sounds like he was in the middle of actually implementing stuff and I wanted I didn't want to be blocked by us for much longer um So I'll take I'll take an initial stab at trying to answer them and then you guys can correct me. Okay um But I did want to leave time for Sure jay to talk about his sdk testing stuff You do want to take the screen? Yes, please. Okay. There you go. Go for it Okay, I hope you can see you can see my screen. Mm-hmm So, uh, we have a great specification and uh, it answers many questions But uh, when I started working with cloud events, especially the sdk part I was missing some guidance on how to test it Which cases to test and so on and so forth and uh, for example, let's take a look at the htp protocol binding of specification Which is the most popular one. I guess um, it answers many questions It also mentions some edge cases like a content type and that it should be You know, even here it says that it may include char set and some other modifiers So it should be parsed So I started thinking how to test sdk's with the same set of tests and an obvious choice for me is cucumber which is bdd technology for Describing the test cases in some human readable form. You're not coding. You're basically writing some structured text And then this text can be interpreted with With the language of your choice be java going or basically any other language. There are many languages supports that we'll talk about it later So here, for example, I have uh, the same example as in the htp protocol binding Markdown file. So we just open it again um Yeah, it's here. So it's binary content mode and then we have things like, um headers prefixed with ce Content type with char set and so on and so forth so I'm just Doing this same here in this cucumber specification file and I'm also adding more variations with content type. So I'm just testing application json, but also Mixed content or not mixed content type but content type is modified And I have the same for structured content mode as you can see here Basically, it's the same. Um, with the same variation of content types And I have the same for Kafka Here, um, since in Kafka there's no common definition of uh, how Kafka message should be represented in uh text form I'm just using key value pairs for header and the payload is defined as string. Um, it of course can also Have something like with base 64 payload and some binary payload or whatever you prefer And uh, the assertions I'm using. I'm also checking, um The quality to some json not just to some text Or the string based value but json because some implementations may decide to parse some implementation may decide to um return The content as it is So anyways, we just define Pro not protocol but language agnostic specifications and scenarios And then we can run them And here I'm using sdk java. So I'm just inside the sdk java project edit a few files So I'll show you later and I'm running the specifications As you can see we have some failures And uh, this one is the most popular, uh sdk issue I would say So I can click and go to the specification. This is our uh binary content mode specification And we can see that char set is specified Apparently java sdk uh does not split the content type and uh does string matching and obviously application json Semicolon blah blah blah doesn't match application json Let me open chat Mm-hmm. Okay. Cool So um another failure is uh binary content mode in Kafka So if I open the scenario it says that content type is application json and data content type should match application json I went to the implementation. I'll not be talking too much about the fix, but I just fixed it. Um here in In the Kafka implementation um, and now if I Okay, anyways, uh, if I run it again I just added support for content type because previously it was filtered out with c prefix Now it's green. So as you can see, um the Kafka sdk works with uh all Kafka types of messages defined in this specification, but uh, the binary content mode still has to be fixed and I made a wrong uh statement earlier today that, uh Structured content mode is not supported in the java sdk. Sorry. Um, I'll be talking about the goal against the k later I'll make a break to read the comments Okay, cool, uh, so we're good And I would like to show you that it takes just a few lines of code or maybe A few hungry lines of code, but still not much to run the test I have standard infrastructure for cucumber java here and I define steps like, um attributes should match something or uh, the map of attributes or the table of attributes should match or, uh, the data should should be equal to, uh, some json I can use any library I want into java ecosystem because here I'm running in java, uh, to do some json assertions And I'm not limited to what cucumber provides Which is great because It depends on language how it prefers to compare things or whatever And the same applies to, uh, for example, htp steps As you remember the payload was just the htp message as it is on the wire and, um I need to parse it so here in java i'm using raw htp library to parse it Then when it says parse this htp json, I only need to use sdk's Kind of internals, uh, but also, uh, the part of the public api Structures to test the event and then do some assertions later As you can see cloud event step is a generic, uh, definition of steps So that doesn't matter how you parse them from htp message or maybe from kafka message Here's an example of the steps for kafka Um You just parse it once you set it the current cloud event and then you can run, uh, standard assertions, uh, on the Parsett cloud event it works both ways so that we can also do something like giving some cloud event We run, um, or we expect It to be converted into kafka message into htp request and so on and so forth I know that some of you, um Are looking at this and they're like meh java, uh, but what about golang? And I will switch to vscode where I have, um The golang sdk I have exactly same, uh, actually reusing the same files the same specifications I'm not touching them and uh, you'll find the same specs here, uh, that can be And should be put to some shared repository or maybe even next to the specification if we go that You know that close to running vdd style testing of the sdk's And I have some code here as well to glue Um the steps and uh the golang sdk So here i'm just asserting, um, the cloud event, uh, the same thing, um Sorry i'm not golang developer, um, and I had to just Unmartial json and compare it with the same, uh, cmp that's used in other tests. Um, so i'm pretty sure folks from sdk Sorry from golang sdk will find better ways of, uh writing the steps And the same for kafka for example, um, you know, we use sarama For creating the in-memory representation And the same for htp htp is It's a very simple one because we already have everything and I only need to call this uh binding dot to event By the way while working on the steps It also helps to get a feeling what it takes to integrate sdk into some code base because Here in golang we have this, uh Wonderful binding dot to event method that takes care of Parsing the content type the spec version selects which sdk sorry which spec should be used and in java we don't have it So I had to add some parsing myself in the step definition just to Decide which sdk or which spec to use Now if I run it with go test The same set of tests We'll get Two errors And if you look at the errors, um, and I'm not on the latest master at the moment. So please bear with me We'll get two Fellers our first one is htp protocol binding with structured content mode and the same semicolon char set blah blah blah issue And the same with kafka so That's something I reported to scott and uh, he fixed it shortly after so if I just do git pool And run the same spec again On the latest master boom the latest master works great. Um Which is really nice and uh, yeah for me it was a great feeling of just Pulling the latest master and telling scott that yes the specification is passing now Uh, I can assure you that your fix is working and I didn't touch anything in the project code or Nor did I touch something as a test And last but not least I also have the same for v1 of the sdk and I know that You're currently working on v2, but still it's nice to be able to run the same exactly same set of tests with v1 And get some results. So I see you can see The structured content mode is failing um, and um What you can also see is that kafka Is not tested at all Because I didn't find how to you know, uh Test kafka with v1. So I just decided not to implement this step and if I open this step definitions Just a second Here We just say, okay. I don't know how to parse kafka message. I return pending and uh, the result of running the specification running cucumber, uh, will be Passing or uh, some pending Features so that if your sdk does not support some of the Futures yet, then I can just keep it pending and it will be a to do for you runable to do To implement it in the future And it's the last So I think for me about cucumber and this approach is that It's a very popular framework and you'll find many implementations for many languages And as you can see, uh, there's golwang gvm javascript ruby And php and others Most of the popular popular languages they have cucumber support And if some don't then implementing cucumber support isn't hard Just a bunch of regex, you know, how hard can it be? And the language itself is gerkin language So cucumber is just implementation of framework for gerkin language But the specifications are written in gerkin and you can just use some other gerkin compatible framework. Thank you Okay, cool. And technically we're about out of time But I think this would be a great topic to bring up on next week's phone call For for the sdk meeting because we meet every other week But scott, do you have something quick to mention? Yeah, I think it'd be really interesting to put this in the performance repo and have like a compact Table to show the integrations. So maybe it's not something that goes into the sdk implementations But a separate thing that uh tests released versions of the sdks or something Yep, I agree. Yep, we'll talk more next week. Um, francesco quickly Oh, my hand went down Okay, so in the last 30 seconds we have let me just do quick final roll call check I think I got everybody except for alex. Are you there? Actually, I don't see him in the list anymore. What about the other dug? You still there? Nope, I don't see me there. What about I'm gonna butcher it. I apologize Tisca Yeah, d-a-s-c-a-l-i-t-a. Are you there? Yes If you can yeah, if you can put your company name in the agenda, um, I'd appreciate it if you want to be associated with a company Absolutely. I I want to start joining. I'm from adobe. I work as principal and engineer for a service buffer at adobe Yeah, okay. Cool. I can spell adobe. Thank you Thank you. Did I miss anybody else for the attendance? All right, in that case excellent. Thank you very much for the demo. I really appreciate that looks really really cool And let's talk about it on the next SDK call next week All right, and thank you everybody. Have a good rest of the week Bye