 Good morning. Everyone's taking a few more minutes to be able to come on in. Good morning. Liz can do that mic check again. I was not quite sure. Yep. That's much better. I don't want to lose Liz in here. That's terrible. I know saw it isn't going to be joining us, but we also don't really require form for this particular meeting as it's going to be a discussion and I really hope no voting. So I will be very surprised, Liz, if you come up with something to vote on then. I think so too. So let's see if I can find the deck. I know you'll be sharing it anyway, but if you can post a link to it, that'd be great. All right. Do you think we're ready to get started? Have we got enough? Yeah, we can rock and roll. Welcome everyone. Hope everyone's feeling both energized and recovered after KubeCon. So I think the main topic for today is the DevSecOps end user radar, which I think a few people, actually it's Justin here. It is indeed. Hello Justin, hope you're feeling better. And Ricardo and I and I think a few others sort of had some opinions on this radar. So I think maybe if we go on to the next slide, it illustrates what it is that we're talking about. I think, so the end user radar is currently something that's handled entirely outside of the TOC community by the end user community. And I guess some of what I'm going to say is probably just my own opinion, but I think a lot of these have been really useful and give us some insight into what end users are really adopting in the real world. But when it came to this particular one, certainly I had some concerns. I heard some concerns from other people. And I think we need to have a conversation about how the TOC community, like what value we would like to get out of these end user radars, how we could potentially, I mean, we're a technical oversight committee. So perhaps we should be exercising some technical oversight and making sure that these radars actually make sense as they get published, particularly, they're getting published in the CNCF's name. We should, ideally we would all be aligned and have confidence in these radars. So I think I wanted to just make sure we had the opportunity to air concerns about these and maybe try and think of some constructive ways that we could engage with the end user community to go forward and make sure that we're producing good versions of these things. So I think the next slide has some of the concerns that we thought of and I would love to hear from other people disagree or if they feel that we're missing things, then this could certainly be a discussion, should be a discussion. I believe that the fundamental source of issues for this particular radar was the scope in the first place. I think DevSecOps is such a broad scope that it's not very clear which projects would or wouldn't be included in that. There's a lot of different solutions in that radar that have some kind of association with security in a very loose sense, but I don't think it really serves very well to compare GitOps solutions against key management solutions or CNI plugins against CI CD platforms and that's what this particular radar seemed to be presenting. I know that when they brought up the whole proposal of the end user radar it was a deliberate decision not to include, not only to include CNCF projects, I think that's great because it means we may get exposure to some other projects that we're not really aware of and it gives us some ideas about what maybe is missing from the CNCF landscape, but personally I do not believe we should be promoting commercial products in these radars and I think as a kind of corollary to that we should be very consistent about whether we're including vendor names in those radars. So some of those projects, if we just skip back to the previous slide Amy, yeah, so there's some things in there that are open source projects, there are some things in there that are commercial products, there are some things that additionally have a vendor name associated with them, I mean you know, HashiCorp, a big fan of HashiCorp but why do they get to have their name associated with Vault and Sentinel where JFrog doesn't get to say JFrog for X, right, as an example. So I feel like there needs to be a kind of sense check on these radars when they're published. So I think those were the concerns I had about that particular radar and Riccardo, Dims you've got your hand up please. I can wait for Riccardo to go first. Can you hear me? Yes. I'll just do a disclaimer, I have my son here with me so if you hear some shouts, you know what it is. I was just going to say, yeah, I agree with what Liz said. I would also say that one good thing I find in these user radars is that we get like a kind of dump of what the end users are actually focusing on which products they're actually focusing and coming back from Kupkan last week, one thing I noticed is that they're, even if we have like a mix of different projects here, there is some value here to see for example a lot of people are using things like Argo and Flux and they would probably traditionally do something like SOPs to encrypt their secrets when they're doing GitOps and it's all kind of mixing things together and one thing that I saw is that a lot of people started using for example Argo with Vault and they suddenly start doing like rotation of their passwords just for free. And it's a huge improvement instead of just encrypting in Git their one-time passwords and never changing them. Now suddenly there's a solution that allows them to do like constant rotation of the passwords without having to focus on this. And when I look at this like multiple projects I agree with Liz like Argo and all these other projects in the same like diagram it's kind of confusing but they do start mixing in the ways end users focus on them. One of the things that came out in the discussion video of this end user radio was this that the focus on DevSecOps was actually at the expense of developer experience and maybe these things are kind of helping, I don't know. It's just I agree it's a bit confusing but they do start giving some value by connecting to each other. Do they give value through this diagram or is their value from combining. I don't think that this this diagram really tells us anything about using Vault to store secrets for Argo CD. Yeah exactly no it doesn't come out of this but they do like when combined and maybe maybe it's just not expressed properly having them like as single bullets maybe we should say more like what is used with what not just pumping names. James you had your hand raised the phone. Yeah I was trying to figure out if there is a GitHub repository where they assemble this and you know just like landscape we have a GitHub repository where PRs are made so I couldn't exactly figure out how this came about or how this specific set of things came about and whether that process is you know open where we can go in and chime. So may we answer that. Yes this is a form that is sent to the end users and it's basically an Excel sheet where each end user puts a column for themselves and then they for each of the products there they say the level assess trial adopts for the ones they actually touched and they can add new lines, new cells if they have other products to add. Got it then the follow up question would be like how did these names came about that ended up in the form right like so basically I'm looking for like what was the transparent process where we as the TOC could insert ourselves to provide some feedback at some point before it gets published that was basically what I was looking for. I think that is at the heart of the problem I don't think there is such a point right now yeah Dustin I think you are next. Yeah I mean I think that I think the process of people just adding things that they lump into this area is just a little bit weird I mean I think that if this was stories about how people had learnt to do DevSecOps better by using these tools it would be really exciting like if GitHub Actions has helped your dev and security and ops teams work better together that's a great DevSecOps story that I would love to hear but it's just this as I said previously it's just the kind of just having assess trial adopters and the data just makes like it just makes it a really weird list of things that appears to be fairly random you know it's just like you know it's different things these are not fulfilling one purpose at all there are lots of different pieces of infrastructure that you know it's not it's not clear to me that DevSecOps I mean you know I kind of given up on DevSecOps being a movement up or about people anymore but even if you think it's about tools it's a really weird set of tools but I'd love to hear people's stories about how these made their DevSecOps process better and Adam making the point in chat but isn't that more of a problem with the definition of the category and I as I said at the beginning I completely agree I think that the choice of category is or the scope of what these things are is unclear in this example Harry you were next I think and then Matt yes yes so first of all I just want to share the same feeling with Justin and I think the first impression when I saw this radar as a user I want to fit I want to say it's more like some kind of recommendation for example if I want to adopt the DevSecOps practice I will definitely look at it adopt section I will say oh there is Argo CD there's OPA there's Terraform but I think the issue here is there are many project here or many product here that are actually have very wide scope for example Terraform it's not guaranteed that if I adopt Terraform I can adopt the best practice of the DevSecOps and also this kind of issue I think also applies to many projects like E-Steel like there is GitHub actions it's still very hard for the user to get how can I practice these DevSecOps philosophies even I saw this picture so that is one that I'm thinking about maybe we need more five grand high terms in this project for example I think somebody mentioned by reference GitHub actions maybe we are talking about GitHub action security module or something like that instead of the whole GitHub action thing that is one fitting when I first saw these the names in this radar yeah Matt you also had your hand yeah I mean I concur with much of what's already been said but in addition just you know if I didn't have context right if I had just heard about the CNCF but I was into open source and I saw just this slide what does it tell me right does this mean that the TOC and or CNCF says I should only assess LinkerD it's not really ready for me to adopt even though LinkerD is you know a graduated project and foundational you know for some for many technology stacks right you know MTLS everywhere like that's a great building block but does this mean that I shouldn't use it yet because it's not ready like you know you could very easily get some very interesting takeaways in terms of like what does the TOC suggest I do to secure my cloud native systems so I think in addition to everything else that's been said you know maybe you know I think it's an opportunity for folks to contribute and you know in terms of like optics and just overall messaging visually like so every slide should be able to stand alone and this one is highly ambiguous for me in a number of different ways yeah agreed and I think the point you make about the implication that the TOC thinks this and we don't at the at the moment have any involvement at all yes exactly this is not from the TOC yes of course of course I just mean like I would just want to highlight it as an opportunity for folks with a background in you know how to do this and what the nuances are this is a wonderful opportunity to contribute and to broaden our tent and to bring on new voices to help with this kind of of an issue so yeah I don't mean to whine or rant I just want to highlight it it's an opportunity to have a background in this that makes sense Alex you're next hiya so apart from you know the ambiguity on this particular one because obviously deaf psychopsis is a weird conglomeration of different technologies I think the challenge is how do we because this some of the issues you mentioned around you know commercial companies versus open source and you know who's contributing to the list of products in the first place is not something which is specific to just this radar but it's also something that maybe happened in some of the other radars you know and should this even be a CNCF technology radar as opposed to the CNCF and user group radar and you know what's the sample size and how many people are contributing as well you know so if 20 organizations which are particularly in a particular area are contributing to this list it doesn't necessarily mean it's the generic list for everybody in fact it might actually be a misrepresentative sample right so the way it's branded right now it makes it look as if the CNCF is endorsing this and I think that's a fundamental problem because the CNCF is not endorsing it and it's not actually involved in generating the list other than facilitating the forum with the end users. Yes and Bob making the point that the site and if you looked at the link that someone posted earlier with the methodology that quite consistently calls it the end user technology radar CNCF end user technology radar but then the image does not I'm not quite sure where I pulled that image from but it's very it implies that it is representing the view of the CNCF Right and the issue with that ambiguity is of course you know then it gets misused in 101 other marketing blogs all over the place right Absolutely You downloaded it from the link at the bottom on the radar page where it says download PNG or SVG Yes you're right I did and it's different from what's displayed on the page Yes Bob's also posted an older one the secrets management one and that I think is a really interesting contrast because it is much more you know concentrated on actual secrets management solutions It does again include quite a lot of commercial tools as well as open source I don't know whether that is something that we should be concerned about whether we should I certainly have a concern that if the CNCF is seem to be giving kind of a free free promotion to some vendors and not others that will so bad feeling and I know it does say bad feeling because some of that bad feeling has been expressed directly to me by some of the vendors Ricardo Araveni you have your hand up Yes so I think everybody talked about you know how this is very broad and I think most of us agree that it may not be that useful for any user so I think the next I'm thinking of the next steps it would have in like a process where the TOC and the TAG engage with the end user folks in the CNCF and the end user community to improve this or to help out so having some way of engaging or maybe having some way of saying before this is published can it be reviewed by the TOC or by the TAGs? Yeah and I think Erin suggested at the beginning of the call yes Erin should we have the TAGs that these before they're published which is I guess if we have it as the CNCF for sure but if we want to just change it explicitly to this is what users are using and this is their experience outside of the CNCF blessing I don't know no matter what we publish even if it says CNCF and user group it's still going to be associated with the CNCF so I think it's hard to detangle that relationship Yeah I agree I wonder whether one of the good points we could try to get inserted or insert as a technical group whether it's the TOC or the TAGs right at the beginning of this group is picking a particular area to focus on maybe the TAG or the TOC could just sanity check that the scope makes sense and maybe propose a list of projects that we think should be considered because I think somebody else made this point that the way it works as users fill out the survey or the spreadsheet they can add new projects and the last person to fill in the sheet you might add something that actually everybody else is using but it wasn't on the sheet when they filled in the form and you could imagine that actually significantly affecting given the small sample number that could really affect the outcome so maybe having hopefully pretty knowledgeable group pre-populating that spreadsheet should be under consideration might help Matt you have your hand raised yeah I think so I participated in the end user radars as an end user and I think they are very valuable and your assessment that folks can just add things is correct I think perhaps we need to do two things really and to reiterate what you had said earlier Liz which I fully agree with the technical oversight committee can express some opinionated technical opinions which it does through the whole framework of projects but perhaps if there was this in its current form as an adoption metric or an adoption radar what are people actually doing and if a project name is there to me that means that they are running the project directly themselves via leveraging a vendor what vendors are being used out in the community to do to have the market that we hope to build and we have built where open source projects fuel an ecosystem of competitive friends that are leveraging the same underlying technology so perhaps the TOC should additionally produce a radar where the only thing on it are projects you know we could say hey for DevSecOps which I agree is a tortured category but for category name you know we the TOC think that these projects and highlighting graduated and particularly relevant incubation projects to me would be like where I would start so that technology leaders and architects that are crafting their own solutions using these technologies have some guidance you know at a high level about what the landscape of building blocks goes is if you will so I think filling that vacuum versus trying to make the one radar all the things might again be an opportunity to expand the TOC's perception as more concretely useful to solutions builders that's it I'd love to find a way that we can kind of because I completely agree it's very useful to get an indication from end users about what they're actually using but it would be really great if we could get consensus over what even falls into the same category we do have categories in the landscape they're not perfect in any way but they exist and they're maintained and they have a list of projects in with reasons why projects in there so that would be one possible task would be to say to the end users perhaps you could concentrate your end concentrate your themes on either an area from the landscape or possibly even a subset of an area from the landscape yeah we could also empower them right like we could actually just generate this you wouldn't have to be subjective it could be driven from the landscape just a more zoomed in view of the entire sector that might be cross cutting or maybe if the categories don't fit but so these end user technology radars or adoption radars might be prefaced with you know from the TOC like a format like for this sector here are the projects the graduated ones, the incubation ones sandbox I don't know probably not but I don't know but at least to preface the discussion of vendors which are thriving in an expanding market that are being adopted and then it's not subjective it's just it's data useful data I guess that does raise a question of why and maybe just because nobody thought of it but why the end users didn't start from the same direction of projects that were together in an area of the landscape if I could just be so bold after this and go back to raising hand but I think the reality for many end users is it's difficult to assemble the talent or the bench that has the capacity on top of keeping the lights on to really go deep and do these assessments that's why as an end user for me probably the most useful artifact has been the due diligence reports and I don't know what it's going to look like but you know many of my colleague he requires a base level of technical complexity and a deep enough technical bench to do those to start from the projects so it's a quick start here which might really accelerate Yeah I think I did have another slide that had some broader ideas I know that was all right it was just other considerations about there's one comment that I had about this is that having a separate waiter for open source project may not be necessarily the best because there's a lot of gaps in the open source projects in the CNCF and some of those gaps are covered by some vendors so I think it might be useful to show that some of those things are really covered by vendors but yeah just a comment Yeah I think it was you specifying which one's open source and which one's on as part of the radar that sounds pretty useful to me Yeah Maybe one thing we could do also taking your sentence there in the slide about getting the TLC community input into the end user radar one thing that would be useful is also the other way to have more input from the end user radars into the TLC community because all these interactions between the different projects and how they are used together would be actually very constructive to better understand the landscape and even gaps in the projects that in the CNCF Yeah and I actually think that's one of the potential things that we can learn from these radars as you know where we see there are non-CNCF solutions that's super useful input for sure So I think in practice some of the things that I've heard people suggesting and I think these make sense were that at the start of this process the end users maybe the I'm just going to say the TLC community I don't know whether it devolves to tags or how we do that but that we could kind of help preface that end user discussion by saying okay if you want to look at I don't know I'm just going to pick storage here is what we currently have on the landscape for storage this is the set of projects this is how we currently see them from a Sandbox incubation graduation perspective maybe there are other projects you would also add to this set of projects but use that to kind of seed the set of projects under consideration the labelling of whether they're open source or not and again that could be done in the spreadsheet I think we're at risk in some cases of ending up with the open source project and the commercial implementation of that project potentially even being in different parts in the radar and that would be kind of odd and there's also the concern that a lot of companies have their open source project and then they enhance the open source project like open core to make it some solution that they can actually charge for do we want to specify that there in the radar or not that would go beyond the scope of what open source is I mean just at a more generic level though if the end users are voting what they're using that may have some overlap with the CNCF in terms of projects that the CNCF is involved in but actually the overlap might be minimal in many areas I mean I don't think it's necessarily a given that the radar is going to be CNCF projects I mean it's going to be they're probably going to be products from vendors or from projects which are in the broader landscape but that doesn't necessarily mean they're CNCF projects and in fact that probably just anecdotally has to be guessed that half or more easily are not CNCF projects right so I mean so that's where the whole kingmaker thing comes into play because effectively the CNCF ends up promoting things which aren't even CNCF projects right yeah and I suppose if I may if I'm a CNCF project and I see a competing non-CNCF project being promoted by the CNCF I may feel somewhat aggrieved Matt you have your hand again yeah I mean I think in terms of the kingmaker concern one way potentially to ameliorate that but also you know set what we do want is that the CNCFs really you know they're deliverable if you will their value is fostering the ecosystem of vendors that are using these projects in all manner of creative ways so as an end user you know what I really look to from the CNCF is to say okay for the vendors that might deliver value to my business with whatever level of UX and optics and usability you know is valuable to me based on my needs as an end user you know I want to part of my diligence you know as a decision maker is to ensure that the vendors that we're using happily right because it's hard to run some of this stuff or it's complicated whatever that they're in alignment with the overall like open standards and protocols and core technology building blocks so that you know fast forward a few years I'm not you know having business critical things that are not that are then reliant on something that's not in alignment with the overall architectural and technical roadmap of the CNCF umbrella and family if you will so you know for me it's an opinionated thing from the TSC doesn't mean like we're making it's not a king maker thing like now it shall use this or we bless that but it's here's the building blocks that come from this open source communities that we're fostering and then here's the vendors that are using these in overlapping and different and creative ways that's still in alignment with the overall trajectory of the cloud and so I think it's nuanced but the governance piece really is providing the framework like that's the output that's the value of the TSC to let the whole thing exist you know with some guide rails so I don't know if it's messaging or if it's just making it clear that you know the first said maybe the preface that comes for all of these radars is like here is the here's the source material that these vendors pull from in the form of project and then some of those projects can be used directly and are and others you know can be used by having somebody else run it for you or delivering sort of a batteries included approach but understanding the relationship between those two kind of building blocks and concrete usable you know by a business today now I think you know is more than the sum of either of those two parts right put together yeah yeah I like that was supercharged the radar you know what I mean and it would make it like a go to you know and then that like feeds this virtual cycle that expands the breadth you know of the CNCF in a way that meets people's needs where they are people being end users and that feeds more vendors and all of it but isn't that sort of like landscape in a way or maybe landscape has something that the radar doesn't or the landscape is too broad and the radar is just kind of like just you know like kind of yeah that first part is like a filtered subset of the landscape right you know if you actually actually put this in a talk just a coupon last week you know like the eye chart of the entire landscape it's just like it's too much to rock at once so it's not perhaps used as much you know I would advocate you know as a starting point just like make a filtered subset that's auto generated that generates all of these radars and then they're just available as a template framework you know to the end user radar efforts which I don't want to disparage right you know they're wonderful they're the best data point of God yeah and I don't want to I don't want to end up with it looking like the TOC is trying to force the end user community into a certain opinion that they don't actually hold I just feel that we well yeah in this particular example we probably would have suggested a different focus and we might have suggested some additional projects and I'm sure we would have used the landscape as a as an initial point to see that and I think maybe not even just the landscape because there might be you know projects that the tags are aware of and it would be really I'm thinking for example of runtime where we know there's a ton of wasn't projects kind of bubbling up and knowing whether or not end users are looking at particular projects so out of that set might be really interesting just as an example but yeah I feel like we do need to seed some of this Tim's so as an organization we you know the end user community and TOC and other people what we have in common is the landscape which is structured information that we have I think if the end user community doesn't see stuff that they are using there then it should be added to the landscape and then making it complete and then like Matt was saying they should be like a filter on top of that which is what is called a radar right now but it is an actually a survey right like it is it is the reflection of what people are doing right now or what people are thinking end users are thinking right now and it's not like a recommendation going forward either right like it's where they are right now in terms of like yes some people have adopted issues some people have not and it reflects what they are using right now so if we shift it around to say hey look here are the end users this is the landscape out of this landscape here is the list of things that they are using and these are the things that fall into different categories right so if we set it up that way and showcase it as current CNCF end users this is what we got from them this is what they are using in the day to day work rather than what it is right now so if we can change it around in this case like whether it is the tags or the end user all they will make sure is that the landscape is complete and then any voting stars whatever is on top of the landscape and a subset of the landscape which can then be published periodically right so they would potentially be running the survey based on a section of the landscape right or they could pick like multiple sectors too like this year we are going to do storage and we are going to do observability for example right like and have two radars published and I think it could be smaller than that potentially I thought secrets management was really interesting because it was a subset of what we it is almost like magic quadrant kind of scenario right like so yeah if so maybe this is a question for end users like Ricardo or Matt having been through the process of these end user radars do you think it would work to start from the landscape and work from there I can say something I think it would work but we would lose this additional feedback of like things that are not in the CNCF that end users are integrating into their deployments like this would be maybe valuable input for everyone actually what I think would be great would be to have more information than just adult assess trial but this is maybe a demand for the end users that would probably reduce the number of replies if there's like too much effort to start writing these things right now it's quite quick to go through this process maybe that's why also the results are pretty good so maybe the answer is to start from the landscape or section of the landscape have the ability for the end users to add these additional things that we're not currently listing for whatever reason and then we can look at that and say well should we be should we be adding those sorry I had a Bluetooth kit problem yeah I think starting from a subset of the landscape to me it's less important that it follow the lines and the categories in the current landscape but over time I would think that we could use if there's a big difference then perhaps we want to make more subcategories in the landscape but certainly starting from just like these are the set of projects because it can be a lot if you have to come up with a platform and a roadmap and then put it into production and into practice meaning there's some boring skills on your teams and whatnot it can be hard to keep track of all the things in all the sectors that you're responsible for so I really think again if there were just a data driven these are the things in scope for this sector as Liz and others had been talking about that would provide that context and a mental model and you can actually think of visualizations that kind of like show a graph with fan outs for what vendors are leveraging which technologies and we could even engage vendors and they just like they're proud to say I'm a CNCF member and a vendor they're proud to say I contribute to this project or that project that's something in the tag observability we have as a work stream that is no small feat so if there were some top down guidance to meet efforts already underway to make those mappings a little more clear it could be here's the context and then here's the end user radar which is again easy to generate and low friction for the reasons that folks had said that can remain the first part of it shows that mental model so that when someone looks at that radar they can make the connections that they need to make about what's important for them which technologies are more or less important to the bottom line of the business thank you Alex you have your hand yeah so I actually think linking into the landscape might be a good idea but I think the key thing in all of this is going to be about the quality of the data and that's both in terms of which products get looked at whether they're open source and commercial and whether they're on the CNCF landscape or not and secondly who's providing input to those surveys because it's not just the number of end user companies but it's also who in those end user companies were polling so if there's a nominated person for some of these companies which I suspect is how this runs then some of these organizations are huge organizations with tens of thousands of engineers and it's unlikely that any single person is actually going to know exactly what's happening in the whole organization anyway but the key thing is if it's these these end user radars are valuable if somebody can look at them and actually make useful decisions based on this and I think that's the value of polling the end users because we find out what the end users are actually using but if on the one hand the data is maybe too specific or too focused because we're not polling the right people or we're not including or excluding the right projects then it either becomes too broad or too focused and then people don't find that useful so this is part of the equation the data is the most important thing and how we keep it up to date because if you're going to go through the phase where you actually say these things definitely use and these things only evaluate then we should probably also specify some sort of review or at least get the end users to review after a period of time because some of those end user radars are now a year or two old and we're kind of condemning certain organizations one way or the other which is also a factor here Do you think we should actively be archiving these after let's say a year? Potentially I mean it would be better if we were doing this under a CNTF banner we should probably be reviewing them or at least say provide some sort of information as to why they're current or not current otherwise otherwise data becomes expired and poor and the poor data is the thing that I'm most keen on to to make sure it's appropriate because 30 people from 30 individual companies which might only be responsible for specific themes in those companies doesn't necessarily mean it's actually a valid set of data it could be very focused or we could be missing entire chunks right but you know it's hard to tell and we're then sticking the CNTF stamp on it to kind of indicate that it's supposed to be fair but it doesn't necessarily mean it's right Ricardo also having a conversation that I think is relevant to this about how sometimes we have graduated projects that maybe appear as assess which is kind of weird and confusing but as Ricardo points out it's weird that that happens because we interview end users for incubation so there must have been people using these things but maybe those people who we spoke to at you know for incubation or graduation reviews are not necessarily aware or involved in these radars and we're getting a very different picture well you know there's a cost to entry to being an end user in the CNTF right people have to pay for that subscription so you're getting people who are you know committed to the ecosystem and wanting to help the CNTF but that doesn't necessarily mean it's it's you know they're end users that we would have interviewed in the incubation stage okay I think we've got a lot of very useful ideas here I don't know how we resolve some of these like how do we what do we do when we have a graduated project that is not appearing you know when it's appearing in the SS phase I don't know that we have a good answer to that but I do think we have some suggestions here that we should take to the end user community and Katie and ask for input at this earlier stage I think the things that I'm taking away now is starting from the landscape as a first point of sort of listing what should go into the survey agreeing the scope of that survey and crossed my mind that this is an opportunity for the end user and TOC communities to work together which we don't have enough of those so I hope that scene is valuable from the end user's point of view as well I think we should raise the possibility of archiving them after some period of time because yeah, if they're two years later and we're still saying this graduated project is in assess that's going to be very misleading and we should try and look for ways of keeping these more up to date I think we need to be more prominent about the fact that these come from a set of end users and maybe indicating the number of respondents like including that in the graphic so that when it's downloaded it's more clearly labeled as hopefully representative but nonetheless it's a snapshot of a survey of a certain number of end users at a particular point in time the last one I had was don't position this as you should do this it's more of like this is what we have found so changing the way to look at the data to reflect that it is a consensus of end users up to this point and it's not like something looking forward right, it's looking backwards that's a great point that those words assess trial and adopt sound imperative, don't I? Yeah, it does I think there was another point about connecting the dots between the vendors and the open source projects Yes I see that as providing more information in the radar because when you take a snapshot or when you see the visual there's a lot of questions and you try to think about how to address those questions and that first visual when you see that and connect the dots it's something to keep in mind Yeah, I think that's a very good good point to add Alright, with five minutes to go anyone have any other points they want to make about this general topic? I'll just make one small point I kind of I kind of probably might have come across a slightly negative here I actually think the end user radars are really valuable and useful and there's probably more right in them than you know, wrong and so this is one of those things where the last 20% was probably 80% of the work and we should think about how we factor that in but 80% of it is actually pretty good and very, very useful and there's been a lot of positive feedback for it too so yeah, I didn't want you know, I don't think we should throw the baby out of the box or so to speak And then the other one that I also thought about was Ricardo's point of how we use these things together it's almost like this is our stack like so if you if you let the end user say design their own stacks like I use these set of technologies together and then basically what on them saying hey, this is the most popular one you know, that might be another way to look at it. This is the whole reference architectures idea, isn't it? I use Argo CD and I use Alt and then I use Litmus with whatever right, like this is my stack that I use in my company right, like so that would bring out these additional you know tidbits instead of like if we just have a radar for storage it really doesn't make any sense but if we work on like here is the stack here is a different set of variations of stacks that might be something there too. Yeah, I love the idea of this and in fact it did come up as a suggestion in the governing board that could we be doing more in terms of reference architectures and I think this has come up in the TOC before and I think everybody is in favour, just a question of finding the time with the caveat that we shouldn't have one reference architecture we should try and have and ideally they come from end users and this is what end user A is doing, this is what end user B is doing, this is end user C again it can be a snapshot in time but if we could get those you know kind of case studies I suppose in the form of reference architectures that would be amazing. Some plus ones. Bob's saying I like the idea of a user story in a stack and describe why and what they're using. Yes, hearing decision logic on why they chose a specific thing would be useful completely agree. Really good. Alright, I think this is super useful I'm sure because people like Ricardo are representing end users on the TOC but we'll also I think in the TOC just write up some notes to kind of propose to the end users in a formal sort of a way so we've got something written down but I think we can take this to the next level of usefulness together. Thanks everyone for all your ideas and input this was super. We didn't have anything else did we? Nope, that was it. Brilliant, take care everyone. See you soon.