 completely blanking on the name but that sort of it simplifies it a little bit for the employer to do a sort of an every six months sort of skills sort of assessment. So I think that you know I think there are challenges there but I think sort of trying to make it as little of a lift on the employer's part as possible I think is definitely the sort of the key strategy for employers and you know we're always sort of talking about sort of trying to maintain as light a touch as possible with the employers while still of course you know you know maintaining sort of a minimal sort of level that they need to be sort of involved and to engage to do the work well but in terms of what we're sort of asking of them try to keep that sort of as light touch as possible while still getting the information that's a very fine line and a tricky balance to take on but I think keeping that in mind I think is definitely something I would strongly sort of encourage and if there are things that you know and I think this is where you know sort of to the point that John sort of has been making around sort of really allowing your sort of research questions to drive your data collection you you know especially with employers who might otherwise be might find this to be burdensome you don't want to collect data just for the sake of collecting data and make asks of folks just to have data that you don't necessarily need so I think that's where sort of that that that planning on the on the front end and really sort of being driven by the research questions can really sort of come in handy so at least then you sort of know we're collecting data that we need to answer these questions that have been determined to be very important so in terms of that and what it will do is actually give them a final competency assessment that when they complete it will go directly to their apprentice so they'll know that they've achieved you know success in the apprenticeship next slide the um the final tool that we use in my apprentice really is what we call a mentoring platform. I've talked a lot about apprentices and I've talked a lot about employers but with our community partnership we call it the Charleston Regional Youth Apprenticeship Program because it really consists of not only the college that serves as the intermediary and educator but also 25 high schools over 180 employers and a lot of different economic development folks who actually have accesses and as I said earlier we'll provide access to anyone who actually wants to help us mentor these folks so they'll be successful so if you actually look at the screen you know you can see what an employer would see in terms of at least their apprentices so they can actually access and see all the information that I talked about um it actually we actually give access to our faculty staff because we're going to award college credit um for some of the work that they're doing on the job so we'll be able to give this information to faculty who will be able to assess the information and actually award a grade um if if the student wants it and also if the employer wants it and also we give access to our high schools because our high schools help us mentoring mentor the apprentices if they're having problems when they're getting through the apprenticeship program next slide so that's my apprentice um you know it's a wonderful tool that we've started using this year um and it's working really well for us um it's pretty cool to be able to see uh winter apprentice actually not only finishes a class but actually can see when they're working and also what they're doing on the work so we can help guide and mentor the actual employers to to actually complete um you know maybe they need to work on some other areas and anything so it's a wonderful tool that is available out there and I'd like to thank you for your time and um my the next presentation that you'll be going to is leveraging youth apprenticeship data to improve program and policy but thank you again and I hope you enjoyed the presentation uh thanks Mitchell that was uh that was fantastic and actually dovetails nicely into our conversation about data uh so there was a lot looks like we also lost our moderator we're going to make sure that she can come back in just a second my name is Kate Kramer I am the deputy executive director of advanced CTE um for those who are not familiar with us advanced CTE is a national nonprofit um that represents state directors and other state leaders leaders who oversee career technical education across the country in all 50 states DC and the US territories across secondary post-secondary and workforce development systems um we have so enjoyed being part enjoyed the partnership and the ongoing learning of the last two plus years of being part of PIA and look forward to um the future as we continue this work over the past year advanced CTE has been leading a work group with representatives from over a dozen members of the PIA partnership focused on data and data quality which has been facilitated by my colleague Austin Essis who spoke on a panel yesterday today we're going to be speaking um digging into that those issues um and elevating what are some of the challenges and what are the real opportunities within data to advance youth apprenticeships within an eye towards quality and equity um so I'm very excited to be moderating today's panel um with a really distinguished set of speakers um over the past two days we have heard from speakers and panels on critical topics such as attending to equity within youth apprenticeship ensuring that youth apprenticeships are well integrated into our K-12 and post-secondary systems ensuring their systems alignment between youth apprenticeships and other key state and local level initiatives and more none of this work can happen at scale or with equity in mind without data guiding us along the way data is foundational to the work of developing and scaling youth apprenticeship but it's also real challenge to get right given the complexity of youth apprenticeship programs and the many players needed to make youth apprenticeship happen that's why I'm excited that we're going to dig into this topic today with three leading experts on data youth apprenticeship who come from those different levels and have different perspectives on both youth apprenticeships themselves but also how we can be building out those data systems structures and processes so we update uh Dr. Amy Firestone the vice president of apprenticeship Carolina for South Carolina technical college system who has unique perspective as now at the state level previously at the local level and as well the federal level in her previous role at the U.S. Department of Labor a purve Medrotra service is here to hear as director of research and analytics based in New York so it's bringing kind of that local perspective and in terms of that that intermediary role that is so critical and then Dr. John Wicker that an education associate in the career technical education and STEM initiatives work group at the Delaware Department of Education so our goal today we'll see if we can get it is to make this as conversational um to really have an open dialogue on this critical and complex topic so I encourage you as you've been doing over the last couple days please use a chat to ask any questions um along the way we'll have we'll make sure to have a dedicated portion of time to really be able to dig into that um so we're gonna kind of start with a nice easy softball question I'm going to ask each of you to kind of give a little bit more about yourself introduce yourselves um as we address as as you answer this first question but really from your perspective from your viewpoint why is data so critical to advancing your overall vision for high quality and equitable youth apprenticeships open up whoever wants to go first well I guess I'll I'll dive in um so yeah thanks thanks Kate for that introduction and yeah great great to be on this panel so you know I mean I I sort of view data sort of broadly and certainly within you know the area of youth apprenticeships sort of from a lens of really sort of helping us answer a lot of sort of critical sort of questions and and you know I many many years ago actually majored in in journalism so I so it might be why I sort of come at it from this perspective of kind of you know the who what where when uh how uh and why and and I think the data can sort of really answer a lot of those questions in a way that that sort of Kate to the point we're making can sort of really sort of serve as the foundation of building and scaling uh youth apprenticeships so um you know so you know the who of who we want to you know who are we most interested in serving uh who should we be uh engaging in partnerships with um you know the what of sort of the specific activities that sort of the data sort of reveals are sort of necessary for us to sort of engage in um the um the the the opportunities that are available sort of looking at different sort of industry sectors and and occupation types um you know the where of the the geography of where the opportunities are available where where again is our sort of target sort of population in terms of who we really sort of feel like needs um you know could could benefit the most from from from from youth apprenticeships um you know the I think the why is is is is sort of sometimes overlooked they're really sort of critical and kind of making the case for for the importance of this work and especially when engaging with partners who might otherwise be um you know if not reluctant then certainly maybe not as as eager as we would want them to be I think sort of that that case making can be really sort of important and using data to sort of say look this is why this work is necessary and important and can be impactful and then I think that all sort of drives the sort of the how of how we sort of engage in this work um and and and what we're you know sort of you know within this sort of you know broad sort of umbrella of youth apprenticeships which you know certainly you know carries certain sort of characteristics sort of across the board but there's always going to be sort of different contexts in which the work sort of takes place so sort of how we sort of you know respond to that context and and really sort of engage in the work I think is really driven by answers to all that sort of the who the what the where the when the why and and that's what I think you know for me data really sort of helps clarify I guess I'll go next um so I'm coming from this at a statewide level um as the Vice President of the Division of Apprenticeship Carolina we work with all 16 technical colleges in the state and local companies on developing and registering apprenticeships so you just saw Mitchell Harp and he's from Charles um Trident Technical College which is one of our 16 technical colleges so they're one of our shining stars um in the 16 but we have many others that we're working with to try to get them at the same level where Trident Tech is with the Charleston Regional Youth Apprenticeship Program so data of course we're held accountable by data not just at um the technical college system office but at the state and at the federal level so it is really important for us to see the impact of youth apprenticeship because we've invested a lot of resources in working with companies not just in the technical college system office but statewide with all of our colleges so it's about accountability as well um we're spending all this time we have all these resources anyone really benefiting from this program so um that's why data is really important to us and you saw the app that Mitch just um presented on that is something that we hope to bring to all 16 colleges in the state because that is a piece that is missing um but it does require accountability it requires funding but in order to even get that far to have an app we need very basic data on how many companies are participating how many youth are enrolling where are they from what industries what occupations we won't know the impact to even get an app or any other tools to advance our data without knowing just the basics so I would say that the vision that I have for South Carolina is to grow capacity to be able to collect the data and hold other areas accountable for collecting the data because I can't collect it all my team can't collect it all we rely on local um the local technical colleges and the companies to help us with this process so that's just a very quick overview of where we're at looking forward to chatting more about this so Kate I love that you you say this is a softball because I think it's really the first of a series of really hard questions for anyone in this space um so from a Delaware perspective you know Delaware really views data as it's a way to tell our story and it's always going to be good bad not ugly it doesn't it doesn't matter you're always going to be one or all of those those things at some point in time but it also helps us share and express our commonality and our shared values across systems and so for us it's going to serve very much as a guide in terms of system development and augmentation program quality advocacy and funding relationships as well as measuring whether we met our goals or not and so what you know I really love from my fellow panelists like the mention of accountability here because in Delaware we're actually trying to move past accountability models that are pinned to our data structure and so we want to move more to a data model where it's about innovation and outcomes based funding that results in family sustaining wages in recession resistant occupations and the ability for learners to navigate post secondary education networks to up skill cross skill you know advanced career trajectories at minimal cost so we see data kind of underpinning all of that work and we actually hope to use our data system to upend what we consider to be kind of traditional funding models so something that's been on our radar is when we look in the youth apprenticeship space especially right now we see some similarities to systemic challenges with historically black colleges and universities in terms of local funding and this idea that you easily get into this funding churn right there's an opportunity there's a funder I get it I'm supposed to main a program and we're off to the next one and so we want to build relationships where we're approaching models with funders that they're helping us on the front end to stand up systemic changes but then providing funding into endowed funds that can help us sustain the programs over the longer term and then coming back again and saying great so now we want to go to version two and we're going to do up front and then on the back end we're also going to look again at putting money into an outcome based pot so this endowed pot being based on us meeting the outcomes the funders their investment is helping us get there and so we see that as a way to not just stand up our systems but to engage in the dialogue the storytelling the relationships and underpinning the entire thing and if you can stand up a structure like that we think you can also see advocacy where maybe there are groups of private funders and philanthropic funders that are willing to match state funds invested into systems right because again we're all into the same data system here and so really what we think that will result in if we can pull this off is for Delaware at least a youth apprenticeship system that results in state regional national and possibly globally transferable credentials and we think we can do all that through the right data system what I love about all those answers it just shows as I said this is such a complex issue and I think that just shows that data is foundational to all of it but there are so many different purposes I mean I heard which I think I'm so glad that it came up the case making the advocacy that given we're still at the beginning of this youth apprenticeship journey and having that data tell the story to engage different audiences accountability to create funding models to make sure you're serving and reaching the right the right systems I think it just speaks to there's so much out there I think something that we often struggle there's that kind of the cliche of we're really data rich but information poor so how do you move and transition that the data you're collecting and have you can really be make it actionable whenever that action might be so I want to I want to dig in this a little bit more because given we just went through a wide array of different purposes and and rationales for data that this is hard it's hard to collect to get the right collection moving and we know there you know we're still pretty early on and we've had a ton of momentum but pretty early on in growing youth apprenticeship movement and we're seeing new programs and new policies come online constantly so I'm curious what your recommendation is for those that are launching a new youth apprenticeship program either at the local or regional level or a statewide effort as some of you are speaking to where where do people start what is the most important types of data to collect at the outset and then what might you add as you build that capacity build the capabilities get the buy-in but like what is the critical if you don't collect this at the beginning you are going to set yourself up kind of for trouble or challenges down the line and I might have Amy star because you kind of talked about the basics like what are those basics and if you want to go on that and then we'll bring in a perv and john to the conversation yeah well thank you k I think the basics are a good place to start and I think in South Carolina we have conquered that area and we're looking at kind of the next level and integrating different data systems to get to the next level but in terms of basic data first I'll say that all of our youth apprenticeships in South Carolina are registered with the U.S. Department of Labor so that is really our benchmark for knowing what companies are registered what the programs are what the occupations and all the different nuances for each registered program that do well data will give us all of that information so we don't have to look elsewhere it also tells us where in the state that company is located and the age range and all of really that high level information so having all of our programs registered with DLL gives us that opportunity in addition Apprenticeship Carolina just launched a pre-apprenticeship infrastructure so now Apprenticeship Carolina will be reviewing all different entities that want to do pre-apprenticeships and we will be tracking and collecting data so we have a little bit more flexibility in what we can ask from the technical colleges and K-12 and other organizations but the basics on our end would be knowing who's in a registered apprenticeship what company what's the occupation we're in the state and you know age range um without that they could be in an internship they could be another program that we wouldn't deem to be a high quality youth apprenticeship because all of ours are like I said registered with DLL so that's that's really my take on it based on the success that we've had and we want to build upon that but we'd have to really look at our resources and integrate with other agencies to get to the next level and that's something that we hope to work on this year maybe you can give a local perspective right come from a programmatic level kind of what what are those key foundational pieces of data that need to be kind of built into the foundation sure yeah so I mean I mean I think you know everything that you know sort of Amy touched on or certainly you know sort of key sort of elements of any sort of data collection plan related to youth apprenticeships I mean one thing that we did at here to here where we sort of incubated career-wise New York the apprenticeship program sort of built off the career-wise Colorado sort of model is we actually sort of engaged in an exercise where we really sort of really sort of kind of utilizing sort of the theory of change model so we're sort of starting with with our sort of goals and in this case in our sort of context locally I mean we're really you know sort of looking at apprenticeships as you know very much a part an integral part of a broader sort of push to sort of you know braid academics and work much more in a much higher quality way than what's currently been done so as we sort of think about the sort of larger sort of goal of kind of you know redesigning our talent development systems in a way where apprenticeships are playing a really huge role we're sort of identifying sort of what the goals are and what the sort of the vision is and then kind of mapping backwards and sort of really understanding well what are sort of the strategies that we're that we're trying to execute in order to sort of get to that vision and specifically with apprenticeships you know where do apprenticeships sort of fit in into this larger vision what are we hoping is accomplished with apprenticeships in addition to the of course the young people that that we're serving gaining skills and gaining credentials and being on the path to a to a family sustaining career which is of course sort of priority one but in order to do that I think the the the extent to which the needle needs to get moved for for educators and for employers in terms of the way they think about you the apprenticeships and in terms of the way they prepare students to to be ready for for an apprenticeship by the time they're in our case in 11th grade um so so thinking about those sort of strategies for sort of how we we sort of you know again sort of make the case for youth apprenticeships but then also the ways in which we sort of shift thinking to really better sort of accommodate value youth apprenticeships and then from there that sort of really gets us to okay well then how do we know what we're doing is really sort of working in those in those regards and and that sort of leads us to a whole series of sort of questions research questions that we sort of want to answer and which leads us to okay well now what's the what's the information we need to collect to to answer those questions to see if we actually are sort of moving the needle in terms of some sort of local momentum around apprentice apprenticeships again coming from from schools and from employers because in our case you know we we we of course are doing this work sort of in partnership with the with the Department of Education but it's not sort of a a state or sort of a broader sort of city sort of program it's very much sort of happening sort of from the ground up so in that case there you know I think with that being the case in particular we really need to sort of move sort of the hearts and minds of people to really again sort of get them to not just accommodate but really sort of value the role of apprenticeships and and so that's a big part of our sort of data collection effort as well as sort of trying to sort of gauge you know how much are we moving the needle in terms of partner you know school employer engagement in addition to all those things that AB mentioned in terms of some of those sort of basics around well you know how many folks are doing you know how many folks are doing it what industries and all that type of thing. Don I'm curious from a state that obviously has a very sophisticated data system kind of where do you start where do you move and I'm gonna I'm gonna pose this just because it hasn't come up yet I also want to kind of put equity into this right we haven't really talked about in terms of making sure you know how you're start getting data how you're setting up that in the front and that's something that I'm sure Delaware is attending to but if you could add that in that'd be great. Sure so I think I want to start with our approach to data is very much at its core right data has to inform our system and our evaluation has to be actionable has to be minimally burdensome to collect right like those are kind of core principles for us in terms of getting the data and we've learned a lot about what not to do from actually previous data systems that we've looked at and so I think to give you some context when we started looking at the data we had in place we realized we didn't have what we felt like we needed and so what we did under our WIOA and Perkins combined state plan is we actually brought multiple lines under the same umbrella right in the same delivery policy so we pulled together credential apprenticeship and degree programs right into the same kind of system driver and what that allowed us to do was have the influence in the system to reset the data system because we pulled all the effective parties right into the same sphere and so what we're actually doing is we're going we're designing our data system around research so we're not looking at saying what data do we have and then what questions do we want to answer we're actually pulling in local state and national partners and saying to them what are the research questions that you would like us to be able to answer that you think are value in the space and then we're collectively designing the entire system around that so it's a very kind of different model and our our goal here and this kind of goes to your equity conversation Kate if you need me to go farther let me know so around that our goal is to just get like 80 percent of this right the first time like the core and get it really right the first time and then build out but what we're doing at the department is because we have the ability to broker and manage data agreements what we are doing is we are handling that lift and that's how we're matching in the snap TANF and all of these other linkages as well as post-secondary clearing house and other systems data so we're collecting from all of these partners we're matching all of this data we're deidentifying the data and then we're pushing it back out and supporting our partners around these routines and so it's a very kind of different way of approaching it we're making sure we can answer the questions up front versus later finding out maybe we can't answer what we wanted to and the other thing we're doing is we're supporting our local partners by using Perkins reserve funds in order to cover the cost of system augmentation so we've already found system limitations at the local level we're using our Perkins funds to pay for systems enhancements and to support our partners through the transition again so that they can execute their commitments with the minimum amount of lift and so right that's going to give us and again I mentioned earlier kind of our outcome space piece so we've instituted some pieces there around the enrollment and concentration of economically disadvantaged individuals with this which is a much larger conversation but um I think to your point we recognize that there's a real moral imperative behind the work right and so for us this data does not lead to what I mentioned before which is the family sustaining wages and recession resistant occupations and the ability to navigate these systems the data is probably not worth collecting for us and we won't attempt to collect it I want to turn back to a perv because my next question and I think John you set up really well is this like local and state role and the balance between the state leaders and building up the data infrastructure and setting the indicators that need to be collected and setting up accountability and then the local the program role right in terms of actually collection and reporting up and engaging all the partners and it obviously looks different from program to program and state to state but I'm curious a per from your perspective right as you mentioned you're you work with your local department of education but you're really kind of your own program it's not in it so what do you see as a state role um I don't know whether if john hit it or not or if you have a slightly different how that balance between what the state needs what guidance and support you from the state and where you need that flexibility at the local level or the program level yeah I mean I think at this point I mean we're we're sort of very early in our sort of you know youth apprenticeships sort of journey and you know so yeah so at this point a lot most of what we're doing has been quite sort of independent so um and I know the the career wise folks now are are are working with the state on on getting apprenticeships registered and and all of that um so you know so from from my perspective at the moment I would actually you know love to have more guidance you know from the state which I know is something oftentimes people will sort of want uh you know the opposite of that more sort of autonomy um but but I would love to actually be able to work with uh work with the state and and and sort of you know integrate what we're doing with youth apprenticeships into their sort of you know a broader sort of apprenticeship uh sort of data systems and and other systems um to get a sense of you know what what what it is that they're collecting um and and and so that we can sort of share information and and share platforms and and things of that nature so you know right now the work has been quite sort of yeah we're we're we're we're somewhat sort of left to do the work sort of independently which is which is really great in a lot of ways because we can sort of make those decisions and we can sort of formulate the research questions and then say okay this is how we're going to go out and collect this data um but then at the same time it you know one it sort of makes it a little bit more difficult sometimes to collect when you don't necessarily have the authority uh that that that estate uh you know um entity um would have um and then also it's just you know I again I think to have that sort of partnership at that level where where information can be shared and where and where you know like platforms can be can be utilized I think could be um could be really meaningful and valuable so I'm sort of hoping that we that that we get there um but and for now I'm sort of grateful that at the moment we're sort of setting uh the terms a little bit in terms of what kind of data we feel like is really valuable to collect and and what we're sort of prioritizing but I'm but I'm hopeful that'll be more of a partnership sort of going forward. Thanks thanks for and Amy and I mentioned earlier you've kind of sat at at all levels so I'm curious um where you land on what that balance is between the state local and I guess if you want to if you want to attend to it the federal role we could bring them in as well since we've already been talking about some of the federal requirements. Yeah I guess um I do have an interesting perspective and I'm trying to navigate it myself to see what we can do to improve data on apprenticeships in South Carolina and I'd say there's a lot I know this is another question you have about challenges but um there are a lot of barriers and I think both of both Aparva and John alluded to those barriers in terms of not being connected to the larger state system of probably we're talking about the um not being as connected with apprenticeships and how that data might be collected and and John you're very data focused but maybe in terms of the individual programs happening um you all don't have as much of a connection so we're kind of all of it here with apprenticeship Carolina because we are the statewide apprenticeship intermediary so um I would say having access and having the federal data from our companies is extremely important because um that's data we can use for different audiences like other companies so I think I'm kind of talking in circles Kate but I think the bigger question is the audience of the for the data and there's data that we need from all levels to be able to speak to different audiences so we have um it's a new ROI study on youth apprenticeship that came out last week which I'll share with you Kate and the PIA network to show um over the past five years how many companies have experienced a positive ROI from hiring youth apprentices that data isn't valuable to show other companies so they'll invest in youth apprenticeship however when I meet with school counselors or other um K-12 individuals that data does not have as much value but showing the outcomes for students such as degrees of tame a lot of the stuff that you saw from presentation all of that data is very valuable to educators so combining all that data in one place is another mountain decline both local state federal level and with apprenticeship Carolina we're looking at new data systems and luckily we do have a lot of funds to be able to connect different systems but all of the processes for MOUs between different agencies and what data they're allowed to share and want to share and can share is is a huge challenge so I think going back to the audience and who you're trying to sell your youth apprenticeship program to whether it's getting more companies on board that ROI data is is golden or showing those success stories like coming from Charleston or the Midlands in our state or the upstate um those success stories are are valuable to other students to parents um that that's a different type of data that's a type of qualitative data so I think we're talking about qualitative quantitative and then the audiences so there's a lot of things to digest here and it's there's no perfect system state local federal they all have gaps they all have holes um but it's looking at your audience and what data will be compelling to them that's a long answer Kate I but I think that I think that's right and I think you know it's not the state can't solve it all but has to set guidelines and has to set some I mean in order to really access state funding access federal funding but it doesn't it's never going to mean that the local program isn't going to have to be collecting more to tell that story to engage um their employers in their community to provide probably more actionable data for their educators and the supervisors to be able to support so it does take all levels um but I think you know starting with the research question starting with the audiences is a great place to start given it can obviously very easily um spiral out of control so we've got a couple questions from audience audience members that I think want to get into the brass tacks like the detail so one of the first question is what tools you're using to collect data um so I don't want to quickly kind of off the shelf things you've created obviously some statewide systems I'll take a stab at this one first I think it's a great question and we're still kind of working through this so our approach was the first survey and find out what come what is being collected and exported from the data systems of our partners because we didn't want to make assumptions about what we knew their capabilities were so what we actually did was engage in a data a data exchange it was kind of two-fold one was student data and one was programmatic data with our partners and we had them each export a certain number of records to us so that we could just see what was coming out of their systems and then we also asked them where that information was going and we just did that with excel um and it just opened the doors for us you know in terms of insight into systems that we've never seen before or where they perhaps data was duplicative right and processes weren't as clean as they could be and so what we've decided to do most of our partners are using the laces system because apprenticeship data and so we're going to help augment and and enhance that system so it fits our shared purposes but we're also looking at what our end goal is going to be in terms of to amy's point who our audience is what are the messages they need to live or what are the data they need so we're actually not putting a stake in the ground yet in terms of what the best way to do this is until we have a better handle on um what the data actually is the stories we want to tell who it's for what it needs to look like and then do we need to build a system around that so that we have long-term system functionality because the last thing we want to do is create another system that's another barrier that's another burden that doesn't talk to something else right and it causes more trouble for our partners so amy's point really uh or um comments earlier really resonated with me around this understanding of what do your partners need and not coming to an assumption first around how all of this needs to work it's really listening and then letting the state be the catalyst potentially right to removing barriers creating the right conditions and supporting its partners all right amy i'm proud of anything to add any tools you guys are using that could be useful well i know we're using sort of for our basic sort of participant um another sort of stakeholder uh engagement data we're using Salesforce so we sort of have the the portal through which students can apply and and and eventually be placed into internships is is connected to Salesforce so we have that information um there and then that's also a place where we can sort of go in and put in and sort of you know notes on pulse checks and and things of that nature and then and then you know when it comes to some of the other data that we're trying to collect um you know we're issuing a sort of a lot of surveys you know to students uh school staff employers as well again to sort of speak to that sort of uh that sort of systems change piece and then any sort of potential shifts in in mindsets and things like that that we're seeing so um so that's you know we use a we use a platform called type form which is you know you can just use google forms for those surveys kind of whatever whatever one prefers so that's another sort of tool that we're using to to collect uh information uh from from the range of stakeholders um in in this case survey information i don't have any if you have anything or the kind of yeah we have a lot of sources whether we get data from individual spreadsheets that we upload into another system our it office has the magic sauce to combine many different data sources and put them into one report so i have here my weekly apprenticeship report i get to see what's going on across the state for youth and adult apprenticeships um the top six industries the top six occupations and that's coming from a couple different sources that are magically combined behind the scenes well that actually answers my next question um that's someone autumn uh pose which is really around like what are your like key metrics i know we've raised them a little bit but then how often are you analyzing that data and looking at the metrics um and so that's your weekly which i think is is probably i would imagine well i i actually don't want to assume when a pervin john if you're looking at you're getting uh weekly data reports but curious what are some of those other key metrics and what are those processes for actually looking at the data reviewing the data making it actionable which i think this is really all about yeah i mean i i think for us you know there there's either certain i mean it certainly is sort of ongoing in terms of you know how how often uh we sort of access the data i think there are certain sort of critical points at which we that i wish we you know certainly sort of take a deeper sort of dive into the data i think certainly sort of after uh an enrollment cycle uh where we have a new cohort of young people um placed into apprenticeships to start their apprenticeships getting a sense of that data um you know what what does a new cohort look like sort of right off the bat and then you know again another sort of deep dive that occurs sort of at the end of every sort of program year again where some of those survey year-end surveys come out and things like that but then also to sort of see um and again something we're tracking you know sort of ongoing in an ongoing way but looking at sort of retention and things like that but then at the end of the year getting a sense of things like you know program satisfaction and some of the other things that sort of come out of the survey so i think you know at the beginning of sort of the program cycle and at the end of the program uh at the end of the program year i think are sort of sort of the two sort of critical junctures at which we really sort of get you know look at at the data and get a sense of, okay, what went well in terms of recruitment for the U.S. cohort and then what went well for all the cohorts in terms of where it didn't go well for all the cohorts throughout the course of the program year. And obviously, there's a lot of ongoing sort of checks on the data throughout, but those are the two critical points for us. So for us, my answer right now is I don't know yet. And the reason I don't know yet is because we're allowing our research to design or research design to answer those questions, right? So depending on how the research design flushes out, some things make sense to analyze say annually, right? Some of the data might be more worth looking at a little more frequently. Some data might be more appropriate to Amy's point for certain audiences than others. So again, I go back to like for us, we're really trying to upend the way we've approached this compared to in the past. And so we're really relying heavily on researchers that we've pulled into this work to help advise us on what is the right approach to this. Because we really believe in this research first approach to our data system. So I'm going to ask one more question. I'm going to ask my rapid question. But something that we've heard a lot about and hasn't come up explicitly here is one of the major challenges is getting the data and information we need from employers. Obviously, again, the showcase we had beforehand showed kind of one model of that. But particularly as we think about the data around student skill progression, their competency, attainment to be able to mark that progress, not just are they participating, not just are they completing both that experiences. So is this a challenge that you've seen and are there any strategies you could recommend that have mitigated or made it easier to get employers to be able to provide the data without putting that burden on them? I don't know, Amy. I've not been paid and say that it's an impossible task. It's very challenging. With the grants that we have, we have to do a lot of that data collection. We have been for the past five years, not for youth apprenticeship. And there's a lot of holes. And people come and go from companies. So it's hard to really have good follow up. And I'd say just having boots on the ground that can do that follow up is really important. But the companies are really there to train the apprentices. So having to give them homework is very challenging. And I'd say that we're we're looking at just providing more support from our staff to do that follow up and hope that that helps us achieve a better return rate with the data from the companies. But it is it is a burden when the companies are really focused on providing that mentorship and high quality training opportunity. I don't know if you have any strategy recommendations, ways that you can get the information we need, but keep that burden as low as possible. Yeah, I mean, I think, you know, so, you know, career wise and here to here, we use a platform that I'm completely blanking on the name, but that sort of it simplifies it a little bit for the employer to do a sort of an every six months sort of skills, sort of assessment. So I think that, you know, I think there are challenges there, but I think sort of trying to make it as as as little of a lift on the employer's part as possible, I think it's definitely what the sort of the key strategy for employers and you know, we're always sort of talking about sort of trying to maintain as light a touch as possible with the employers, while still of course, you know, maintaining sort of a minimal sort of level that they need to be sort of involved and to engage to do the work well. But in terms of what we're sort of asking of them, try to keep that sort of as light touch as possible while still getting the information, that's a very fine line and a tricky balance to to take on. But but I think keeping that in mind, I think is definitely something I would strongly sort of encourage. And if there are things that, you know, and I think this is where, you know, sort of to the point that John sort of has been making around sort of really allowing your sort of research questions to drive your data collection, you know, especially with employers who might otherwise be might find this to be burdensome, you don't want to collect data just for the sake of collecting data and make asks of folks just to have data that you don't necessarily need. So I think that's where sort of that that that planning on the front end and really sort of being driven by the research questions can really sort of come in handy. So at least then you sort of know we're collecting data that we need to answer these questions that have been determined to be very important. So she engaged in the employers on it. I mean, it is it is challenging. And I guess, yeah, again, sort of keep that as light lift as possible for the employers, I think is is is the way to go. All right, well, we have, thank you so much. We have one minute left. I asked you each, I gave you each 30 seconds in the prep, but I'm going to strike down in 20 seconds, we can get it done in a minute, which is what is one lesson or piece of advice you have around accessing or using data to drive quality equity and decision making. So we'll do a quick rapid one or two sentences when we start with Amy. I think I go back to what I said at the beginning is start with the basics and make sure you have that with even the first program because once you start growing rapidly, it's going to be really hard to backtrack and get that data. Some of these folks don't even live in the state anymore. They may have gone somewhere else. So to try to backtrack and find people and find companies, it's just impossible. So I'd say start from the beginning, get the basics down and then build from there. Okay, a perv. Yeah, also just to sort of reiterate before just really sort of be clear about sort of what your goals are and kind of what your strategies are and allow that to sort of drive your data collection. Imagine, John. One single thing with four components. Go slow to go fast. Make sure you know where the point of inflection between outputs and outcomes are by asking so what questions. Be clear on your shared goals, barriers, partnerships, right, and connections in your data systems. Know in the beginning who will find value in the data. So to Amy's point, your audiences. And then the last thing is keep it as simple as possible because that's what everybody needs in this world is a little bit of simplicity. We can always make it more complicated later. That's great. Well, thank you all so much. I hope everyone will virtually join me in thanking our panelists on behalf of the rest of you. Just as I just one heads up I want to give I should have mentioned at the top which is Advanced CT as I mentioned has been facilitating this work group focus on data. We will be releasing a memo summarizing the findings from that next month. So I know our friends at New America will help make sure that gets in the hands of all of you. And with that, thank you again for your time and I'm going to turn it over to Taylor.