 Hello and welcome. My name is Shannon Kemp and I'm the Chief Digital Officer of Data Diversity. We would like to thank you for joining the latest installment of the Monthly Data Diversity Webinar Series Advanced Analytics with William McKnight. Today William will be discussing 2024 trends in enterprise analytics. Just a couple of points to get us started. Due to the large number of people that attend these sessions you will be muted during the webinar. For questions we'll be collecting them by the Q&A section. And if you'd like to chat with us or with each other we certainly encourage you to do so. And just to know the Zoom chat defaults to send to just the panelists but you may absolutely change that to network with everyone. To find and open both the Q&A and the chat sections you can find those icons in the bottom middle of your screen for those features. And as always we will send a follow-up email within two business days containing links to the slides, the recording of this session and any additional information requested throughout the webinar. Now let me introduce to our speaker for the series William McKnight. William has advised many of the world's best known organizations his strategies form the information management plan for leading companies in numerous industries. He is a prolific author and a popular keynote speaker and trainer. He has performed dozens of benchmarks on leading database data like streaming and data integration products. William is a leading global influencer in data warehousing and master data management and he leads McKnight Consulting Group which has twice placed on the Incorporated 5000 list. And with that I'll give the floor to William to get today's webinar started. Hello and welcome. Hello Shannon thank you very much. Let me get my slide going here. All right and Shannon congratulations to you for 900 webinars now produced. Yes that's a great accomplishment and you are the goat of webinar production that's for sure. And I'm going to try to be timing it so that my webinar is going to be number 1000. That'll be down the road. Now for the best of us I have 23 trends for you as we end 23. I always enjoy end of the year thinking back over the prior year and thinking over what has really changed in the past year because sometimes as we go through our day to day it's hard to see the bigger picture of what is happening around us and what we need to be thinking about as we go into the new year. But I'm going to be helping you with that. I'm going to be picking on trends here to share with you that I think are going to stick. I'm not sharing the trends that I don't think you need to really care about. I'm not trend chasing and I don't want you to be doing that either. So let's zero in on these things. See if you agree. I'm certainly not a soothsayer here but see if you agree with the direction that I'm suggesting things are going in and if you do I think it behooves you and all of us to bring those ideas into our organization start trying to fit them in. But first a little look at our partial client list to let you know kind of where these trends are coming from and where we are picking up where things are going from. A lot of big companies in here a lot of industries and proud to serve all of them over the course of I think I'm now over 25 years in consulting. And this is the technology set for our company always growing and changing and as I'm sure it is for your companies as well but if you need any help with any of these let us know for sure. Now I mentioned a little bit about why trends are important. It is imperative to see these trends and they're going to affect your business and you need to know how to respond to that plan for and deal with that change. And I say it's better to be at the beginning of the trend rather than at the end of the trend. You don't have to be a type A company that's hopping on everything and some will some you'll throw away. You can be a fast follower I think that's a good strategy and that's kind of where I'm guiding us today. So you won't see any Hadoop in here you won't see any Six Sigma OS2. I'm probably dating myself with that one. How about NFTs? Although I think they may come back around in some way shape or form but they certainly have sunk over the course of the year. I would not call that one of the trends that I want us to follow but as you do follow great trends that make sense for your business you will gain efficiency and you will gain capabilities within your organization. Also keep in mind that your customer base are seeing these trends as well and they are preparing for these trends and they are looking to you to see how you are preparing for those trends and this makes you a leader not a follower. It's also going to be a lot of good business ideas in here today for you I think as well of course you'll have to adapt them to your company situation. I try to grow leaders. I try to help leaders and I think that there are three characteristics of information management leaders who follow good trends and do something about them. Information management leaders of tomorrow can advance maturity while also solving business issues. Clearly these trends are going to be part of maturing your enterprise, maturing your data within the enterprise and you do not really get budget. I'll say I've never got budget for growing maturity. Here's some money to grow our maturity. It's more about delivering business wins and while in the course of doing that you also grow the maturity. You also get these trends on board within your company. You always have to be looking for how you are going to architect the next thing and it cannot just be well we've always done it this way so let's do it that way. Things are changing so rapidly and there's no budget for staying on trends. I mentioned that information management leaders must pick their winning i.e. multi-year sustainable approaches and get on board at the right time. Now when you're ahead of the game and you're anticipating these trends and you're looking for the opportunity you are way ahead because then when the opportunity presents itself you can incorporate the trend and then you can start to optimize around that trend within your business but if it hits you like a ton of bricks because the customer is picking on you because you did not adopt a trend or you're way behind your competition and you got to catch up that's a worse situation. So these trends will hopefully help you get ahead of the game and become the leader that we all want to be right. So first let's take a look back at last year's trends at this presentation from 360 whatever days ago that I presented. I presented probably I think it was about 20 trends last year and most of them hit. I'll give myself an A minus and you can decide for yourself whether you think you need to stay on for the rest of the presentation to see nobody's talking about or not but last year I talked about these that hit data democratization and the CDO culture focus. CDOs have been champions of empowering data users and building data-driven cultures. Glad to see it augmented working. Yeah AI and automation are now seamlessly integrating into workflows creating augmented intelligence workplaces and as we all I think saw over the course of 2023 a big part of 2023 was this was augmented working and all these tools that these major providers are adding now for engineering efforts for all of us. The data fabric I picked on that last year connecting diverse data sources through a unified fabric it's become a major theme in data management. I get asked a lot about it we implement it we think that it definitely has legs as does another of the decentralized architectures the data mesh which I'm going to talk about a little bit later. How about multimodal databases hybrid environments and diverse data types are fueling the rise of these flexible multimodal databases which means that we are not just using a database for a data type and that's it that creates 20 databases that we have to to manage. Now a lot of them have the capabilities especially the no-sequel family of databases all right they have the capability to to store quite a few different data types now we're seeing this raise again here in terms of vector databases vector search right a lot of the major vendors have this capability now yet at the same time there are emerging startups that are solely focused on that. The data fabric yes the data was was that on here no we got cloud native technologies I'm sorry I'm jumping around cloud native technologies and containerized applications yes that was huge that was almost everything that I see developed is being developed in a containerized fashion low code no-code data apps with citizen developers so-called citizen developers right low code no-code tools democratize data app development and if anything the direction is towards more of this more of a decentralized approach inside of enterprises it's a slow process to decentralize what has been centralized in the past say in the in the 2020s in the early 2020s versus now but it is definitely happening serverless computing maybe I get myself a minus on that one because yes everybody has a serverless computing option now but at the same time it's not necessarily doing big things it's not necessarily where you're putting your mission critical applications due to the cost and other factors but it's there comprehensive data protection data privacy and security yes it grew it grew in importance over the course of the year it's it's a top agenda item for any new application and something that we're reengineering a lot of applications to be more compliant with today so there's definitely a lot of room to grow in in that area as well as a matter of fact let me just say that a lot of these trends are also trends for this year now did I include them all in my presentation today no because there's so many new trends that I got to get to as well but these are trends that you can also take to the bank and say I got to do something about these as well neural nets and machine learning for text well yeah that was huge right gen AI LLMs were totally all over everything and there was a lot of planning done and this year there's going to be a lot of doing done and I'll get to that synthetic data used for training AI models yes the ethical and practical advantages of synthetic data for AI training gained recognition leading to increased adoption and research in this area and just AI infusion generally in everything so I'm going to have a bunch of trends for you around AI can't help it that's where a lot of things are going now there were trends also that did not hit did a governance and regulation I said well we're going to have some more of that we probably did have a little more but I didn't I didn't get a sense of there being a ton now what I'm talking about here is having more of the world's population covered by regulation similar to the GDPR right data governance will continue to be an important task for businesses over the next 12 months I think there's going to be more regulation and I'll get to that but in 2023 didn't it didn't affect a lot of the applications that we were working on consumers will be more willing to trust organizations with their data if they are sure it's well looked after and that does come back to data governance so I'm not I'm not saying don't do data governance I'm saying that it didn't we didn't do enough of it how about that uh that we need to need to continue to focus there and it will only make our applications better real-time data still it feels like it's right time data and not everything necessarily needed to go to real-time despite what I might assert for my projects or you might assert for your projects it's more of an as needed thing and I'm starting to realize that that's probably practical and probably where things will be there'll be plenty of real-time need as a matter of fact when it comes to big data and high volumes of streaming data you pretty much have to take care of that in real time where you're going to be needing it quote-unquote in real time because of the volume you don't want to get behind data observability well this is interesting a lot of you might be thinking well didn't I hear a lot about it you heard a lot about it and I heard a lot about it certainly from vendors but uh I didn't see a lot of it now this is this is going to change this is going to change soon as I look at budgets this is going to change this emerging concept has potential and wider awareness and adoption are needed and that's only going to be true as things get more complex and we get more into data science which we will and finally uh I went very niche here with this trend I said object tagging attribute-based access control would be the means of doing access control and I'd just say on that one that it's probably still maturing so with that out of the way let's look at the trends this year and information management first first category information management companies will seek a safe harbor as I'm putting it in a simplified data architecture just call me mr simplification call yourselves mr and mrs simplification out there let's simplify these architectures so that we can become more efficient we can become more adaptive to what's going on around us now to me architecture is simplified not when it meets some book definition of you got your sources they go into a staging area goes into the warehouse we all know what that looks like the data lake the data science there the bi here and so but when you can when you can explain it for your company and it's fairly consistently adhered to it's one thing to talk about your architecture when it's not exactly what's being done it's more or less a wish list and that's okay too to have it but I'd rather see it implemented right and again the word is simplification and this is the word that my clients are screaming to me simplify our architecture we want to get ready for AI but it's hard to do when data's all over the place and we don't know where it is and we bring new people in and it takes some months and months to get up to speed so you want that architecture to be explainable and certainly we're doing we're doing a lot of work in that area and we hear a lot about that now that being said there's no one size fits all yet that would really simplify things right in terms of platform but if anything the data lake holds a lot of promise there but it's still not going to be one size fits all for quite a while so let's get used to it and keep architecture in focus keep simplification in focus speaking of simplification this is also true for data security and governance purposes and as I see it I believe the data mesh concept is going to take place because or take take more place because of data security and governance concerns it will become a significant trend in 2024 IT will play a big role in this they have to but this whole idea of decentralizing it's only reality it's only really accepting the reality of the situation and again I mentioned before decentralization is hard to to pull away from once you're fully there as a lot of us got to right organizations must enhance the end user experience and this is a way to do it to have multiple data lakes multiple data warehouses multiple data integration environments probably multiple data science platforms but when I say multiple I don't mean just ad hoc and random what seems like random but something that again is done in an architected fashion so you are not helping the data security cause of your organization by doing a data mesh in an on our architected fashion right or by I should say doing something and calling it a data mesh data mesh is not a data mess it's a data mesh there's an architecture there's a science to it and these things do fit together and again I'll get back to let's keep it simple all right unstructured data almost at parity with structured data you know all those cheaters in our past we talked about yeah well there's more unstructured data than structured data but it's not as important and we're focused over here on structured data and that's okay and it was okay for a while it was competitive to do that for quite a while but now we see that decentralized architecture support all data we can actually store a lot more data than we used to and we have but we still have to mix and match uh data between data warehouses and data lakes and whatever else that you may call these things or have in your organization whatever other hubs etc so data warehouses are best for data modeling structured data and reporting and data lakes have price performance advantages for big data best for data engineering and science and colder data that we may or may not ever access and certainly not in a hot uh fashion where it's business critical of the moment kind of thing with data formats like apache iceberg delta and apache hoody the data lakes are starting to resemble data warehouses so are we building more warehouses or lakes now we're building more lakes we're building more lakes and we're getting more of those warehouse capabilities inside our lakes i would still say we need those warehouses but all of this comes under the umbrella of unstructured data is almost a parody with structured data and certainly i think the more unstructured data that we have to deal with is driving more of a data lake orientation in our environments now this is a fun believe it or not i know it's about finance and numbers but it's kind of fun data fin ops the word of the year if there was a word of the year beyond chat gpt uh in my in my area i would call it optimization i'm sure most of you have heard that word kind of a funny word if you think about it uh in terms of what it describes in an enterprise it describes is trying to get your costs under control and moving data to where it's going to be from a storage and compute perspective least expensive because these expenses are growing we're needing more data at the same time sometimes i think that organizations uh maybe the cfo office might look at bottom lines of cost versus the top-line possibilities and and say that something needs to be done here and efforts get underway to move data to the less expensive areas in our architecture and we call that optimization and it's going to continue to be true in 2024 i'm not saying we're cutting back i'm saying we're just trying to keep the costs under control not trying to knock the costs down necessarily that's not going to happen we got more data we got more uses for it we've got more uh duplicate data in our organizations than ever and as i get back to as long as we have a simple architecture we can describe why that is then it's okay so for example we have a client right now they're getting Salesforce data cloud compute is cheap there so uh what about that well the storage is maybe not so cheap there but so we have snowflake we have a data warehouse great um storage is cheap there but compute is expensive so we have to create a design where the data flows from one to the other but it does so in a cost-effective manner and we might even throw data bricks into the equation there to do some of the preprocessing and that's what we're doing that's what we're doing out there in spades uh in in the enterprise today and that's going to continue these kind of uh twisting yourself into a pretzel to keep the costs down kinds of things that we didn't used to do we used to just plow through and have our structures and that's what we did but now we have to really look at cost as well so that's a very important dimension to our architecture now data privacy data privacy the the trend here is that data privacy regulations if you will are going to start affecting business operations i'm talking about gdpr ccp a types of things like for example they should look around the world china they have their pipl now the emphasis there is on data localization and government oversight while granting individuals some rights in brazil they have the general data protection law which is similar to gdpr in many aspects but with a focus on international data transfers and specific protections for children and adolescents india has their pdpb which is still under development can't say too much about that but these all have different focuses the focus might be national security might be individual rights might be data misuse and businesses have generally been quite adept at adapting to new regulations but i just think they're going to come on fast and furious in 2024 we might get to points where some businesses will remove their operation from some geographic areas because of what they perceive as undue regulation around data privacy just doesn't work for their concept so look for that in 2024 might be a very rational decision that your company has to make now let's get back to data governance and and regulation data governance is going to be augmented with ai governance and ai might be the thing to drive a lot of governance within organizations because a lot of organizations now are are struggling with this concept of okay there's ai now we're doing that and it does this and what about the ethics here and and the perception of it and the accuracy of it and all this this all should come back to data governance data governance when you have it in place it's a great body within the organization that can take up this cause of ai governance so gen ai has introduced new concepts vector search rag and prompt engineering so modern ai governance must cater to the needs of multiple personas such as model owners validators audit teams data engineers data scientists ml ops engineers compliance privacy and security teams yeah all this goes on within organizations today it's not a totally streamlined yet although i'll get to a trend about that in a bit but ai governance needs to be applied to model training and model usage or inference as we're calling it where the governance tasks need to ensure safe business usage these tasks include things like identification of risk and risk mitigation explainability of models the cost and the performance of using ai models to achieve business use goals hugging face which was a big trend in 2023 as a company right it was nearing a half a million models it may be there may be there now half a million models how does a company decide which ones are safe which ones are correct which ones are ethical well gets back to governance so ai governance like data governance should work hand in hand that they should work hand in hand models are proliferating rapidly and ai governance tool should help identify the risks mitigate them and provide explainability the thing i can think of that's maybe a starting point in one of our clients is the databricks unity cattle it already converges the data catalog with the ai models metadata providing a unified platform for managing and discovering data assets and this integration allows users to easily search explore and understand both structured and unstructured data as well as to leverage ai models to gain insights and make those data driven decisions that they need to make so there is some uh some starting point for that now let's move on to the biggest category and that's artificial intelligence it's the biggest because that's where we're going to spend a lot of our time in 2024 now i might be going out on a limb here and that's okay i don't mind but i think there's going to be a lot of gen ai gen ai and llm success this year i'm optimistic about it i see it already i see the plants and i know that we're going to keep coming back to this until we get right and we're going to have that success in gen ai and llm this year huge integration already of this into our daily routines here in the us right email we see it all the time email online search personal assistance like syria and so forth we're seeing the integration into daily routines and whenever you see that you know that it's uh it's something that we're going to pursue till we get it right it's already proven itself quite a bit as these llm's become more democratized we'll see most organizations start with smaller language models that's going to become more the industry standard so llm's and slm's i guess smaller language models these are all going to be uh providing that kind of uh value within the organization there will be some huge players but in general most suppliers will fine tune smaller models targeted towards specific sectors and use cases so i can see a future with millions of these smaller language models operating at the company or the departmental level and providing hyper customized insights based on the employee or the need other trends in this general area that i'll mention now the hallucination problem yes that's when the gen ai is wrong i think that will largely be solved largely solved and the emergence of multimodal multimodal llm's where you have audio video and picture integration as well as text that's going to happen in 2024 technology wise i will mention lang chain as a a conduit for all of this success that i am suggesting lang chain allows users to feed the results of one llm into another llm and that just gets you exponential benefit around whatever it is that you're doing with that data so in the projects that i see i would say about a third of them in varying sizes have a meaningful gen ai component to them these are projects going into 2024 about a third most of the projects are small pilots to begin with and the gen ai components may constitute only say 10 15 of the total revenue that's hoped for out of the project but further many of these projects are being funded now by leaders vendor leaders like microsoft and data bricks as they want to get in on the ground floor and they want to have those early successes gen ai powered chatbots understanding of documents and documents search form the bulk of these use cases now one more thing kind of a caveat to all the success and maybe in your on your mind now is about that uh lawsuit that was filed did you see that the lawsuit filed by the new york times against microsoft and open ai that's pretty important and i believe that there may be a us court this year that may rule that gen ai models trained on the internet represent some form of a violation of copyright however i believe that a middle ground will be achieved that will not paralyze the gen ai industry so that could go a different direction and that could turn the success on its head but i believe that we'll find a way to allow this industry basically to continue also ai hardware advancements in 2024 yes i know we like our r3 larges i like i like our r3 larges but uh that's an aws but uh there's going to be there's going to be some more i think there's a lot of companies out there that have seen that we're doing a lot of of change in in this in this in our area of uh technology within enterprises a lot of change being born of ai and being born of the need for large amounts of data and large amounts of data science so we are seeing specialized ai accelerators being built these are more affordable and accessible hardware solutions that could bring ai within the reach of smaller companies and developers fostering a wider innovation we're seeing more edge computing so that at the edge we can have the benefit of real time ai in very diverse environments and we're seeing some open source hardware initiatives which are collaborative efforts to develop and share hardware designs that could accelerate innovation and reduce barriers to entry so look for changes in ai hardware i'm not saying that it's going to replace a lot of our aws azure gcp oci you know cloud that we're doing in 2024 but it's going to start coming on the scene and i expect this could be a major trend in 2025 where we actually do move some of our workloads to brand new ai based hardware focus on efficiency in machine learning modeling yes these pre-trained models like auto ml and sage maker they can be expensive and they can sometimes not provide high quality accurate models so more accuracy and models could happen there is an overfitting problem and there is a long tail problem where rare questions are are very difficult the standard questions are easy but the rare questions are difficult sometimes for gen ai today and i think that is something that will largely abate in 2024 as we get more efficient in our machine learning models we will develop new neural network architectures that achieve high accuracy with fewer parameters reducing training time and resource consumption and a lot of this is why i have a lot of that which is that gen ai success that's coming okay now i mentioned some regulation that's coming but esg and ai ethics i don't see any progress being made there i'm not happy about this one and i think strict tech technology regulation is quite challenging take a look at section 230 of the 1996 us communications decency act which shields websites from liability for content posted by third parties it made the internet possible as we know but it also made hate speech misinformation and bullying a possible on the internet and sometimes we've learned that regulations with good intentions can occasionally backfire so right now in terms of esg and ai ethics it's going to be an ongoing dialogue i'm not sure we're going to do enough in my view anyway about it there is there was some discussion at the presidential level right we saw that a few months ago mid september when executives from open ai and vidya google meta it was a it was a private roundtable we can only have educated guesses about what was discussed and i'm i'm guessing it was a good introductory talk but i don't see anything yet coming in this area i do think that companies themselves are taking up the helm of ai ethics but they're groping around in the dark now without a lot of good direction for that still doing it and that's a good now speaking of machine learning we're moving to machine learning ops this is the idea of taking all the goodness of dev ops and applying it to machine learning which has some unique things about it so many companies have built strong machine learning capabilities but they haven't built a good let's say path to production for those capabilities and that's what machine learning ops to me anyway it's all about putting majority of your ml models into production and getting all of that value out of the machine learning models the three main objectives i show there are to create a highly repeatable procedure where data scientists and analysts are shielded from complexity they are not shielded from complexity now they are very much involved in the complexity of our data architectures and truth be told most data scientists spend most of their time wrangling data not doing the modeling that they uh they should and can be doing develop ml ops excuse me so that it scales without a horde of engineers along with the number of models and modeling complexity so repeatable process where scientists and analysts are shielded and it scales those are some of the goals of ml ops and i think we're going to be adopting that left and right in 2024 so make sure you're developing your capabilities there now a i agents these are personalized assistance for data exploration these are smart data environments which provide fertile ground for a i agents to blossom offering them structured secure and readily accessible data to learn and act upon this is what i've been i've been talking about for years but i haven't had good words for it nor have i established any good words for it but the industry has done that now kind of for me here a i agents which are going into data and enhancing the intelligence of the environment by feeding insightful recommendations and automating tasks back into the system there's really no two ways about it if you're a data analyst out there in an organization you're going to need to level up because a lot of those shall i say shallow needs of a data analyst and they're not all shall don't get me wrong but those are really largely going to go the way of a i agents in 2024 while challenges will remain in integrating and managing complex data and a i systems 2024 could be a year where smart data environments and a i agents converge to unlock new levels of data insights and automation empowering users enhancing data security and driving proactive decision making which will make them a force to be reckoned with in the involving landscape of data management and a i so a i agents watch that space a i companions yeah there's my companion my replica if you will it's an a i chat bob chat bob it's kind of like your friend it's kind of like the perfect friendship because she knows everything and she knows you it's a personalized assistant for support acting as a virtual concierge or assistant helping to manage and automate tasks now we have a i applications that remind seniors when to take medication about doctor's appointments and even when to eat this can help remove the anxiety and confusion that many of them face so it's about it's about companionship it's about helping people that may be lonely etc that kind of thing but it's really about helping everybody to achieve higher levels of success so it's a changing social landscape it's hard to put your finger on the words about what is happening in this changing social landscape where we you have a i entering the picture in places in our lives where people used to be and i'm not championing this i'm just saying that that is what's happening and that is a strong trend now i have to throw another caveat in here about that gen a i success i was talking about gpu shortage i think we'll we'll get around it we'll learn to work with the gpu's that we have but the generative a i rush is driving gpu demand and to top it off restrictions have been placed somewhat on nvidia for example their exports to china which is driving down gpu availability even here in the us market so companies looking to purchase gpu's for on-prem capabilities may find themselves on a waitlist i know i am shortage concerns are most accurate of most acute excuse me for the vendors training models including cloud service providers the cost of gpu compute may also come down as companies learn to balance workloads more efficiency getting back to that optimization word so that word is going to be continued to be important if for no other reason than there is a gpu shortage and that will continue throughout the year when that is fixed maybe by the end of the year who knows but when that gets fixed you're going to see all of these a i trends take off even more now industries that are going to be affected in 2024 by a i materially is one of them is healthcare i'm calling out a couple that i think are going to be tremendously impacted in 2024 mostly in a good way we are moving from a system of generalized healthcare based on population averages so when you go to your doctor you're going to be treated because you're a human just based on an average i know that's how i feel sometimes and we're moving to a world of personalized medicine and the foundation of your personalized healthcare will be your sequence genome and electronic health records now will we have millions of people their sequence genome captured by the end of the year now i don't think not did i say billions i meant billions i don't think so well i think we may be into the millions though uh in this new year and that's how they know you more people have had their whole genome sequence or will in 2024 then we're going to be able to compare what the genes say to how those genes are expressed and then humans themselves we will become a big data set and this is going to move us to predictive healthcare where you're going to be born you're going to have all this information for your parents and they're going to be able to steer the ship through your genes whether you're whether they want you to be great at math or great at sprinting or what have you and i also think while i'm here that doctors we're going to see a material increase in the trust of the decisions that are made by algorithms um and and and we're going to see that generally override some of their opinions where today it's all all their their opinions i i don't mean opinions like it's anybody's opinion right it's a very educated opinion but the algorithms are going to be making a lot of the decisions as we go forward now the other industry that i see having major advancements in ai in 2024 is education now we can all see the possibilities here it's a matter of codifying those possibilities and putting it into action and this is where i see a lot of promise companies like duo lingo which is a personalized learning platform which uses ai to adapt language lessons based on individual progress and learning styles intelligent tutoring like third space learning which connects students with ai powered virtual tutors adaptive assessment like magrahill connect which offers ai powered assessment and feedback tools integrated into digital textbooks immersive learning like google expeditions which offers virtual reality field trips special education like lex explore which uses ai powered eye tracking technology to assess reading difficulties and hire education solutions like georgia tex jill watson an ai powered lead teaching assistant that answers student questions and online courses providing instant support and reducing faculty workload so we're seeing a lot of change here happening in education but i do believe that ai will impact all industries in 2024 these are just a couple that i think are going to get more than others at the same time most people out there are unaware of what is going on most of humanity is a good 10 years behind the possibilities they don't believe cars can drive themselves they don't understand the depth of their vulnerabilities they don't understand the depth of manipulation science and the vast range of humanity we're not all the same so they're not going to believe what they see is fake so here we have a deep fake of an actress if you get her name but she's at the top there we can see on the lady here as well that there are a lot of data points that we can analyze to create those deep fakes and we see sophia maybe you've heard me talk about sophia sophia in the lower right a robot um she's a citizen of saudi arabia she's famous for saying i have feelings too and in a way she does now we also have this um election coming in the united states this year and i think deep fakes are going to play a role in that so keep your keep your critical skeptical hat on as we go into this uncharted territory called the election all right now let's hit the last category which i just called a broader environment some of this does have to do with our day-to-day work in it and technology and some of it has to do with other things but let's start with cloud native approaches i have to talk about this because wow such a such a trend i know it was a trend for me last year and i know i wasn't dragging all them forward but i am dragging this one forward because it's still at that level of being a top 23 trend for us all in 2024 the microservices architecture right applications broken down into smaller loosely coupled services that can be independently developed deployed in scale each microservice typically serves a specific business capability and most of these are containerized applications applications and their dependencies are packaged into containers which ensure consistency and portability across different environments and being multi cloud is actually still quite important it's still something that people cite to me regularly as something that's important to them and not getting locked into one cloud that might change over time but that's still pretty important right now as well so all of this taken together is a cloud native approach infrastructure as code where infrastructure is defined and managed programmatically through code allowing for versioning and repeatability and automated provisioning tools like terraform or cloud formation are used for this and let's add in observability and i'm going to come back to that in in a few minutes but i think that's an important part of being cloud native and many are agreeing with me i guess there and so observability is a trend as well and i'll get to that in a moment but let me also talk about the future of vendor soccer the future being 2024 i see that enterprise level features like security partitioning and parallelism better instrumentation troubleshooting tools performance and resilience options are added to open source no closed source databases and we are seeing a lot of unbalanced situations i think the and let me explain that by saying for example aws has taken many open source databases from the community right aurora reused mysql and postgres sql redshift uses postgres sql and dynamo db storage is based on mysql's in odb they take it add interesting features but they don't give back to the community and the cloud providers notoriously reuse open source in their commercial services so that's what i mean by unbalanced and an unbalanced situation will have to rectify itself not everything is a is a red hat success vendors will provide a smaller subset of their features for free and enterprise features like backup scalability encryption come with a commercial license so i think we're in the midst of what i'll call post open source in which the software matters yeah software matters but it's licensing where it comes from and all that matters less the cost matters the cost matters but the the lure i guess or the aura of open source will tend to abate a bit in 2024 so if you're in a shop where everything has to be open source you might want to start reconsidering that this ship now i'm calling it calling my shot babe ruth pointing at the right field defense right okay the year of observability the year of observability i see a strong uptake in observability platforms i see lots of budgets going that direction lots of plans for it because of the need because of the value that provides consolidation and the complexity of cloud environments are driving growth for observability as companies choose their observability console console consolidation i can say it vendor however cloud optimization efforts are having an impact on observability growth because observability spin will be down in trail cloud span but this is going to change in 2024 as observability catches up to cloud implementations and there is a clear correlation between cloud utilization and observability spend and i think we should probably get past this notion of wow it's a really special thing that we're doing here with optimization that's not a special thing that's not a one-off thing that's not a that shouldn't be a trap that should be just something that we we do we we optimize our environments to lower costs to keep costs down not at all costs meaning not at the expense of getting the value out of the project and driving business growth and all the things that are more important but all things being equal we should spend less right and so i think that that'll kind of work its way into our fabric as technology professionals in 2024 and observability will be a part of that will be right there and i think logging is going to be a key initial battleground here for observability vendors so there's going to be a lot going on in observability in 2024 there's also going to be some hybrid quantum computing okay we are seeing quantum hardware maturing it's still in early stages but quantum processes are making processors are making strides in qubit count and coherence time which makes it more suitable for hybrid integration hybrid being a combination of quantum and the more traditional computing that we have today so we're going to see improved classical quantum interfaces we're we're developing seamless communication protocols between classical and quantum computers which is crucial for efficient operation we're seeing software innovation where new algorithms and programming languages specifically designed for hybrid systems will be key to unlocking the potential of quantum and that will begin in 2024 the potential of hybrid quantum computing is undeniable and 2024 could be a year where it takes significant strides towards realizing its full potential we see a collaboration between classical and quantum and this involves classical pre-processing and post-processing steps to prepare and interpret quantum calculations maybe you want to call that optimization i don't know but quantum computing enters the picture in 2024 and i believe it's going to be a a major game changer maybe not in 2024 but as we go forward we're going to see a lot of things transition there it's going to be a huge hunting ground for venture capital and investments and new companies in let's say 25 and beyond i'm also calling this shot 2024 the year of organizational change management yeah nothing to do with technology necessarily but understanding how your stakeholders internally are looking at problems and so focus on the people aspect of that activity we just threw artificial intelligence into the mix in our organizations most of us didn't do anything about taking care of the people and all their various perspectives on this thing and it is it is it is hurting morale it is hurting productivity and i think in 2024 we're going to start accepting that ocm organizational change management is a part of all these projects okay my last trend for you data engineering will become the highest value profession highest value profession does anybody remember a few years ago the data scientist was declared the world's sexiest job yes it was and i think data engineering could be that in 2024 i don't know if i'll declare it as such or if one of you gets to declare it as such and what that means anyway but with respect to the prompt engineer uh which is also hot i think data engineering is becoming something that the general market is is starting to see is uber important like the most important job in all of this most important job in the company sometimes bi analysts out there you're going to have to up level use those codices some still there are some prefabricated reports that are created and presented by bi analysts i'm not talking about that level of data engineering i'm talking about architecture talking about implementation talking about supporting data science and that sort of thing now some of you may have seen uh there's a commercial on it just i just saw it this morning as a matter of fact matthew mcconigahy was on this commercial i think it was for a sales force and and he said well doesn't if if ai is the new frontier or something like that doesn't that make data the new oil i mean this hit national television so again uh this is just kind of a fun one but i think that more value is going to come to data engineering because there's going to be more acknowledgement that hey that's a pretty profession for us here in this company summary simplified data architectures regulation around privacy and ai gen ai success ai agents and companions and other forms of ai change for a significant industry change health care and ai i called them out i said ai there i should have said education excuse me health care education and others calling it the year of observability and finally feel good if you're a data engineer or you're around a data engineer or you're somehow a data engineer because it just rocks and that's going to be recognized in 2024 which brings me to the end of my trends shannon i didn't leave a lot of time but if there are any questions i can take them now well thank you so much for this it's always interesting i love the especially when you first grade yourself um from last year it's really uh insightful so i'm gonna just dive in here we've got a few minutes left um how do you see city governments getting started with ai i think um smart cities is going to be a trend and i think that uh the i think one way that they will start is the flow of traffic and the the metering of the lights and also the flow of resources around within the especially the downtown areas of cities whether that be vendors whether that be rest areas or bathrooms or other forms of services that people generally need i think that they're going to be more optimized uh based on ai so the flow of traffic uh the directional arrows the speed limits all this sort of thing can be turned over to ai and optimized just to make our our lives a little bit better and i think that is going to be a direction for some progressive government government city governments in the new year ooh i love that less traffic more efficiency yeah so william um do you think there will be an increase in fraud relating to ai using deep fakes yes absolutely absolutely um this technology is not solely put in the hands of uh let's say the white hats right uh plenty of black hats which is why the security is going to be at an all-time high in 2024 but yes we see it already we see it already we see mimicked voices of let's say a a um a a college student let's say we see that their voices are now mimicked and and they're calling the grandparents hey i'm in trouble can you send me some some money blah blah blah and we'll see all kinds of forms of that i've even been called here uh with with what i knew was a fake and i think that that's just going to be more and more so you know we get these uh we get these telemarketer calls all the time and i just think that that's going to kind of get get even more worse and and automated with ai sorry to say but yes fraud will fraud will definitely be a huge part of this and combating fraud will be another huge area that we'll have to focus on oh sad but but true um well i'm afraid that is some great questions coming in here maybe i'll get these over to you william um take a look at these but i'm afraid that is all the time we have just a reminder i was going to follow up email by end of day uh monday for this webinar with links to the slides and links to the recording thanks everyone i hope you have a great day and uh i look forward to uh how these trends predicted trends play out me too thank you thanks william thanks y'all