 drive you want me to all right so now we've come to the meat for those people like to see things that are real right or something like that it's it's very easy to talk about AI and it's very fun as well but this stuff becomes more meaningful when we take a look at some actual examples understand how they're built the types of AI systems that are being leveraged and then actually show them in action and this will illustrate you know the general process of adapting AI to business use cases this will showcase a couple of actual use cases in the open group with AI applied to them and then we'll have a bit of a discussion around you know where the system is today maybe what we could do if we wanted to move it to production right so our first example is an assistant or the colloquial term is chatbot really hate that because it makes it sound easy and churlish but in fact it's not what a chatbot or an assistant is essentially a conversation between you and an AI and like the conversation between two humans when I asked Michael you know something about cybersecurity he says no and that means I need to find somebody else but when I do find somebody who knows something about cybersecurity we're likely to have a conversation where I asked them a series of questions and you know they use their knowledge base around cybersecurity to provide me with answers at the same time a more motivated employee might say I don't know anything about cybersecurity but I can go find out for you because I know someone who does right so if if you have your cell phone and I don't and I ask you some question you could actually type it in and go find it and then still give me the answer which is a relevant analogy to way to the way that an assistant or a chatbot works so in that context what does that mean that means that I have to be talking to somebody who has a knowledge base who has a corpus of information that has been trained around the subject matter that I'm interested in and that means that you have to teach your assistant or chatbot the subject matter in which you want to use it and the use case that's most likely one that's familiar is a support use case so there's an awful lot of situations where companies or organization staff will constantly field the exact same set of questions over and over and over again and they really don't enjoy that very much so what you want to do is be able to provide a an experience where you feel like you're having a conversation you want to create modality that kind of syncopates with the human themselves so it could be voice recognition that is being used or it could just be natural language processing using text which will be the example that we use today but in reality Michael has this set up so that you could actually with TJ bot ask TJ bot a question just you know via the voice and the microphone and TJ bot will pick that up and then we'll pass it on to the chatbot that he has integrated into it so in the case of the example we're going to use today we're going to use the open group professions standard and the support for becoming certified as the case study and what we want to do is create a chatbot or assistant that answers questions about becoming certified or moving from one certification level to the next or resetting your password or something of this nature and so what does that mean for our the process that we're going to go through to build that well first off we have to know something about what are the common questions that are asked right and so we went to Debra and on the staff and she came up with a document Michael and I fortunately had already been building a chatbot for something very similar called Kerry what does Kerry stand for career architect certification journey it's not the best acronym but I like it we didn't come up with I don't think I came up with a very good name of this particular chat you didn't I try but you know like I think it's the hardest part of these systems really I think I came up with something like Toby or something like that anyway so so first off you have to have some knowledge about the subject you want to know what are the questions that are going to be asked you certainly want to know what is the correct answer you want to know the entities that are involved right so you want to know this you know the the data entities that are part of the subject matter in Corpus and you want to know the synonyms there might be you so that is the alternative ways for asking the same question and getting the same expected answer so I might say I'd like to know something about certification or architect certification but somebody else might ask can you tell me about work I would like to know about architect certification or I have a question about architect certification or even better something as simple as certify me yeah certify not even a please on that one right so let's get into this a little bit you've got the word document if you want to show some of the yeah I'm doing exactly that excellent I don't know if I can yeah all right so here are some of the top questions that Debra came up with I'm an interested in applying for open CA certification and would like further information and she even has a reference answer for us and a set of links so one of the things that I can do is that I can actually integrate this within an existing system and I can pass in from the environment information about who is using the system now I didn't do this in this minimal viable product but it can start off with saying hi andrash welcome to Bob or whatever are the name of our our certification or chatbot might be I can I help you with information about the open profession and I can also if you know anything about Bank of America and you you might be from the United States they launched a an assistant called care Eureka and Eureka is an IBM solution and Eureka actually you know you just press the button you say can you show me in my balance Eureka and Eureka will come back with the information about your balance and you say I'd like to know the last three transactions about my master card you'll come back with that and you can ask it all sorts of other reference questions and it'll come back with guidance on how to find that information so here it is yeah you know first question was I'm interested in open CA the other one is I'm an open CA certified but I can't find my name in the directory how do I set a personal code I'm certified but I haven't received my badge that's a nice one I'm certified through my employer but they left the company how do I keep my certification I want to recertify so on so forth so actually because of the work that we did with Kerry I had a little head start from the work that she provided to me and there was actually much more that you really needed to know if you were going to make this a real conversational chat bot so I've I've augmented what she gave me with the information that we already knew so this is the Watson Watson assistant canvas I'm gonna go ahead and open my profession assistant space workspace and immediately you have you recognize that we have a few elements here one of them is intense the other is entities then you have dialogue and content dialogue we won't be going into content dialogue but let's say that you know again you want to get a jumpstart and you're in a particular space what we come up with is you know taxonomy for you to use kind of out of the jar so let's look at our intense we're in the intense space so what are some of the intense that somebody might have when interacting with an assistant around certification let me and let me let me paint it a different way for those who are tuned to some of the other machine learning AI language think of these think of this as a supervised learning system and think of these as your sentence level classifications how many of these do we want well the question as Andross has posed really comes down to what do we want to recognize when the users interacting with us talking about the the open professions well one question might be is who is the heck is the open group itself and so I've created an intent called the open group I can add a description here to which I didn't do it at that time like you know I would like I can see I'm typing on Michael's machine which is driving so anyway what are some of the intense here well let's look at this one I need to know more about open group certification how do I get to certification I want to get open group certified you know all of these things are good examples I don't think I have any synonyms here but one thing I'll call out for this particular intent is you'll notice the questions are actually rather broad all of them are asking questions about the open group but if you have a savvy eye you could jump back to that intent just for a second I'm gonna go to that one yes okay so this one is a lot tighter around more simple what is questions the other open group intent you notice to sort of a general catch-all this one is much more nuanced around the other one was actually more about certification so when you ask somebody about the open group it was intended to trigger a conversation around you know just what is the open group itself the other one was more about well tell me about open group profession certification and how are all the different ways that I can ask that question right so I could use a synonym for a particular word like tell me about or I'd like to ask you those are all essentially equivalent phrases so what you try to do is build a corpus that leads you to the same response or leads you to a particular response so let's see here let's do another one certification pay like do I get a raise if I get certified well certification get me paid more well I get paid more well I get paid more by getting certified all of those questions are questions that you might have people ask and we have had them ask them because we're tracking this information this may or may not be the number one intent it may or may not be the no no it might not be here are some of the entities that you have for example architect discipline in the profession certification there are three different disciplines right now one of them is business you get certified as a business architect the other one is a enterprise architect and the last one is a solution architect or an IT architect so IT architect has the three different synonyms IT architect well actually for IT a architect and solution architect and basically all of those boiled down to IT architect now you're following along from the general AI machine learning handbook this under entities you could think of as phrase level classification so you'll have one classification for the entire sentence to say here's what I think they're asking about and then you'll have phrase level classifications that say in in the context of that for that whole sentence in that class here are the subclasses that we're applying right he's asking what is the open group and he's also asking about solution architect that's context that might drive the kind of response we give so if I ask about the open group and this is the dialogue and the dialogue has a set of nodes and those nodes reflect that different intents and entities and states so I can establish states by setting a parameter like for example if I you have previously set up if you've previously asked me about becoming architect certified I can save that state and I know that context in a variable now or I could actually just go back and look and see if the last thing you said about an architect you know what is either about getting certified as a business architect or as an IT architect as an example and I can create state and I and flow within the dialogue so I can mirror what your expected experience should be so for example in this particular dialogue you come in you get a welcome statement you're expected to probably ask about what type of architect or specialty that you're going to get certified in and then you drop down into the intents that are specific to those particular types and they in turn provide you with information about where to find more context around those areas so here I've got for example we look at this generalized questions generalized questions are pertinent to both and soon all three certification open profession certification program so they're general questions about certification within the open group under the open profession certification themselves so questions about accredited program certification converge certification certification pay where I can find you know help with respect to a claim I need to get an extension on my certification where do I find the open CA FAQ and then I I would also add one here for open sits I need a mentor I need support in other words I got to talk to somebody how do I get you know a question to a human being so on so forth you want to make sure that that last piece about getting to support is kind of the last option when using a chatbot because you're really trying to solve their problem before it gets to be right and many of these capabilities like at least the capability IBM has has analytics on the back end which is pretty important because you know the questions that you expect that are reported but when when you end up with an AI system that is as easy as going to a URL you don't know what people are asking and the example on draws provided about the linkage between certification and pay was a question that people didn't realize we were getting asked as often as we were getting asked so early on we have a set of ideas that represent what we think people are going to ask and answers that we think address that but this system is a stake in the ground it's almost like a customer sentiment station I can hear questions and later on give you answers and inside around what do people really ask about yeah maybe when you tell people that there's not a direct linkage between certification and pay they get really upset and that might inform the way you create your messaging and perhaps the way you address some of your policies so in the case of asking you know about the open group like the heck is the open group or just open group I would just say a minute's left on this section so we can bus faster short of the last piece what this does is says hey I recognize that this is a question about the open group or the entity the open group and here is the text that I'm providing back and a URL and the image of the open group and and in here I've got different additional answers that I can provide ask me about the open group open CA open sits ask me about open profession certifications and and you can do that random or sequentially so this is the back end that gets trained on this let's let's pretend like I need to add here I want to recertify what do I need to do so the question is really I want to recertify so I'm just going to copy that and you can think of that as your data curation step you'll see all the answer all the questions you get asked but ultimately you want to try to refine that into a signal that is obvious to the system or more obvious will say and I'm going to add an intent and I'm going to call this ask I can't see that does help you got a space between the underscore and the R yeah I know right so now I'm going to add an example I want to recertify certification keep saying that that's misspell why is that just another spelling how do I renew my certification that's a good one now it's important when we're creating these kinds of examples to represent a class that we need to provide a lot of diversity if we just kept on with recertification and recertify the system would likely be trained that anything with the word recertify or recertification is immediately about this that's not wrong but then we'd miss the opportunity to recognize question statements like how do I renew my certification my certification is about to expire I have expired certification so it's important as you're generating even for this rather simple example of a chat system you really have to make sure your data set is diverse enough to represent reality you don't want to bias it in one way or another so it fails to recognize all the possible inputs a user may provide so in in this process I am setting context so that the profession is equal to architect so I know I'm that the person is already either selected architect or or specialist as their as their profession so I'm gonna go ahead and go down here and and add it to my add to the dialogue here a node and I call this node and if the bot recognizes the intent to recertify then I want to respond with the text that that was provided or at least part of the text in this case and finally I want to wait for the user input now so now I can try it in my user interface and as you can see here and this is just one way that I can test this out we're gonna talk about modalities here in a second so when I bring this up it starts off with hello welcome welcome to the open group profession assistant I'm here to help you with the open profession questions ask me about open profession certification or about specific certification programs like open CA and I can say I want more information on open CA or profession certification and what this test dialogue tells me is that well it told me that the open group was and I probably misspelled something so let's try this again I'll drive if you'd like no it's okay all right there we go I did miss so you can see right here that it found that I was asking a question about profession certification and it returned the information on the dialogue for it professions the open group certifications are globally recognized credible and portable validation that you have the knowledge skills and experience to get the job done and it goes on and on and it gives me an opportunity to go out and read the rest of the profession certification for sure but in reality I really want to know more about open CA so I can say tell me more about open CA and it says you can find more information about the all the open group professions at so on so forth now really I got an error here I want this just to actually give me context with respect to becoming an architect becoming certified as an architect or profession certification so I'm going to pick that and the model is actually going to retrain it says Watson is training so that I get the right answer here now part of the process here is actually training the AI so that you get the right answer at the right time and it's not only training it when you're developing it but training it as people are using it and we'll show you how that works in a second so there you go now I'm getting more information about open CA right out of the box there and let's see I want to get certified let's see here and I got an error and it's retrain so this highlights some of the complexity that exists because in this system you have on one hand a set of classifications at the sentence level another a set of classifications at the phrase level these are all distinct data sets that you train in real time as you develop the system powered by both of those sets phrase level and sentence level classes you have this ultimate the brain and a logic of chatbot and you know these systems as you're seeing all have separate components that might evolve separately the classification and the data behind those classes may change over time the logic that he's showcasing here may change over time based on the business process so even for this simple example of an assistant as we said chatbot was cherlish there's still some complexity and you have expertise here people who know the questions that are being asked but it takes some effort to characterize you know what are people going to ask how do we want to answer them and then how do we connect those two to some experience that is meaningful so we collect data that's relevant and we give them answers to the questions that we think will resolve their resolve their concern this is one of the modalities is just a embedded web page and it comes up with the information about the profession certification and here I ask about open CA and I don't you know what's funny is that I'm getting different you know answers from when I was on my machine have you did you move this I didn't move it anywhere might be some of the updated training you're providing let's see here I want to get and this in real time kind of highlights the fact that I've made a change to everything and it's not actually providing the output that I expected so this is again representative to the complexity I just described we've got the set of sentence and phrase level classifications and as we add to that classification as we add more data more classes we influence the underlying model so that a sentence that would have been classified one way is now classified differently and in our exercise right now we added a couple of new classes we edited some existing ones all of that has effects that you don't see until you do testing yeah so in our case we're here showing you some of the raw stuff that happens and well at the same time showcasing the kind of business application that exists this complexity and under the hood view is not meant to scare you but just to inform you know the message we're sharing right even something as known as this kind of use case requires some amount of coordination definite amount of discipline shout out to DevOps because there are DevOps ways to test a lot of this to make sure that your system continues to be consistent this is for things like your reference data set can be used to validate that when we made a change does the system work the way we expected to very similar to unit testing and traditional software applications yeah and nothing like programming on the fly during a conference but I can actually delete this and probably get the right answers there's also a let's see here we've got a slack bot modality where we can ask questions via slack bots here start over you can always just say start over and I'll start me from the top I want to be to get open and it said I don't know what the heck you're talking about you had to ask me about architects so I'm just gonna say open CA and how it tells me about open CA and I would like to get certified as an open CA IT architect and it provides me with more information about how to get to that there we go it tells me that I can get in a self-assessment use the self-assessment tool and and where to get information about the fees so that goes to show you that you can embed this in different types of clients I'm gonna have to actually figure out why that I retrain the model to to break but hey that was a good exercise that at least showed you that you had to be careful about how you actually program these things but so prior to breaking the model I was actually getting all of the right answers about where to find information about open CA open sits I could go down to the stream I could ask how to get certified you know as an enterprise architect which seems to be working right now well and what's noteworthy and meaningful about showing something like an integration with the chat application like slack is slack has a web experience a desktop experience but also a phone experience so by having this AI system that we've now trained on this open CA these open professions and we can now see it experienced in a different modalities and most AI systems are likely going to have different manifestations across you know web thin clients thick clients and even here mobile and courtesy of a mobile device I have you know my mobile device providers in an automatic speech to text I wanted to yell at this thing and get an answer back so these are some of the ways that you take these basic AI capabilities and plug them into what is the existing ecosystem and end up with very rich user experiences where you're not responsible for the entire implementation so it actually is finding the right answers now so I'm asking a bunch of questions about you know where do I find my bad where do I go to get more information about where my badge is that's in a claim so I've embedded here images from the open group about badges you can do all sorts of fancy things like that you can pass context in about who is logged in for example you go into the open group website you log in you can pass in the user information you could theoretically integrate it in with well not theoretically you could integrate in with the the certification system and have it spit out very specific information about the the user that is using the the chatbot just like you would with Erika and the bank so that that that is you know all I have on the the chatbot itself the artificial intelligence model that is using is using natural language trans translation understanding and it's building a context model for the corpus that we've created around the certification entities and the questions and intense that we had set up so so we've got a picture we can show before we jump off this I know we spent quite a bit of time but I think it's worth being in a bunch of architects so this is effectively what the high-level architecture what you've just seen right you have some user experience could be web-based could be slack-based could be a robot and they have some questions some statement we classify it we enrich it so we understand somewhat of what they're asking somewhat of you know given the context of what they're asking about what are they asking and then we come up with some answer based on a context of sentence level and phrase level classifications and ultimately we give an answer and hopefully they like it if not they yell at us and we figure out how to do it better next time this high-level architecture is something that can be extended and interpreted different ways for different contexts in the case of open CA we saw a system built to be a bit of a customer service system right the purpose of this was to make people to give them ways to understand how do I get information about these open professions and how do I do it without having to bother him because he doesn't read you know with hundreds thousands of emails right you don't need hundred thousands more but if you have a system that can understand what is these what are these people really need then they can provide that to them in real time so as you see this system in front of you you can imagine an enterprises right other chatbot use cases that are also common things like support help desks for it as well as other support help desks for core enterprise functions both internally and externally and as Andross mentioned a great thing about many of these systems is you can integrate them with some third-party system you know whether it's an existing enterprise SSO or some customer some customer experience so you know before they even say hello exactly who you're talking to and that can inform some of our dialogue as far as the answer we provide for instance the answer you give someone like him may be different when you ask about certification versus someone like me who's not yet but soon to be certified so now looking at another system which is similar but with a different context and we're going to look at another question-and-answer system but in the context of Togaf 9.2 now Togaf 9.2 for folks who are aware of the open group there's a great open group standard it describes different ways that we can create and do enterprise architecture there's a lot of great expertise and many folks may even get a certification in Togaf to show that expertise one common occurrence though is that folks who are new to Togaf even if they're certified may have some difficulty in understanding how do I do a thing with Togaf I mean they may be certified they may have good expertise and good mentors but they may just not understand how to translate a real-world issue into something like Togaf or they might know something about Togaf and they want to learn more about that something so the approach we've taken here is to take a similar thing like we showed with a chatbot but now oriented to Togaf and the case of Togaf we have a document that's very knowledge heavy so what we did is run it through a data pipeline similar to what we showed earlier and create an application which latency permitting we can use to run queries against this application serves to provide some kind of insight into Togaf now keep in mind our core user here includes people who are experts at Togaf it may likely know answers to people who have never heard of Togaf before and they don't really know where to start but they have some ideas they have some words that they know that carry some meaning so the way one would interact with this system as you take whatever question you have about Togaf and you know Togaf pretty well andross what's a good question let's see what are the stages in the ADM architecture development let's be easy let's make it a long version architecture development method and let's see how latency treats us today so what's happening as we're sending this question right this is question is being sent in this implementation as an unstructured query so this sentence is being diced up it's being tokenized they're using traditional search technologies but they're also using a bit of AI empowered search each of these documents relating to Togaf have been you know processed by AI digested into pieces of meaning segmentation is what we call that and it's been presented at within an AI empowered search index so when you do a search and you ask this question it goes beyond just the simple tech search behind the scenes we could implement more customized training that maybe when we see architectural development method we also look for instances of ADM we may also do the reverse kind of inference that if you see ADM we might extend that to be architecture development method whatever the case we do a search and we get back a set of a set of candidate responses now these responses are again chunks pulled from the Togaf standard itself in this particular system we've developed they're just provided to us with some metadata and the link to see the extension of that section we don't make any additional inference or assumption from there because this system is really meant to be a basic example implementation so we see our top item here with a score of 0.5 that's this system sense that I think this is relevant to you this system under the covers has not been finely tuned to know exactly what we think is relevant we've just loaded documents and worked with it in that state to showcase just where AI algorithms can be without much training so the first section we have is called building blocks and this is a section likely if I jump out here real quick from the Togaf standard itself and if we open it up we can see the actual text from this section and the the parts of our search query that were found now in this case we're doing a fairly rudimentary fairly rudimentary search so the responses we're getting back include the tokenization of each individual word highlighted here there's also a possibility for us to take this kind of query and again break it up into a search just looking for passages where we ask the AI system under the covers to go a step further and not return us the whole answer the whole section but instead to return us what it thinks are the most relevant sections within that document and so here within this section we see some description about the Togaf ADM we see some description of the different building blocks within Togaf ADM as well as general characteristics of those building blocks we can because we had to break it up we have there are 109 documents that or segments of Togaf that match with 93 positive sentiment 6 neutral and 10 negative I would assume 10 10 negative or like anti-patterns or something like that you know because this is a methodology but for whatever reason somebody wrote those 10 sections in a negative tone and that's that's exactly correct and what's important to distinguish here is this system in the current basic implementation again is not doing a very complex processing of the question so we've not we've intentionally done that to showcase what systems can do sort of without much tweaking what what what one can do is one of all the system like this is progress it by adding more advanced analysis of the search phrase and I'd like to showcase that architecture here so this represents a possible extension of what we've just shown where instead of passing the question directly to your search index that has AI behind it you do several levels of pre-processing of your question this pre-processing can be similar can be thought of in a similar fashion as to what you saw with the profession's chatbot we're going to try to classify the question to figure out what manner of question is it and we're going to try to do phrase level classification to say you know he's asking about a relationship and he mentions togath and he mentions ADM there's an implied relationship between those two concepts that we think is meaningful so to showcase that at a very high level we I've actually got another very simple system here and this very basic system just accepts a phrase and attempts to digest it using a very shallow set of training data that we've provided so in this case the training data here is around asking some of the similar togath questions so this system is trained on our custom you know sentence level classifiers and but it also has phrase level classification that's untrained so this is stuff pulled from things like Wikipedia so if we send this question out we get back is a classification of our question again this is based on the training data in that spreadsheet it's very shallow we trained it that based on questions like this we think you're asking about inputs we think you're asking about you know at what point in this ADM process do your business continuum requirements serve as an input at the same time in our untrained model that does you know Wikipedia style analysis of phrases it picked up ADM as an organization now we know that's not right we know that has different meaning but this general purpose model that we're showcasing perceives ADM as the acronym representing a company because in most other context you know an acronym like IBM like TOG for the open group does represent an enterprise or an organization but so if we were to take our search system on top of togath and evolve it further we would do an approach like this to try to understand at a deeper level the semantic meaning of what it what is a person asking about combined with about what are they asking and use that to do a much more targeted query so another system that we'd like to at least bring to mention as another possible example of something with AI aligned towards you know some of the business of standards though unfortunately we don't have an easy way to demo it is a conference called transcription system within the work of standards development there's a lot of great dialogue you've heard some of it here in the conference today people presenting people sharing ideas right now there are folks in member meetings having discussions about problems and having spirited dialogue about it's this way no it's that way no it's a service no it's not a service etc an important part of generating standards from that kind of activity is taking notes taking minutes on recording what people are saying and distilling that into a sense of significance where you say so-and-so from IBM said this and there's the implication of it so-and-so from another company said that here's the implication or the action inferred from that so in a current process these minutes are generated using an artisanal process they're handmade and there's value in that because there's a lot of context required for many of these minutes to be meaningful you have to know what the group is doing to understand what's the significance of what someone just said so another way this could be empowered by AI would be the use of AI for audio transcription so in the case of a system that you know it exists on my laptop but not something fun to demo you can play audio to the system and the system can transcribe the text of what people are saying and then send it back as text associated with that meeting this represents a fairly low hanging fruit when it comes to AI application where things get interesting and potentially a little scary is what you do with that data because in that data set you now have a lot of insight coming back around who said what what is this forum talking about in this forum's meeting yesterday that they talk about that forum and those kinds of insights are valuable and useful though for our initial example here we looked at it more from a cost-saving perspective of figuring out how to help automate the process of minutes so really for many of these systems it comes down to understanding the question that you're being asked as we saw this is not a trivial thing but it can be addressed if you're very intentional and narrow about how you want to scope your your efforts in the case of the chatbots you have the brilliance of your chatbot engineers of your business process owners folks who know the domain well like Andross and some of the staff at the open group in the case of something more open-ended like Togaf you have a sea of insurmountable questions the ones you know people are going to ask and the ones you don't know and you can attempt to use AI to create intelligence around that and you can also attempt to use a little bit of staff expertise to train the system to be able to you know decompose those questions into some kind of meaningful query but ultimately you go from that question from that unstructured data into some kind of structure that enables deeper insights that enables things like entity resolution that enables us to understand that when someone asks a question to our Togaf system they're specifically talking about a particular concept within Togaf that we can have the system meaningfully assert is part of this section so at this point we've we've sort of concluded the live demo section session and we've done it probably 21 minutes over so we've got about nine minutes I mean one of the things that I didn't show you was that we can actually get information about how the chatbot is being used and here we have analytics that show the conversation and the amount of conversation usage and the top intents and the top entities that were utilized and this gives us kind of an idea of whether or not we're getting the right information about that particular intent like you know that we can select the open group and see you know what the what the context was there we can see that there were a total of 12 conversations about the open group and that the conversation over the last few days you know dipped and then increased in context so we can also take the logs and look at the types of questions that are being asked and determine whether we're getting any errors or not and then we can retrain the chatbot based on what you know that that particular data showed and you'll see some of this process might remind you of some of the work you do in software development you deploy a system you have it in a point where you think it works and then you just have to continually analyze it see how it's working modify it update it as you need to and this is where we see you know a lot of importance on treating that as a separate formal process hence some of the guidance we've given so far and some of what we see with many clients today yeah so one of the things that you would do is probably pretty this up put it on another interface maybe even change the modality so that you could use just natural language recognition and one of the other things that we can do without any kind of effort whatsoever is run it through a translator and go back and forth between English and another language and and talk to somebody who is a non-native English speaker so they can say I'd like to talk to you in Japanese without any effort whatsoever and I don't have to spend any money on translation now that approach is one approach to localization IBM has done both in some of our larger enterprise systems we try to compare answers in the translated language to answers in the native language itself but based on the complexity of your system the the the sort of translation upon question received will likely work for many use cases and there are some parts of at least IBM where we do use that to take advantage of providing you know multi-lingual experiences without having to invest in deep translator expertise for you know variety of translation tasks or you could just do the translation and see if it's right and tweak it right so you don't spend a lot of time with it that's the other thing so and you could use the information that comes out of this as analytics to tell you about what the sentiment of you know working with the open profession a chatbot is you know certainly and though you know for many of these AI projects you initially build them with the intent of addressing a business problem like automating the creation of minutes or saving the staff from having to respond to a bunch of important questions that how have the same answers but as Andres mentioned over time you'll create data that allows your enterprise to do the next step then maybe you look at all the questions you're getting and determine that hey a lot of people need guidance on one particular part of OpenCA perhaps we should change or add additional guidance to that section right perhaps we should evolve our offering and the documentation for that offering similarly to the Togaf example if we find a lot of people are asking questions about something we don't really address in Togaf maybe that now informs us that that's a new area of extension and this is you know an example of how organizations truly become data-driven is that you have systems that feed on your data thrive on your data and generate more data in turn that you can use to evaluate you know the next best step for better business outcomes all right where do we go from here well we've got some lessons learned to share in the last four to five minutes and I'm sure we can jump through those relatively quickly oh bias my favorite and brand identity so bias my favorite example of bias recently it was a situation where a company was actually using AI to find the best candidates which by the way we do actually do an IBM we score your skills and we actually are trying to rate employees based on AI assessments and those assessments actually use things like information about the skills that you attain through a claim the classes you've taken in within the formal training you've taken information about your social eminence how many assets you've contributed stuff like that don't forget your certification and your certification yeah but recently a company actually was using a model like this and they found that it was selecting against certain universities and potentially against women and and you know when Michael and I really dug into their model the AI was doing what it intended to do find the best candidates but it was using some data that it got from a few sources that was leading it down the wrong path right from training but in reality did they really need to have gender in there at all right and that's you know that's the question I mean this example right this organization looking for the next big tech talent and it just so happened in their existing data set of the people they had hired many of them were men and the AI system which you know they didn't have hands on everything kind of reasons well what do these top performers have in common well one of the thing happens here that they're men so what ended up happening when people were submitting is it would look and recognize mention of gender and score that as a not negative thing in the case of women but rather just a positive thing in the case of men because the system just blindly reasoned there are ways to handle this and I kind of have a laugh when I learned about it because they're relatively well known there are ways to hide the features that you don't want the system to learn from so the system doesn't seem male or female it just sees strong technology background maybe leadership maybe you know they maybe they're in the arts as well right gotta have your balance but when you give it data that's not curated or shaped properly the system will pick up on weird things I mean I was a little sad to find out that the system didn't reason that people with dogs end up being better employees yeah in my sample size of one that's the trend I see but in reality to you might want some bias in the system you might want to have more minorities because you don't have enough minorities and even though they aren't really you know kind of bubbling up to the top of the performers list you want to promote them or score them higher to assess them sooner in the cycle so in some cases you you actually culturally want bias put into the system so you know it's a little tricky dribble right yeah I mean in the case of the tech company again they might notice that their best performers have formal tech backgrounds that just might be unarguable and I know some people say hire the bachelors in history and not to say they're bad programmers but let's say their data today says all of their strong folks have computer science undergrads and maybe masters as well but you want more diversity because you say we need people who don't come from strict tech backgrounds but learn tech later you might have the system say if you're a non-tech background like Andross was mentioning let's give you some extra points let's not exclude you immediately or let's score you in a separate pool with separate parameters that being one of them yeah that's one way of doing it so you look for top performers but maybe they don't necessarily have you know tech background or the same tech background but they're trainable just like the system okay so we're formally on the hour but I think we can check through what we've got so you know this is bias comes in different forms and you know some of the challenges that you face for deployment of these things into the enterprise you know that that kind of boils down to trust and transparency is around the difficulty in you know integrating it into the business applications themselves managing the you know internal policies the resistance to AI and lack of DevOps or those those skills that we talked about earlier that aren't necessarily readily available you're gonna have to train them and maybe not even understanding the analytics of the data itself right so you know there's three different or four different particular roles here that we really have to you know focus on we talked a little bit about this before but there's you know building you know the the solution so the data scientists roll the running of the solution the creating the solution which is part of the software engineers responsibility this whole idea of AI management and business user coming together and and those folks have a lot of responsibility and trust and integrity to make sure the model is working the right way so we did come up with a project that we call open scale that we're inviting other companies that were open sourcing and it's it's really kind of scrubbing intended to scrub your model to make sure that you're not putting an unintended bias into it things like adding gender or or ethnicity or biasing against a certain school let's say that you know you're you all sudden your AI models are picking West Coast schools versus East Coast schools if it doesn't know anything about them then possibly it can't bias selection from you know bias folks who go to the University of Virginia versus you can't you Cal Berkeley or something like that right so open scale is intended to you know actually look at payload logging from an integrity point of view making sure that there is a visibility into how the model is performing operationally be able to more fully explain the model define some tests to defer the term in fairness so it generates data for you to actually run through the model and then creates that model ops piece that we have been talking about and this is really important because bias has actually become an inhibitor to use AI because people had you know looked at the unintentional you know use of gender that was included in the model previously so how does AI impact your brand Michael well you know as we saw the couple of interactions we had with the sample systems we showcased you have whenever you interact with an IT system representing an organization or providing expertise attributed to an organization it in essence becomes a representative of that organization this is why when I work with any customers to build any AI systems that are externally facing I always have someone from marketing in the room and I always have someone who represents their business transaction or interaction because these AI systems ultimately define someone's experience like if someone interacted with our chatbot and managed to confuse it they'd say man this group is crud you know this certification is no good and we all know that's not true but the experience they had would give them that sense they would walk away with that same idea if they interacted with someone with the open group who was just as rude because that's a frustrating experience now if I if I get questions I'll send them to Michelle because I know she's nice so they'll like the open room but this is a key thing you have to understand all of these brand touch points becoming automated still has huge implications this is why we're big on human in the loop for most things because this human experience that AI creates again is going to define that market that brand identity you have in the market yeah so there's a few approaches here there's five to be honest with you and obviously and we've showed two of them you know the customer service interaction with chatbot and enhancing the work of the knowledge worker getting insight into the structure of toga through the use of IBM Watson discovery so but there is also managing complexity and risk so we integrate AI into things like Watson cyber security so taking massive amounts of data that's coming out of the enterprise on how your security is functioning is certainly a good model using it to find the best talent so we do actually use that in our talent systems within IBM and to empower developers to actually create AI based applications themselves right yeah I mean in our case we shows a few example applications but in reality you know you would build up the underlying AI systems and then that's a thing you can integrate with for other experiences you could take that chatbot and have it a phone line that one calls in for example instead of just a web based experience so I mean here's you know just some other examples of some of the ways one can get started and work but again it really follows pretty much everything we've been describing the way you can sort of improve some of these business processes target what's critical to your enterprise and figure out do I have data around this and can I apply just enough AI to get started preferably AI someone else built where then I can just focus on deriving value from that interaction now subject you really like well when it comes down to these systems at the end of the day the architecture still matters I mean in our example of showcasing some of these few sample applications right we edited a model on the fly and things got crazy and Harry very quickly that's all realities right these systems have an inherent complexity and maintaining them and making them successful still requires architecture architecture from the application perspective architecture from the data perspective from the model perspective in the enterprise systems that we've built doing even chat stuff you've got multiple environments you've got some kind of you actually have a change review board a change management board I know we think it's a nice word but you know those processes still serve value and architecture is a core part of that that's certainly true and we tended to throw architecture out with the you know the baby with bathwater kind of analogy we went up we went to agile we went iterative we created this idea of minimal viable product but the product is not really viable and it's sometimes not even minimal but all the data is generating is building up technical debt it's not the right solution so you definitely have to think in terms of the illities because right now the agile and design thinking is all about outside in and it thinks of it in terms of how the user wants to interact with the system that's great but as we know you know from the open group a lot of the success of your system is all of the illities the 40 different illities the non-functional requirements that are necessary to build a system that's maintainable and you have to begin to think about the architecture from the inside of the system out instead of the the end user perspective which is mostly what we're doing these days I think that's it yeah wow we made it to the end any last-minute questions before we let you escape it is 509 nope so if I had actually created an intent that was bridges versus badges I can actually go back and fix that in the model pretty easily if somebody actually retrains the AI network to somehow you know befuddled badges and bridges then I'm probably gonna have to fall back to a past corpus yeah I mean it comes down to data curation right this is why Microsoft a suffered such an untimely fate if you let just anyone you know adjust the model you're gonna have a bad time and this is where I brought back the comment of DevOps you want to be able to move these things quickly in the case of our internal version of the nameless chat bot we showed we've got our IBM's global career team who works with IBM's technical career path and once a week they look at the data once every two weeks they propose changes to each other and once every month or so they actually make these changes sometimes they make them on the fly but you know we sort of empower them using DevOps methodology to make a mistake and if something goes bad they have a button I put that they push and you run your DevOps pipeline and you take the old stuff and you know throw it away and put back in the put back in the put in the new stuff or vice versa right I mean it goes back to that same fail fast fail forward methodology in our case you saw right the caveman version of it people typing oh I think it should be this or whatever and if you make a mistake going back it's really hard but with DevOps practices that we all know and love testing rapid iterations you can start to address a lot of that complexity