 Thanks, hello everyone, I'm from review 4, Han Sheng the Hansen, if you want to call me Hansen, I don't mind. So tonight I'm going to share with you how we are using deep learning to change the advertising industry and some of the issues and solutions that we have encountered and that we have found when using deep learning to solve our problems. And these issues you may have when you are using deep learning or other machine learning techniques as well. So I hope you can... Can you speak louder? Oh, like this? Yeah. Oh, hello. Oh cool. Yeah. And yep. So first of all about myself and the team. I'm the head of the data, but anyway. And my background is actually from electronics engineering and computational intelligence. But unfortunately unlike other speakers, I'm not from Google and I'm not from Facebook as well. I'm just from review 4. And our team we have an analyst with my left arm. And another infrastructure guy who is creating a scalable infrastructure. And my heart has a research scientist based in London. But my two legs are missing. So actually we are hiring. So we are looking to double our head count by the end of this year. And next year we will double our head count again. So please submit your application. After today's talk if you find like what we're doing is cool and you are interested in joining us, please submit. So to start the talk, since we are an attack company, right? So I got a... Oh damn sorry. Sorry, I need to connect to internet. Am I going to do that? Yeah. Okay so the password is umballfunan. So I'm going to attach that over there. Which one? Funan SG? Oh yeah. Yeah. I wonder use. Probably like their Facebook page or something. Because I'm using Windows. So I don't want to connect to internet. So if you have the passwords, don't pass it to me. Digital ads are great. But eventually their performance starts to deteriorate. And they need to be redesigned or refreshed. So time consuming. But with Refill 4, your ads are automatically designed and revised to keep campaign performance high. To get started. Let's go down as well. Okay. All right. Let's see. Started. Just connected ads are great. But eventually their performance starts to deteriorate. And they need to be redesigned or refreshed. So time consuming. But with Refill 4, your ads are automatically designed and revised to keep campaign performance high. To get started. Just connect your ad account to our platform. And create your brief. Which is sent to over 10,000 designers in more than 100 countries. You'll receive your creatives in 48 to 72 hours. And you can set them live. Request revisions. Yeah. But it's like a shit. Yeah. Yeah. Oh my. It's crazy. Yeah. I tell y'all to drag the coach in. Yeah. So I get presented. Okay. I have another one. It's here. Enjoy this one. Yeah. All right. Yeah. Yeah. But eventually their performance starts to deteriorate. And they need to be redesigned or redesigned. Your ads are automatically designed and revised to keep campaign performance high. To get started. Just connect your ad account to our platform. To create your brief. To send to over 10,000 designers. Ah. Okay. I think. So you guys, we are not doing good enough. So that's why we need people. It's okay. Just never mind. Breaking them in the order of printing and performance. Your ad is selected. Your ads. Yeah. All right. So I guess you guys are still not knowing what we are doing. So I'll explain. Yes. Okay. Basically we are quite a mature star. Actually we are profitable already. And for a star, this is actually quite like difficult to be profitable within like three years. And we have offices in London, Tokyo, Singapore. And San Francisco, Austin. But we are headquartered in Singapore. And currently we are like one of the major marketing partner with Facebook and Instagram and Snapchat. We also won the last year's Facebook innovation award spotlight. But this year we are submitting to the Kent Slion. So we haven't received a result yet. So in advertising industry today. Now they are like the global media inventory is like 20 billion. So estimated 50% of the media cost should be spent on your creative production. You should have 3 billion globally in creative production. However, nowadays there's only like 2.5 billion. We are having more demand than supply. That's why our business are booming. And this problem actually affecting the entire digital industry as well. Because more and more advertisers are coming to digital advertising instead of doing the traditional TV advertising. Or even newspaper. But somehow they are suffering several issues. For example, the performance. You create a creative, right? You want to launch an air. So you got to have your creative. Like be an image or a video. But you don't know its performance. So the only way you can do is to have more and more creative and try to test each of them. And the other thing is the relevancy. In digital advertising, those advertising platforms such as Google and Facebook. They don't show like irrelevant ads to users. They only want to show relevant ads to their users. Hence you can have a lower bidding cost. So if your creative is not really relevant. If the algorithm decides that your creative is not relevant to your target audience, right? They are giving you lower priority. So they will not spend your money. They will not show your ads to the target audience. Unless you are willing to bid higher. So this is the effect of relevancy. And the velocity with the more and more advertisers coming to this industry, right? Now people are pretty quite aware that like a targeted advertising is the key to the next generation of advertising. But more and more people are targeting the same audience or different audience, right? So the result is that you will have to target more detail and detail the audience. Otherwise if you have a very wide target audience, your relevancy will not be high. Hence you will not be able to spend your money and show your ads to your clients or to your target users. And what we are doing is actually that with us you don't need to imagine what kind of creative will perform. We have a fleet of designers that can help you create any sort of, given our domain knowledge and our understanding in different industries. We are actually creating creative in a data-driven manner. And we also have the machine learning algorithm to help you to predict the performance of your ad before you put your ads online. So that you can actually save your money and save your time if you work with us. And the way our process is like this, you want to create, you want to have some creative for your campaign. So this is called creative production. And you submit your brief to our platform. And we have our designers globally to create creative for you in a matter of few hours. Then you can select the creative that you want and put it online. After that if your system detects that these ads are not performing well, we will just swap it for you. And our designers will submit more and more creatives. So this direction keeps going on and making sure that you are refreshing your creative faster enough. Because if you don't, your creative die within a few days. Once, let's say if a user sees your ad, you are a user, right? You see an ad, then you don't respond. And then tomorrow you see it again, and you don't respond. The next day, you will never respond. That's called exhausting. So like what I say, the performance relevancy and the refresh frequency are now the three key elements in digital advertising. And what we have for tackling the problem is this engine called Shaka. I think he's a guy. And we use this machine learning model, given the data that you have. Be it like social media data or your targeting data or your creative. And we feed it to our Shaka engine. Then we tell you whether your ad will perform or not. If not, we'll ask our designer to submit a new creative again. Otherwise, it will be a waste of your money. And coming up, we will have sort of auto targeting for your audience expansion. Once you are advertising, you know that you want to spend your money, but you don't know what kind of audience you should target. You know some like a future, some profile that you should target, but you don't know which one is the optimal one. We will be able to tell you that. And you don't have a creative. You don't know what kind of creative that you should produce, so as to achieve maximum performance. We will have a tool to tell you that, even before you produce your creative. Let's say we all know that in the, I think the last two, three years, video ads are getting more and more popular. Before, people were mostly using image creatives only. And later we have like a carousel, which is a set of static images. But now we are seeing more and more video ads on Facebook and Google YouTube, because people realize that video ads are actually a lot better than static images. But however, producing a video, the cost of producing videos is a lot higher than producing images. And once you, if you don't do it right, you have to do it again. It takes time. And I'll show you some of our clients. We are able to increase our scale and reduce the cost of our client by like 70%. In industries such as e-commerce, online gaming, travel and entertainment, such as casino. Paypal, they are one of our clients. High profile one. And we also have Spotify, Spotify, Tiger, and Cooking Dash, Tiger Air. These companies, they have their own in-house designers. So actually, but the reason why they want to use us, because our creative is better than theirs. And if we don't charge them by the amount of creative that we provide them, instead we charge them by the performance of our creative. So if our creative don't perform, they don't have to pay us. So this is our like a value proposition. Okay, next I'm going to talk about the deep learning part. Any questions so far? No. Or you want to be our client, you can raise your hand. Okay, right. So I mentioned the Shakarai. Actually it's not really a magic, but it's a multi-class, multi-target, air performance ranking classifier. Same like you can come up with a classifier with many other techniques such as SVM or simple regression, the sort of thing. But the reason why we want to use deep learning, because we are trying to predict the human's behavior. Unlike predicting the, unlike a classifying image, you have an object in your image, then if it's a chair, it's a chair. You don't have doubt. But we are human, we see a creative. How do we decide we want to click on this creative or not? We don't know. That's why if we use a simple method, simple machine learning method, I don't think it will work well, given the capacity of other algorithms. So these are our input features. We have properties from advertisers, be it their industry kind of information. And they sell out a campaign on Facebook, so we have the properties of their campaign. Maybe they are, for example, if it's a cooking dash, they have mobile apps, so they want to launch a mobile app instead of a campaign. All other companies might be retailers. They want to have a conversion, so they will launch a conversion campaign. And targeting is the most important part. They always have a target, i.e. teenagers, so these are their asset properties. And the other key element is the creative itself. Either they are maybe the image or video, and we have metadata from the creative as well. So all together, we have 48, like a fixed lens or variable lens features. And give it to our chakra engine. So what is our output? We actually output like seven target values for them. These target values are actually the metrics that those advertisers concern. Maybe the click-through rate, or conversion rate, or light rate, or video viewing rate, or video play rate. If you run a conversion campaign, you will be looking at conversion rate. If you run a brand awareness campaign, then you might just want to look at click-through rate only. So these are some per-sample input features. These are actually from Facebook targeting. Everyone uses Facebook, right? So we see Facebook ads every day. I don't want to see them, but they still show me that. And actually, we are all being targeted based on the websites that you have visited or the keywords, the things that you gave Facebook. So for example, they know your family status, whether you are a parent or not, and how many children you have. And they also know your interests, like vocation or others. But some of these features are input by users. So you don't have a fixed set of possible inputs. For example, the AAH goes to Singapore here. This one looks a bit weird, but it's there, defined by lots of users. Others might define vocation, but sometimes they can be typos. For some vocation, vocation, or Star Wars with space or Star Wars without space. They mean the same thing, but the terms are quite ambiguous. These are country codes. And colors are the data that we have from our creative metadata. These are like a sequence of colors. If your creative is a video, then a video contains several friends. So we analyze the video and general metadata are telling us the salient colors existing in these friends, key friends in the video. So basically these features, they are largely recurrent. By recurrent, I mean, if you look at the, for some family status, you have a sequence of words here. Parents back a war or parents with early school age children. So this is actually a sentence, so a sentence sequence of words. And furthermore, if I have, for example, the color here, one image contains several colors. Then I have a video, so I have a sequence of sequence of sequence. So they are sort of in nested recurrent structure. And these are, here I'm showing you some like a target, target value that we are trying to predict. Three of them, one is the CTR. This CTR is the click through rate and the CVR is the conversion rate and video viewing rate, video viewing rate is for video campaign. So, and why are there like several courses here? It's because for a certain kind of campaign, you only have few metrics available. For example, if you are running a brand awareness campaign, then you don't have conversion, so you don't have conversion rate at all. And if you are running a video campaign, so you have a video viewing rate, but if you are running a campaign, you add with a static image creative, then you don't have video viewing rate. So this is how our data look like. And so now here's this thing. My data actually, my features contains the industry or the campaign type. So all the samples are from different industries or four different kinds of campaigns. And there are also three kinds of creative. So if they are actually lots of distribution, the combination is over 100. So traditionally you should separate the distributions by creating independent models for them. But I have over 100 distributions. I don't want to create 100 models. And from marketer's experience, they know that if I launch a campaign with a video, I know it's most likely 90% of chance, it's going to be a lot better than having a campaign with just image creatives. Why? Because people are more responsive to video. So my model, if I create a model, then my model for sure tell me that, okay, now you have a video creative, right? Then my model will tell me, okay, this one, you should get higher because it's a video creative. So you should use a video creative. But this makes sense, but is it useful? Because a lot of time, for example, you have a video creative, but you just cannot... Oh, I'm sorry. If you have only image creative, right? The model tells you one company tells you, okay, you should use video creative. But you just can't because you don't have time and money for producing a video. Or you are in the, let's say, game-boarding industry, right? They tell you if you were in the, like, travel industry, the purpose of your app will be a lot higher. But you can't, you can't switch your industry, right? But because of the way you mix the distribution, right, your model will only tell you if the campaign is from the travel industry, it's not going to have higher purpose. Whatever it's not from the industry, you have lower performance, right? So what we did is we still had, we still tried to maintain one dataset, okay? But for each distribution, right, we transformed those numerical targets, i.e. our CTR, CVR values, right? The shape of them is looked like this, largely skewed, always like that. But we wanted to be more of a normalizer, Gaussian distribution, right? So we transformed those targets with SparkTux. So after transformation, the distribution of... One distribution only, sorry. Let's say conversion here, I'm using maybe conversion campaign plus, like, gaming industry and video creative as an example, right? So this is one distribution. For each distribution, we do this so that we have, like, a proper Gaussian distribution in target. Then the question is how do we decide that what kind of... what level of CTR is good, or what level of CVR is good, right? You can look at all the values of the CVR of the CTR and decide, okay, you're going to have a threshold. Anything above this one, you're going to label it good. Anything below this, you're going to label it bad, right? But we don't know. And I can't do it for, like, a hundred distributions, seriously, yeah. So we further apply the, so-called the inverse of the Gaussian CDF to figure out all those separation points, the threshold here. The yellow lines here showing four thresholds. So I have actually five classes, right? So for each of them, I converted those numerical values into scores, five, with the five levels, okay? Then I put them in one model, in one dataset. So the dataset becomes this, okay? I have now see categorical values, but for, like, invalid, for invalid targets, I level it as negative one. So this is my, this is my, like, training data, okay? And here you may have a question. So if you have created, for example, a softmax classifier, right, you need to specify a non-negative class, but I am having negative class here, right? On the other hand, right, sample one doesn't have CVR target, but they are, normally people do batch processing, then they are always, if they are included in the batch, right? How can I predict it's CVR? Does it make sense or should I need it? Do I need it? Right, okay. So this is the model itself, right? Or simplified. Let me try it soon. Oh, great. Okay. I'll start from this part. Okay. Basically, this is a, what's your code? An R, there's an RN here. Record new network, right? So it takes a sequence of words as input. Then I stack two layers of it. So I have two state factors, right? And each of them feedback to its corresponding input state, right? Then the state factor of the last step, right? I use it as an input for next level, right? Which is this one. Again, this is RN as well. But here we are using words as input. And actually there's a word embedding in between, converting them into the factors. And here is the output vector already. So the input for these are actually a sequence of factors. And after that, we attach two dense layers, right? Okay. And similar for this one, right? So remember we have 40 something, 40 features, right? So actually we have seven sets of RN dense structure like this, right? And each of them taking different sequence of input, right? And the right-hand side, this is similar. Except that we have one more layer. And here we have dense layer in between, right? The reason why we want to have dense layer is because having dense layer in this way, right? You are creating like a hierarchy of dense layers instead of just having very large dense layers here. So your capacity of your model actually performs better in larger capacity. Although you can argue that, okay, you might want to have a very large dense layer here. But this way you have here, so like a sub-window of your input feature. Like a convolution neural networks. Why convolution works? Because you only look at the local features. Right. Oh, yeah. Sorry. So actually here you can, the input here is a sequence of factor. So this actually becomes like an embedding factor. So usually people do what embedding? Or they do sentence embedding or document embedding. But here I have an embedding factor for my input sequence of factors. Right. Right. And same. And same. Sentence. Sentence embedding. So sequence of sending embedding factor. So it becomes a article. Right. Okay. Right. So together the model has, if you, the model has over a thousand TensorFlow variables. If you have used the TensorFlow, then you will know that each TensorFlow variable actually is of a multi-dimension. So the total number of three parameters is like 34 million. Right. Which is not very big. Comparing to those AlexNet or Inception model, they have more than billions of parameters. Right. But this one so far is big enough for us already. Okay. Next we come to training. Right. Right. So because of limited capacity, so we got to have a strategy. I realized that I don't want to train the whole entire model at once. Why? Because it's going to take a lot longer and takes a lot of time. Right. If I can do the training layer-wise or module by module. Someday, if I want to update a sub-module, I don't have to retrain the whole thing. I need to retrain a few sub-modules and then train the entire module again. Right. So turns out this method is called greedy layer-wise training. People have been using, for example, auto encoder to achieve greedy layer-wise training as well. Right. But here what I did is that I trained this R1 first. I tried to come up with an embedding factor. Then I removed the decoder. Then attached it to the second layer. And then I trained this module again. Then I attached another two layers. Then I trained this module again. Then this one. Then finally joined training. Right. If without this layer-wise or module-wise retraining, right, my number of epochs that I can converge my model will be around like 50% more. Right. I tried it because initially I thought I didn't need to do a pre-training. And the other thing is people, you guys all know RELU, right? Rectified linear unit. So basically it's a formula called max zero energy input. Okay. This is pretty common and proven that are very effective in deep learning. Before that, people were only using sit-moil or logistic functions. And they realized that there's this called vanishing gradient issues. So RELU, rectified linear unit, actually tackled this problem very good. But somehow you would solve a problem that is upon your initialization, right? You might see some big units. Right. Anyone know this issue? Okay. Because every time I ask in my interviews, he seems no one knows. Okay. So that unit is actually that if you pick all of your training samples to this unit, right? This unit never generates positive output. I is always below the negative range. So no matter how you train it, it doesn't generate output. So your gradient never flow back. So this unit actually use less. And during the training, during the training, if you attach it, let's say... For some here, I always... If I had RELU here, right? Then because the activation function in this RELU layer actually hyperbolic tangent. So the output range of hyperbolic tangent from negative one to positive one. So if you have too many negative one, right? Then most likely the RELU unit here will be there. Because once the input is negative, then your output is zero. Or your connecting weights are too much negative. Okay. Right. So my experience that after training, you may still have at least 20% of dead units. Of course, you can argue that or I just increase the number of units that I have. Yeah, that's correct. But you will have more dead units as well. And that's the waste of your commutation capacity. Right. Right. And we train the model on the DeoTitanX machine. Right. The joint training part took like 8 hours for 45 epochs. Actually, I set it to run for 60 or 80 epochs. But because I have validation data. So if the accuracy doesn't improve for a certain period of time, I stop training. So I have to not waste my time. Right. Okay. Make sense? I hope so. Okay. Then I'm going to talk about how I came up with the embedding factor for those iron units. Iron layers. Right. So. Okay. So I think some of you may know the skip thought model. Right. It's actually from SkidGridModel. SkidGridModel says that if you have a word right next to another word. Then these two will tend to predict each other. Right. Let's say like I love you. Right. But actually I don't love you. But I love you. So love actually predicts I and you better. So if you see a love, right. Then you have a higher chance of seeing you after that or I before that. So this is the idea. Then SkidGridModel took this idea and expanded it to sentences. But of course they create, they use like a recurrent structure for it. Otherwise they would have been able to do it. Right. So we implement this model in this model intensive flow. But furthermore we expanded it to our so-called article embedding or document embedding. Right. Skip thought model only does sentence embedding. But we have a sequence of sequence of sequence of words. So we have article. And we need to do embedding at article level. So we do this way in this. We have an encoder here. Right. Which takes another encoder as input. And they are like several steps of it. And there's a decoder taking the output factor. The output stack factor of your encoder. Right. And try to predict your input here. Okay. Right. So this way we are able to verify and come up with sensible embedding factors. I'm showing you some examples here. I think let's look at the download shop. Okay. This is a sequence of, a sequence of words. Okay. Right. So this is one of our inputs. And the minimum input like granularity is what? We don't do it at character level. Okay. So turns out once we done the, after we done the training of the embedding factor, we found that by checking the cosine similarity, you can actually see which input is the closest one to your, to the factor that you want to see. So we realize that the another embedding factor representing Donald Trump is the closest one to here. And save for this one. But how about this one? Queen Ben and the Feddy Immigrant. I don't know who they are, but later I Google it. And I realized that Queen, Queen and this guy, they actually had some dispute with Donald Trump. So turns out obviously there were some advertisers targeting Donald Trump and they were also targeting Queen and the other guy. Right. Maybe they try to convert that, convince them that, okay, we are showing you Donald Trump's air and you should support him. Right. And the other one, the Fed 365, right. It has very, the distance, the distance between Fed 365 and the Ghana Football Association is very close within top three. I didn't know why as well. Then I asked my British colleague, right. Because they, I think he's more familiar with football than me. So he told me Fed 365 actually is a sponsor to some of the football teams there. Although they are gambling, gambling, online gambling website. Right. But they do sponsor football events a lot. And some other teams are as well. So if you are doing like online gambling and you are targeting Fed 365, you might as well want to target football association. Because people who like football association are most likely football players or football lovers. They are Fed 365 clients. Right. So this is how they do targeting and this is how we verify that. After we train the embedding, once we have done the embedding factory training, our embedding factory actually works. Right. Okay. Right. And there are some other issues as well. Right. For some of the memory issue. Okay. Although my Titanic's machine has 12 gigabytes of memory. Right. But it's still ran out of memory. Yeah. Because I have some videos now. So the number of key friends in those videos can be over 30 friends. Right. Which is quite a lot. So I couldn't handle all of them. Then my machine just told me out of memory. Right. So I'm showing you some here. Right. These are again a sequence of sequence of words. Right. Okay. But you can relate to video. Video is a sequence of friends of some metadata. Right. And the length of sequence can be as high as 680. Right. So, but I don't want to truncate them. Because truncating them, I'm not taking the rest of the sequence into account. Right. So I don't want to truncate. And one way to handle it is to have a fixed lens of. It's for your eyes to have a fixed lens. But once you have complete this iteration. Then you take the output factor. And you feed it as an initial factor into the same iron again. Right. But this is time consuming. And I need to do something for you. I didn't want to do. So I come up with quite like a smart idea. I'm sorry I said that. So instead of truncating, right. I did reduce the lens of sequence. But I didn't truncate them. Instead, I randomly sampled some elements in the sequence. But I still maintain the order. So example here, I have for some of the second elements in the sequence called call of duty. Right. In one batch, right. This call of duty may be taken out. Right. But in the next epoch, it may be there. So it's sort of random. Then after enough epochs, right. I at least I still know that each elements in the sequence still got trained. Right. And turns out this method works well as well. And so this is how I tackle the long lens sequence. Okay. And coming to the question that I asked before. My dataset contains like a weird target class. I did negative one. Right. I didn't want to include them in my loss function. Right. So how can I do it? So this is what I was inspired by. By the way, we do dynamic RN. In dynamic recurrent neural network, you have you got to supply a mask telling the dynamic RN that where your sequence is going to terminate. Right. So same idea. I apply it to my submax loss function as well. Right. I mask those negative one classes. So for CTR, right. The submax loss includes all samples. But for CVR, it includes only one sample. Right. And for video viewing rate, it includes only one sample as well. Okay. Then you do averaging, then you're all fine. Right. So given what I have said, I transformed those distribution into sort of normalizer and comparable scores and put it in one dataset. And my dataset, my model is able to handle this kind of situation as well. Right. So I don't need to create like four or five models like what I did before. Right. Okay. And okay, so the performance of the model is like this one. Remember I mentioned we are trying to, what we are trying to predict is human behavior. Right. So the accuracy or error rate of the model is currently at 47% for five classes exam match. I say five is five like that. And this actually, this is based on very little data, not training data. Okay. And my average score is 0.427. Okay. And okay. But we have one, the previous generation in production. Right. And here you see the relationship between our prediction and the actual CTR of those ads. Right. So. Oops. Okay. Here we are seeing positive correlation although not all of them. But at least the creative that is, that was scored five performed at least two times better than the worst ones. And I believe that the newer version should be better. Okay. So we also have a marketing teaser for you guys if you want to try and view our marketer. Okay. You can go to the analyze my ads with your world.com. Right. Then you see the interface. Follow the step. One, two, three. Upload your creative and tell us your targeting spec. And we will tell you is your creative good or not. But bear in mind, this one, this marketing teaser is a simplified version. So he only had three classes. Right. Because it's free. The payable one is in our platform. Right. Right. And so, questions? Okay. So it's the recovery bureau that works for you. You mentioned that there shouldn't be any dead units. But actually they are. Sorry. But actually they are. Yeah, yeah. It's the TensorFlow decay function. No, no. To get rid of new dead units, right? Just don't use RELU. You can use either RELU or what I'm using parametric RELU. Yeah. Okay. So I'm going to show you some of the other things that we are going to do in the coming months or the beginning of next year. Right. So we, oh, damn. Okay. The title is wrong. Sorry. So this is what we imagine the workflow of how advertisers should be creating creatives in the coming future. Right. What is it? Oh, yeah. So you are an advertiser. You have a list of targeting specs. But you don't know whether they are optimal or not. Right. Then you use our design recommendation AI API model. We tell you what kind of creativity you should produce. So that you can ask your designer, you can ask our designer to produce those creatives or video for you. Then you further use our performance prediction AI. Right. To see if these creatives are going to be good or not. Right. By taking more, like, factors into account. And from the base creative that your designer or our designer has created, we are further generating some variance for you by using some moving variation or expansion techniques. Right. And with all those, all those creatives in hand. Right. We will use our automation AI to help you manage your ads online, dynamically. Right. This is going to change the, how advertisers do their marketing or creative production. Right. And one thing that we will have to create is this one. Right. So you are an advertiser. And you know your target audience. They are teenagers. They are involved. They like gaming. They like to play football. But you don't know how to design a creative. Right. Then I will tell you, you really should have a human face. Or the content should be more about sports media or computer gadgets. Right. Yeah. Then we will also provide you with an example creative. Right. Right. And how do we do it? Right. Now, so we have had the black box, the X here. Right. So the X can tell you the optimal, the estimated performance of your creative. So if your input can maximize the target value, then your input value is actually the best one for your desired target value. Let's say I want to maximize my CTR. So what kind of input should I have? No. So I have the function already. So what I need to do is to use a solver and solve this optimization problem. Right. But the difficulties are like my input contains integers or words. How can you optimize input sentences words? Or how can you optimize integers? Right. Normally those solvers can solve floating points only. Okay. And next one, we will have semantic integers. Right. Our research scientist has created this prototype. So given an image and a sentence. Right. Yeah. Okay. Taking the blur image as an example. Right. And then I tell the model. All right. I want this blur to have a blue cone and wings. Right. And white breast. So this model, this thing that is model should generate, generate corresponding images for me. Given my input sentences. Right. So with this we will be able to do like a variation and the creative expansion. Right. And simple flow as well. As long as you have the base creative. Right. And the way we do it is this. This is actually a generative adversarial network model. Right. So our generator is encoder decoder structure. So our encoder takes the image as input as well as the embedding factor for your text. Right. Together. And encoder. So from the feature factor that this encoder has encoded, then you can do things like a deconvolution to generate the image. Right. First in encoder you do convolution and you also take the text embedding factor into account. Then decoder you decode the picture factor into an image. Right. So given the image. Right. You feed this image to your discriminator. Right. Your discriminator again use convolutional architecture to encode your synthesize the image and try to discriminate by taking also the text embedding factor into account. Right. See if your discriminator can discriminate this artificial image or not. Right. If you can, you have just created a good image that the computer cannot really discriminate. Right. Okay. Question? No. All right. I think that's it from me today. I hope you have enjoyed and we are indeed hiring. So I hope anyone who feel like you are interested or you feel like you want to contribute or you want to learn. Yeah. And if you feel like you know more than me. Actually a lot of people know me. I'm not the best one. Actually I want people who know more than me as well. So if you know more than me, please join. All right. Thank you. Mr. Xu once again. Yeah. And then this Friday, you'll see the half-time attendance. See how it's going to be here.