 So thank you everyone for joining to my session as well as our committee today. So I believe you guys are not sleepy or tired at this moment. Typically any software engineer, after lunch, you feel a little bit of sleepiness. So I hope my session, I'll try to keep you awake in my session. So let's get started. So today in this session I'll be talking about automating the deployment of software agents centrally using AWS Systems Manager. Can I ask from you guys, how many of you have used System Manager? Can I raise some hands please? Okay, thank you very much. And I believe most of you guys have had to deal with software agents. So because when it comes to security, when it comes to compliance, there are various agent-based tools that you have to deploy, especially. So in this topic I will be talking about to easy-to-instances, how we can deploy to multi-account AWS environment. So before I talk about the topic, I'll just introduce myself. I'm Charter Saver Singer. And I'm a senior engineer person. And I'm AWS Ambassador and AWS Community Builder. And also I'm volunteering at AWS User Group Singapore. And also I have a few certifications in cloud and DevOps on templates like AWS Kubernetes Azure, those stuff. And on top of that, I'm a musician as well. Can I plug it? Okay. Oh, thank you. So on top of that, I'm a musician as well, which is... And you can find me in YouTube. So I have been contributing to Sri Lankan films as well, as a singer as well as a music director as well. So I believe that's enough about myself. So let's talk... Let's see what we are going to cover in this session. So I will be first trying to introduce what is a software agent if you guys... Because for the benefit of all the audience, I believe it's better to understand clearly what is a software agent. And then what are the agent deployment use cases that we find? So I'll be doing a demo video actually because due to network constraints, so it's hard to do a live demo. So I'll be talking about that and... Sorry about that. And so I will show what are the AWS services that I'm going to use in my demo architecture. And I'll show you how my architecture works. And then the final demo video. And if time allows, I will try to give you some time for questions. Otherwise, yeah, we have to... Because we have a quite tight schedule. So yeah... So what is a software agent? So software agent is a self-contained software program. So I think you guys know... Typically you find installers like MSI that is for Windows and you know, RPM, you can install in like Red Hat or Amazon Linux and Dev packages you can install in Ubuntu. So these are typically... You can package your agents into one of these. So that's what I tried to mention here. And then... So what is the software agent? Typically it's acting as... There's some goal to achieve and also it's acting as a representative. Think about this. I believe if I try to simplify even more I think you guys have worked with real estate agents, right? And you have a seller and a buyer. So in this scenario also like a real estate agent tries to get the information from one party and give it back to another party. So there should be some communication between those two parties to happen to make the deal, right? So I mean software agent, it's also doing something similar and so it creates a communication channel between two parties. So typically this will talk... So you install a software agent in an EC2 instance and then the agent will help to communicate with a SaaS solution probably. So that's how the software agent is. I hope that is clear about software agent. So agent deployment use cases. So as I said earlier I would... I mean you would see a lot of tools in security and compliance. Those are agent-based. And these are used... I mean you guys probably using that for endpoint security and some threat intelligence and also software asset management especially license management and inventory. So these are some few use cases. There are many more. But I try to highlight some important use cases in this scenario which I see in most of the... because I'm a consultant as well. So we see a lot of customers. They have this business problem. So that's why I was trying to bring this topic here which is not too technical topic but I think it's around level 200 topic. But I believe that it is really important because you cannot avoid easy tools even though we talk about serverless even though we talk about containers we still have to manage and we still have to maintain the compliance and security for those instances. So I believe this topic is still valid and then the... So I will talk about here what are the AWS services that I'm going to use in my architecture. So as you saw in my topic the first one is the system manager. So system manager is kind of a very big service I would say. It has a lot of capabilities. But if I try to simplify what it really does it helps you to manage the easy tools not even that it can manage the on-premise instances as well if you can install your SSM agent. So I will talk about that later but then also like some tasks that you can do is applying OS patches, configuring the instances. These things are possible. And the good thing of system manager is you know it's a free service probably some of the small exceptions but it's really... You don't need to spend too much to use these services but I think in my opinion I still believe system manager is one of the underrated service when compared to other services. I don't know whether you guys agree with that but with these capabilities there are a lot of things that we really can do to ensure the security and compliance but I'm not sure like everyone is properly using that. So in order to use system manager so there are few prerequisites that we need to... we need to make sure that some things to happen. So first thing is you need to install the SSM agent in EC2 instances. But it's actually like this so most of the AMIs currently they have the SSM agent installed but some cases there are certain cases you might have to install the SSM agent manually or you can include that in a user data script in the user data when you launch in the instance. So in either way so you will most probably you can easily get the system agent in EC2 instances. And the next thing is access control access control. So in order to work so you need to... your EC2 instances needs some permissions you need to make sure that some appropriate permissions especially most probably you might have to include this policy in your instance profile otherwise it cannot call SSM endpoint so that is one thing and also if you are using system manager or your IM user or whatever your user need to have some privileges to use otherwise you cannot use system manager service. So other than that the instance... so since I talked about EC2 instances in order to communicate to the system manager endpoints you need to open the 443 port outbound traffic so that will ensure the connectivity between the EC2 to system manager endpoint but that is if you have the internet access but if you don't have the internet access there's additional thing that you have to do that is to create some VPC endpoints so there are around four VPC endpoints if I remember well like EC2 SSM, SSM message, EC2 message so those VPC endpoints you need to create to privately communicate from EC2 to system manager endpoints so these are some of the features like popular features I would say you can find in system manager I mean system manager service so run command is one of the simplest tool I would say you can find that it helps you to run shell script and power shell or whatever scripts in your EC2 instances and the automation is I would say it's like a big brother of run command it has more capabilities I will talk more about automation later I will be using that in my architecture so it has more capabilities like calling AWS APIs and multi-region, multi-account so you can centrally run automation this feature helps you to run centrally to multiple accounts and multiple regions and then the patch manager actually that is as the name implies it's useful when you want to patch your OSS windows or Linux and then the state manager also that I'm going to use so as the name implies it actually tries to maintain your state it tries to ensure the state so if you know about Kubernetes Kubernetes also they are when you define the ML file actually you have defined the state what it really should happen so in state manager also it works something similar so you define the state and state manager will take care of the state so it will make sure let's say if the agent needs to be installed so it will make sure that it's always there and the maintenance windows also I would say it's not the same as state manager but it's there are some similarities in state manager as well so and parameter store is to store your parameters and you can refer back so and you can store those parameters securely as well and the distributor is the one that I'm going to use to store the package and also to distribute the package to multiple accounts then I will talk in detail so there are many more there are many more but I don't want to talk too much about those so system manager automation I mean what it can run the playbooks at a scale so at a scale means as I said across the multiple regions across multiple accounts you can do that and API actions including cloud formation templates sorry, cloud formation stacks and also you can run Python scripts PowerShell scripts and also like service catalog self-service actions it actually is a very very powerful tool I'm not going to talk too much on that but you must remember this feature I think definitely it will help you a lot and then the state manager so here you have you can associate SSM documents it can be like predefined documents there are predefined documents already there existing in AWS so you can use that either you can create your own SSM documents so once you define the SSM association and in the SSM association you can define your targets and you can run whatever these documents on those instances and how you can run the association so there are four ways you can do so first way is so by default if you create a SSM association it will automatically run once when it provisioned and then that's it but if there's any configuration changes happen it will automatically run again so that's how the default behavior is and then if you want to run a like cron job like as a cron schedule so for example if you want to stop at RDS instance or whatever if you want to run in a schedule you can use cron or even you can define rather than hourly or daily or even if you go to the console on demand you can run the SSM association so it will automatically apply those to the target instances so state manager so these are the target types that it has so first way is you can target on a node ID so that means instance ID or else you can target on a tag and so if you have a proper tagging strategy and if you want to target on a certain instances only which have a certain tag so you can target on that and also resource group and also if you want to run to all the all the EC2 instances you still can do that as well and so as I mentioned earlier so this is about so imagine like if you create new EC2 instances after you create the SSM association if you create a new EC2 instance with a tag so state manager is intelligent enough to identify that new instance and apply the changes what it required to do so you don't need to give any other information just you need to tag the instance it will do its job so right and the next feature is the distributor as I said this helps you to securely store your agent package so don't worry too much on this I will have a demo as well so you will see that you can get a better insights on that so once you see the demo so as I mentioned here so you can use to store your agent package or drivers or whatever if its packageable you know you can use the system manager distributor and it has the ability to share across all AWS accounts you can set the permissions in the distributor package so you can set what are the accounts it can be automatically done or you can manually give that so likewise you can share that and there are packages I mean AWS defined packages as well as AWS have given some software packages as well as we can create our own one so in my demo I will create my own one and so let's say you want to update the software package so 6.5 to 6.6 so that will be a new package right so you can use the same definition but you can upgrade the version so likewise the version controlling also possible with this system manager distributor and also the controlling the access to packages using IAM so which is I mean IAM is one of the important service if you want to control the access so this is how the typical distributor package looks like so you need there are two ways to do that so first way is if you use the AWS management console you can you just need to give the you just need to upload the MSI with our PM and DEP packages then it will automatically generate the manifest file and the installation scripts so that's the one way of doing it but if you are following like infrastructure as a code and if you want to code I mean if you are not using the console then you need to prepare these SIP files so SIP file includes the software file and as well as the all the scripts and then you have to prepare the manifest.json file so then you can put that into S3 bucket as shown here so so then this is how the manifest.json should look like so you have the files defined here and you have to mention the checksum so but if you are using console you don't have to worry about too much it will automatically generate it so right then I mean so once you create the distributor what are the options that we have to install the package so these are the two options first one is the run command so you can run it I mean you can install using run command the other recommended approach what AWS also recommends is state manager association so I have already told about what are the benefits of it why so because of that even the AWS recommends to use state manager association so then control tower so in my architecture I will be using customization for control tower I will show that later so if you are dealing with multiple AWS accounts I think you guys already know about control tower and what is the benefit of it and especially if you don't know about control tower and landing zone I just give you very brief introduction like landing zone is a pre-configured multi-account environment based on best practice blueprints that means like AWS helps to set some best practices on your landing zone so control tower helps to automate this thing the creation of the landing zone so that's what it really does and it's more secure and scalable so because you have an account factory and you can create the account from once you set up your control tower so using that you can do that so yeah and so this is the customization for control tower that I wanted to share so here later I will what I will do is my cloud formation templates will be uploaded to code commit and it will go here it will trigger the pipeline and it will deploy the stack sets to the managed sorry the member accounts so that's what it really does so in my case there are two cloud formation templates I will show that in the demo so these are I mean this is how it triggers based on the control tower life cycle event which is not relevant to the my topic what I'm trying to show you here so demo architecture so this is how I mean how I deployed the software agents to multi-account environment so you have the management account so you have the control tower and I have deployed the control tower customization this part and then I have prepared the manifest file and there you have the SSM association definitions and also the agent distribution definitions so it will go it will once you commit the code it will go and deploy the stack sets to so SSM association goes to all the member accounts and the agent distribution stack will go to the shared services account so I have used the shared services account because because this this is a security agent that I wanted to deploy and I want to share it across the from one place to I mean the member accounts so this is I mean typically you might have heard if you have a control tower environment it's not by default it's not creating the shared services account but typically in organizations we maintain a shared services account for such purposes so here so this is the SSM distributor package that I'm creating so it's another SSM document and here what I'm trying to do is imagine now if you have control tower and if you have multiple accounts so there could be new accounts creating gradually right so it's not a one it's not just static right so let's say if there's a new department or if you need a new environment so the people ask for new accounts right so we have to make sure that especially the security agents those are deployed to all the accounts it covers all the AWS accounts so what it really does is the event rule this will trigger daily and the automation document helps to share this it will calculate I mean if we evaluate what are the list of the accounts in the whole control tower environment and it will update the permissions of SSM document so there's a python script I have included in the SSM automation script and then so that ensures that you know so it makes sure that it share to all the accounts then once it shared it so it can be seen from the members account can see this package so then the SSM association has a command document in it so it will do the pre-installation like installation and the post installation so there are some scripts there so then it will deploy the agent to the EC2 instances so I think yeah so this is the demo that I will I'm going to show you so let me play that it's around 5 to 6 minutes in this demo I'll be presenting how to deploy snow agent to multi-account AWS environment if you haven't heard about snow snow is an IT management platform used by the software asset management teams to better manage and optimize infrastructure spend and governance across on-prem as well as clouds like AWS this includes software agent which required to be deployed on each endpoint that means in this case all AWS instances in order to pass information from instances to integration server to deploy any agent through this method first of all you need to prepare your CIP files and the manifesto json file in order to create the system manager distributor package as you can see there are 3 files including MSI file, installation and uninstallation scripts these 3 files are required to be placed in a CIP file in order to package your software in system manager distributor likewise I have prepared CIP files for Linux as well if you are manually creating these files you have to include one line commands correctly in your installation scripts if you are using management console you have the option to auto-generate installation scripts and manifesto json file when you upload your MSI or rpm files since we are trying to deploy these through code you need to manually prepare these files once you have prepared these files these need to be placed in a s3 bucket so I have uploaded these to one of the s3 bucket in share services account as I told earlier after you have done this the rest of the resources will be deployed through customization for controlled over pipeline using cloud formation templates let's have a look in snow package distribution in snow I have defined ssm distributor package definition which is also a ssm document but its document type is package also I have mentioned the source URL and the path to manifesto json file then let's move on to the next part of the code this package sharing solution comprised of three resources first one is ssm automation execution rule second one is ssm automation document and the last one is event bridge rule so these resources help to share the snow package across all AWS accounts also it ensures that the package is shared with any new AWS account whenever it's provisioned I have deployed all of these resources to share services account through customization for controlled over pipeline after that the state manager association will be deployed to all organizational units which covers all AWS accounts where you expect to have your instances in this template I have defined a system manager command of human with pre installation installation and post installation scripts for both windows and Linux instances then the state manager association which is specified below uses this command document also you can see that I have defined the target based on the instance tag value so if any instance has this tag the state manager will execute the ssm command document on those instances and install the snow agent I have already committed this code to customization for controlled over pipeline and deployed through stack sets if I go to dev account then go to system manager distributor if I click the share with me I can find the snow package share through shared services account then if I go to the state manager I can select snow agent installation association and check execution history I can see the last execution the resource status having success equals 1 which means it has successfully ran the commands in one instance and there is no failures if I had 100 instances I would see success equals 100 under the resource status if it failed it will show the failures as well if you want to get more details you can click execution ID then you can see all instances and its command outputs so if you click the output you can see the logs of each step which can be useful when you want troubleshoot it here I can see the package has been successfully installed in the instance finally let's go to the EC2 instance and see if the snow agent is running if you list all processes you can see its running on this instance that's the end of the demo so finally I would like to give the key takeaways of this session so this is one of the way that we can deploy agents to multiple AWS accounts but not necessarily the only way of doing it so I believe you guys agree with that so one of the ways other ways of doing it is it could be including the agent in golden AMI but in this method the advantage of this method is you are not coupling that to the AMI pipeline so for example if you want because you might have various kind of agents and it have a lot of updates let's say if you include all of those agents in your AMI pipeline which could be very complicated and people prefer to have the agents to be simple as possible so that's the reason if you want to deploy a custom agent so this is one of the important way but quickly I'll tell you there are another thing is the exception handling so for certain instances sometimes you have to avoid certain instances like virtual appliances or there could be many reasons so if you have that problems in your organization you probably use I mean you should have a proper tracking strategy so then you can target on those instances and you can avoid other instances which you don't like to install the agent and the last point what I'm going to talk about is you should be able to install the your instances should support the SSM agent so there are some incompatibility issues and especially like red hat 5 the older versions sometimes you cannot install the SSM agent and also some company security policies also some companies they don't like to have the SSM agent in their instances so these are some things that you need to think of if you are using this architecture to deploy your agents so that's it so thank you for listening to my session and I don't think we have enough time isn't it so so feel free to reach out to me if you have any questions I'm happy to discuss if you have a different thought if you have a you know if you want to clarify anything feel free to reach out to me and I'll be here the whole day so until that so see you guys