 I was at the speaker dinner the other night and one of the gentlemen at the dinner that I met or the other speaker was talking to me about so what is your talk and I gave him the title and he goes yeah what is a shiny object right so I forget the context the terminology is a little different here in Europe so raise your hand if you understand what shiny objects are okay good so I don't have to explain too much but in the context of this talk in the US you know we have this term I don't know if it's probably similar here silver bullet right solutions I don't use that because of obvious reasons in the US there's a little sensitivity to things these days so yeah shiny objects are just basically solutions to problems that we all want to believe in right to help us when we're desperate for the most part so today's talk is gonna be centered around that so before I get into that I'm gonna share a little bit of information about myself my name is Angel Rivera I'm a developer advocate for Circle CI and as a developer advocate I love coming to these conferences and speaking to people like yourselves I think I spoke to a lot of you at this conference and learned learning about how you use technology and bringing that back to my team so that we can build better tools right but this also enables me to you know share information about how we're using technology at Circle and vice versa if anyone wants to reach out to me please feel free to hit me up on Twitter these days I travel a lot so it's the best way to kind of get my attention I really not into email so please just feel free to hit me up I do respond yeah so let me talk about a little about my experience as a developer when I started out in my career I was working in the United States Air Force US Air Force Space Command which is now the Space Force right they merged that I don't know if you saw the the new uniforms we used to wear these blue like basically flight suits like the astronauts when I used to go to work now it's camouflage I'm like what like isn't it space dark should act right anyway I hope they changed it to a black like cool flight suit but when I started my career you know I was young I learned to program professionally in the federal government and this was in the 90s so there was no like internet the way we know it today when I developed code it was siloed on my own on a Windows machine usually and then when I was done you know compiling my release it would go on a floppy disk or a zip drive remember those anybody remember zip drive yes see anyway so it was the 90s or we would burn it to a CD right then walk down the hall to the data center so this was on Cape Canaveral so we're launching rockets and I'm working at NASA and I'm going in to this data center and this security everywhere knock on the door and it's the system administrators right all in one room and the air conditioned data center and they're like all right who the hell you and I'm like I have some software to the release it's like do tomorrow give me that and they close the door right shut the door my face but I've always been a curious developer right so I consider myself a developer with operational tendencies and the reason is when I was younger I was like I'm writing this code I'm spending a lot of time and then all I hear back from the the operators were a your shit doesn't work right it's broken and here's a log file on the on the CD or the floppy right and yeah that really pissed me off because I wanted to know how the next steps were for my software it's like you're sending it into a void so when I started going into the op-center I was just persistent kept going kept going and eventually they realized like oh this guy just cares right he's a developer that actually cares so I guess it's kind of I was doing DevOps before the term DevOps became you know normalized today and what I did learn though was as I was talking to the the system administrators they started realizing that I cared and then they started giving me root access or pseudo access to servers right so that could help them deploy and we actually started working together as development teams right and operations teams and I learned so much and throughout my career ever since then I was literally working every job I went as a developer I consider myself a developer to this day but I would sit in a you know development team writing software and then I would go and deploy it and help the system administrators deploy this software after a while they were like hey why don't you just work with us as a developer so you know I lived both sides of the house and it was it was really enlightening all most of my career was like that and today I want to share a story about my experience working on a really really good team of developers very experienced and a really really good team of experienced operators and tell you how we experience a shiny object moment together so yeah like I said I was working with a team and we had really strong developers really strong ops people DBAs right really good team cohesive so a lot of folks were a little mature they understood how to communicate with each other it was a really strong team and we were asked to build a new UI for some features that the customers wanted and we were excited to build it I was working on the front-end team building out these new features and back then you know we we were using SQL databases so everything was kind of a flat structure and at that point we had to build a hierarchical model so obviously JSON right to fit these new frameworks that were coming out and people were using them and it just made our lives easier right the data was structured really nicely we wanted to modernize a bunch of things but there was a problem I don't know if anyone ever you know worked with the relational data models and then try to make them nested in hierarchy right I think I had a couple conversations with some folks yesterday about this and it was really hard like we had link table after link table after link table things were slowing down indexes really just whole trash right it was is horrible we were really frustrated and trying to come up with a solution for this I mean we were reading books by Kelco I don't know if anybody read that book about SQL and basically mapping you know hierarchy structures it was a mess so I was on my own working with an open source project called MongoDB so I said wow this is amazing anybody here ever hear of Mongo anybody use Mongo you love it awesome I saw that little head shake but anyway yeah I was like this is great right the data is already structured we just submit it as it is it persists as it lies and bring it back and forth between the the system the service it was great the problem was it was still kind of new the technology right and a lot of folks didn't know about it how how the how to use it right they didn't understand we didn't understand the complexities behind that but we didn't care we were so desperate that you know when I showed it to the team they were just falling over this like wow this is the solution so you know whatever silver bullet it's awesome and I'm like yeah but nobody knows it that's all right we'll figure it out yeah as we all do right so we agreed the ops side and then the development teams we were like okay this is the solution let's move forward management signed off on it and we began to work right we started building out the infrastructures the operational teams were building out clusters they were you know creating VR capabilities HA making sure everything is you know failing over all that stuff those are the easy parts for the for that team so again no MongoDB experience right and I was the guy driving this which I was just learning it myself but we were all excited about this new product and using it in our in our in our development process so we quickly built the minimal viable product pitch that's everybody showed it they signed off on it and then we you know built out the the legit release for for the application based on the MongoDB cluster and we did you know we made quick work of that we actually finished it about a month ahead of time roughly and you know everyone was happy management was like excited about it our customers were excited and we were super super happy about deploying this thing again customers were super happy you know and they started using it right away immediately and we saw huge you know peaks and spikes and then didn't they never they never went down it was just kept going up ascending so we're like I said we were super proud ourselves life was right we were just static until 90 days fast forward yeah bump bump bump right Mongo starts kicking over clusters or you know our Nagios is just blowing up alerts everywhere we're seeing high distrashing memories just peaked CPUs peaked it's just a disaster so services are failing we're getting customers are calling hey I'm timing out nothing's happening and like I said we were a good team we went and started troubleshoot mode right put our hats on start looking at logs start looking at all the details and we quickly figured out what was going on the volume was about 15 million records of MongoDB they call them documents and you would think yeah this should handle this no problems we have tons of beefy machines behind this right and we were just befuddled like what's going on and when we start doing a deeper dive into this of course my queries were shit right it's terrible I didn't understand what was going on till I looked at like you know online start looking at we called Mongo they were really generous and helping us but pretty much the queries are garbage indexing on on the wrong things and if you really boil it down what had happened was I don't know if you all know this but MongoDB is non-relational so that means all the normalization that we were used to that I was used to I was recreating that inside of MongoDB which is by the way never never ever ever try to do that hey don't do it it doesn't work I don't care if you think you got a hack for it it's not gonna work right and we also so yeah so we discovered what was going on and we quickly realized right to get to recover the system we decided to do some math and said all right well it was working up until 15 million records so let's take the least use or you know least access records move them over temporarily into a kind of cold storage so that the system will recover as soon as we did that right moved over however many records boom the system was back up and running we didn't have to reboot a server literally it just started working and then we got to work on a solution right for a permanent solution for this but the damage had already been done and our customers had experienced you know this outage and it was terrible right the organization that I worked for and our customers lost a lot of time and money and this was the thing that hurt me the most was like our team the lost all of our street cred right literally we were just in shatters right we were just it was inconsolable we were very disappointed in ourselves so everybody from me with the term hot wash or postmortem right we like said we were professionals we conducted a postmortem trying to analyze what had happened and then fix the solutions come up with the ways to not happen that make this happen again right so of course right bad programming queries these are all obvious but one of the other things we discovered was we were so confident that you know we were just neglected any kind of testing we were like yeah this works it worked in you know it's typical typical when your developer and you it works on your local machine and everything's cool but when you start adding volume to it and right it's you don't test anything we didn't do any load testing nothing so that was a big mistake and obviously right we didn't properly vet the technology we didn't understand Mongo's capabilities and we were totally in the dark on how to actually manage it right and configure it manage it and then also write the data schemas that we were doing that I was doing they were totally off so that's the story about my experience and my shiny object moment our shiny object moment with MongoDB and I'm gonna talk about a little bit of things and these might be obvious to you but I'm gonna share them anyway so let's talk about avoiding those situations number one don't believe the hype right so if you find that your team is so arrogant and complacent there's a problem right start addressing that I don't care you know it may be just one person that refills that way but it's okay to voice your opinion and start getting your teams to like understand hey I think we're getting over our skis here you know we're a little bit complacent and just be on the lookout for you know that kind of complacency because it's it's a problem and it just degenerates the team in the culture from there obviously thoroughly vet unit technologies right understand the capabilities like with me in my example I didn't understand the non relational no relational database schemas right I didn't understand that you couldn't do link tables or you shouldn't do link tables within this this platform because it's not built for that now today I think MongoDB has has actually added like atomic transactions and there's also a relational components now that you could actually implement but back then right this was very early a early stage for the for the product but again you know thoroughly vet the technology understand just look past the the the marketing material like does that kind of got me too is like oh you don't this is the fastest database for any kind of web app oh yeah I like that don't believe it or at least you know do your due diligence I definitely recommend with all the technology today we have like Docker we're just set through Joel's talk today you have Docker containers right stand stuff up in those environments and then that also actually helps you to equalize right your development environment and also you can equalize your staging environments you can equalize your production environments to a degree right it's doctors won't let it become your shiny object but it can it can also it could it could it could work and help your teams kind of you know work through the software development problems I've actually you know at Circle CI we use Docker all the time and it's really great for our development teams and it you know it literally helps us deploy applications quicker so yeah start you know looking for technologies that everyone can understand and share and there's a low barrier of entry as well one of the things that we did was we broke protocol right SOPs everybody have SOPs raise your hand I know wow okay and the next slides for you guys and but basically with protocol right and SOPs it just helps you document what you're doing in your processes and your operations meaning if you bring a new person on board right to do some work then you want to give them these documentation like a standard operating procedure so that standardizes and it brings consistency to your your operations and also it lends itself to also being a very very let me say it dictates all of your processes now one thing I will say about SOPs is if you have them make sure you iterate over them because what worked yesterday will not work tomorrow right always I have a rule when I was managing teams I was like live we have to look at our stuff every 30 days if there's a change we're using a new technology let's update the documentation right and it also keeps management off your back a lot of times if you're working with big companies these days you need to have that documentation they want to see it and so if you're a startup and you think you're gonna go work for Google with Google or a bank or whatever big corporation this documentation just do the work up front get it done maintain it because you're gonna need to every year annually certify things remember that big pieces I can give you if you don't have SOPs please start building them like I said the benefits are the consistency right so you're literally documenting how you do business in a technical way you can even automate them these days right like I don't care if you just stick a document up in GitHub and have a version you know version controlled just build something I see this in a lot of like startups new teams that are within organization sometimes big companies will start like a research and development type team and these folks don't build any kind of documentation and next thing you know they're you know a year in and have to build up all this kind of stuff it just start off the bat doing it you know and that you'll get into a flow and then your culture will kind of evolve around that and that's one of the best pieces of advice I can give you to avoid like shiny objects in germs and don't set and forget one of the things that we did was when we implemented MongoDB was oh yeah the teams like said we were very complacent and we set this thing we never nobody was actually monitoring it we just had Nagya saying hey if you hit a threshold kickoff and alert which cost us so I'm gonna switch it up a little bit here so obviously right technology isn't about is about people it helps our lives right it brings convenience for us lets us do things consistently in a stable manner and when you have people involved that usually means there's a culture right forming and cultures are things that you know generate attitudes and and help people kind of form bonds and it also brings people together in a sense that you can work together for a certain goal now I'm gonna talk about everybody who this guy is I'm from New Jersey by the way so I had to make a reference for Jersey this is Tony Soprano from the Sopranos show yeah alright so I'm gonna talk about managers and some of the traits of management that you know if you are a manager maybe just refresh you a little bit and we'll talk about individuals as well so I'll talk about my style management a long time ago I coined this phrase pseudo management and the reason why I call it pseudo management is when I joined teams as a manager and when I used to manage people I would tell them look I am a peer I am a colleague right while we're doing the day-to-day work let's you know share the wealth and I would pitch in where I could now when I needed to do managerial things and you know I tell them hey I'm gonna pseudo up and handle the HR stuff handle conflict resolution whatever it was that was going on that day and then I'd exit gracefully and go back to work with them right what I found with this style of management it helps with folks giving them kind of knocking down as intimidation barriers right so it gives them a sense of ownership and I've been doing this about 15 years with this kind of style and I really enjoy it and a lot of folks you know I get a lot of I used to get a lot of work out of people and they felt really good about you know the work and they like I said they owned their their their piece of work they didn't feel like oh he's the manager or this person's that right it just broke down all those intimidation barriers and they saw that and it's the old military thing right it's a lead-by-example for the most part but with a little twist right so again as a manager always empower your people you want them to own their their decisions right a lot of times when I even to this day sometimes I see it when I'm working with partners some managers are telling their people you know like oh you got a check with Jim or this and that I didn't run my shop that way I was just like look if you're interested in doing this own it build it and then share it with the with the with the team and let's make decisions based on you know what you're showing us and a lot of times we would improve on those ideas and they were really good ideas and we had a culture of you know basically empowerment this is really important I work for a CICD company but you know is a reason why I work there and because I believe in this as developers right we're always failing and that's a good thing because you understand how your system right you start forming the way your system is going to work processes you understand your limitations you understand when things are failing alright that's great now I need to find a solution for that right and it's okay to fail that's how we learn that's how things get better mentoring is really important you should always foster a culture of mentorship you know empower your subordinates to or feel comfortable to come to you for for things that you know you're an expert in and if you're not an expert in something find someone who can mentor them it's a really important growing contribution right this is something I struggled with quite a bit and making those hard choices sometimes you know you really like your team you really like the personalities on your team but sometimes they I think we all outgrow right our jobs and our roles and when it comes to that managers you know you have that personal connection you're doing more damage by keeping people who are not really fitting in or not contributing and you don't have to fire them you can you know recommend them to another unit or team or whatever then sometimes you do have to fire them and and that's okay like I have been I had an experience where I had a person working for me and you know I was paying him a certain amount of money and I was really conflicted about firing this person but he was dragging the team down and I was that finally I had to I had to make that choice when I let him go he reached out to me about 30 days later he was a little angry right but I let him go as gracefully as I could and he says hey by the way thanks for firing me and I thought oh boy here we go like the first that was the the subject line thanks for firing me and then I read the message and it was like I'm making like you know I think he got like a 60% raise right so and he was much happier right so he was it was actually a thank you letter which you know again it'll work out if that's the the worst case scenario but definitely if you got to make decisions make them and you know don't look back so let's talk about individuals right like us all of us you're not a manager yeah don't have this attitude I hear this quite more often than I'd like like I said this is a way for you to level up so if you're an individual and you know you feel like oh well that's not my job I that I think that's a terrible attitude to have it's it's self-defeating number one number two it doesn't bode well with your colleagues either right like and then that also creates these toxic cultures that I've worked in in many companies and organizations so yeah you know try to level up and understand that you are contributor on a team and and that affects the culture when you're you know we all get frustrated I get it but sometimes when you're just consistently take a break maybe you need a vacation or talk to your colleagues be like hey look I don't know I can't handle this it's okay you'll be fine this is a really annoying one when someone points something out in a pool request with no solution to it right don't do that if you have a solution if you point out yeah this is broken have some you know even if it's not the right solution have some ideas it's okay put yourself out there no one's gonna be laughing at you right you're contributing and you're offering up and you're being you know you're showing a little bit about your your knowledge in things just don't leave it broken and sitting there for someone else to repair right or team up with someone sometimes that's what I do is I work with someone that I know hey he knows about Kubernetes or or you know Docker and we work together and we build out solutions for like some serious problems that we have anybody ever get one of these yeah I used to give them up like candy back in the day but yeah if you're a mentee or someone you know you're under someone's wing have respect for their time put the work in because I always say this job as a developer it's 90% learning in my opinion right and then the 10% is doing so you're constantly learning things are changing right PHP obviously you had 25 years now right it's changed a lot from when I started and you have a ton of different avenues that it went down but at the you know it just shows that technology is constant it's gonna change all the time so we're constantly learning and as a mentee right if you're being mentee mentored you know put in about I would say on average like 20 minutes if you don't figure something out 20 minutes then reach out to your mentors or someone else I used to get folks that didn't you know I'm like did you look at it did you even like no come on right so help yourself and and again it's another way to level up in your skills so yeah I have a hard time with this I still struggle with this so I'm always looking for ways to improve my listening skills meaning you know sometimes you're in a meeting things are you're excited because you have an idea maybe someone sparks an idea when they're speaking and then I just jump in and it's not a good it's not a good look you're not a good fit so I struggle with this my advice to you is all to just sit back let the people finish their thoughts and then you know add to your own no matter how wrong they are and when you disagree just you know be polite and say well I have a different perspective and and such I know as engineers sometimes we're a little short with each other let's not do that oh sorry and then the last I think this is one last tip here is learn how to communicate just like that example I shared with you when I read that subject line right like hey thanks for firing me I was reading way too much into I was thinking like oh either this guy's gonna piss me off or this is gonna be like I just thought negatively right right off the bat and literally when I opened it read it I almost started crying like wow this is amazing I fired a guy and he's thanking me for it because right he made more money so like I would say don't infer any kind of emotion from text that's one of the things that it's sometimes you know we get we're in a bad mood and we read some text an email from someone and then we start kind of saying this person's bad if you feel that way give them a call and say hey let's have a coffee and talk because I promise you most of the time it's gonna there's be no emotion to it there's the intention that you thought was there is not even there and I can tell you from experience I've done this so many times in my life I thought I'd share that with you so in summary tech problems are not tech problems right it's it's all about culture and we need to especially today in this day and age and this is kumbaya and I'm gonna say it anyway we should all be respecting each other let's try to like have an open heart open minds and especially with our peers and colleagues right it's really important we spend more time with each other than we do with our families in most cases right and you don't have to love each other but I think you have to have a mutual respect and I just want to ask all of you right before you leave today to let's drink some beer together but let's go move forward and leave here and build better cultures together right let's get our people happy and and enjoying the innovations that they bring let's let's like I said have an open heart open mind I think the world really needs it especially this day and age yeah this is all I have thank you if anybody wants to reach out Twitter yeah thank you