 Good morning. Welcome to my presentation. 1.21 gigawatts. This is a talking talk on hacking and solar panel controllers I am professor Plum. I am a security research researcher for stage two Usually we do pen tests and IoT audits this one though. This one wasn't a customer request This is just kind of something I did on my own on the side a little background What got me started on this path everyone in 2020 right found themselves maybe at home a little more Maybe with a little extra free time and about that same time or a little earlier My neighbor got this new solar panel system control installed on his house and as they was telling me about it There's some features or things it did that kind of piqued my interest And so I thought this would be a good project to do while I'm stuck at home So it's kind of started me down this path I'm gonna talk as I go through this presentation and kind of say What kind of things the vendor did to better improve their security over typical IoT stuff We'll see versus what kind of things they did to weaken their security just kind of a plus-minus win-loss Scenario and with that I'll just kind of jump right into it. So whenever we do IoT work or ever I research something it's always open-source research first, right? We want to see how far we can get with the data that's available online before you even purchase or acquire or touch the hardware And so in this case, I did the research on the product and how it's set up The particular solution that they had was the in-phase solution where you can see here from their their own slides in phases slides They've got a number of solar panel controllers up there at the top They're all connected to this in phase IQ Controller system, which is over on the left and that controller system is what? sends the output information the consumption the power production up to in phase cloud container and at the cloud then your iPad device or your browser can kind of see what kind of data your solar system is producing Or what kind of data what kind of production numbers you're getting etc What I also found interesting is that it's installed via a mobile app And so the mobile app wirelessly talks to the in-phase controller in some sort of way So that was something I wanted to look at and Then the cloud component also peaked my interest. So I want to kind of play with those to see what see what was there So like I said first I grabbed the mobile apps these there's two mobile apps There's the installer toolkit, which is what the installer uses when they're setting up your system And then there's the enlightened manager app, which is used by the end user to View power production and see how well they're doing based on weather You know, etc. How much power they generated over the last 24 hours days weeks, whatever The installer toolkit though what's kind of interesting to me is that you pressed a button on the in phase Or as the manual says so you press a button on the in phase and then the installer would just automatically connect to it And so there's no real kind of authentication there, which really was surprising to me And so I started looking into the app and what the app is it's actually a dot net application So for Android, it's like Java that's wrapping dot net, but then that dot net application will call into a native assembly DLL for these password functions and That always gets me excited when I see something like that Typically when I see in an app It called into a native function It's almost like it's trying to hide something and so it puts it in this native application I think many developers understand that dot net or Java code is very easy to decompile And so they think well, I'm gonna write this in assembly or C code And because that gets compiled on a machine code and that's harder to disassemble, but it's almost like For for me as a reverser when I see something compiled on the machine code It's like oh look here for the secrets because they obviously are trying to hide something and so they wrote a machine code So that's the first place I look when I see something calling into an assembly DLL I'll look into that and sure enough right it right here. There's a few exports to you that are Java You see C code that's ported for Java But you see get mobile password get password for serial number Get public password and all these functions kind of boil down to the same thing Which is it basically it'll take in a string. It'll do an md5 of that string plus this token string you see in phase energy there on the screenshot and Then it computes an md5 of that that hex digest is then ran through I wouldn't say it's a CRC algorithm It basically is just replacing ones and zeros and other characters that are that are confusing sometimes with More distinct characters, so it's basically just taking part of that in d5 Modifies it slightly and but that's the password. That's the installer password I'll talk more about this in the future, but this is This is kind of a big problem with these guys where they kind of do their own Password system and we'll talk about the little bit that the input to this is the serial number for the device So if you have the serial number then you can get the installer password for that device Another thing I found when I was looking at the app is that all the firmware packages are On an s3 bucket and so I use my command s3 command line tools Just kind of poke at the bucket and we can see that the buckets wide open You have to do an LS on the bucket and list all the firmware packages there along with those firmware packages There's XML files which explain which firmware products apply to which devices And there's even some installer scripts in there as well to kind of give me an idea of what the system's doing So that was awesome. I thought I had everything I needed However, if you'll notice there a lot of those files have the EE PKG extension This is an extension. I haven't seen before and when I tried to look for the files It turned out that that all these files were encrypted So although the data was up there. It was all encrypted One of the install scripts did reference into decryption tool called EE crypt You can see this online seven seven one there But this EE crypt tool wasn't to be found it. I'm assuming it's a file on the firmware So here I am now stuck with a catch 22 I have the firmware images, but they're encrypted and to be able to decrypt the firmware images I need a file off the firmware images this one thing kind of sent me back a little bit and I'll talk about that a little bit more later, but Suffice it to say I have firmware images, but I need to find a way to decrypt them before I can really start to do heavy analysis So with that I have an installer password. I have some way to go forward, but I needed the device That's about as far as I could go With the open source security. So I acquired one of these devices off of ebay and started poking at it Here's a screenshot of the device Let's see if I You can look at this with me real quick So there's this line that divides the board in half the bottom left side is the high power side That's where your 10 110 or 220 voltage is running through it And then there's these chips here which will convert that into it'll track how many amps are running through there and can and calculate power consumption and power production that has passed to this Custom in-phase chip here and then that communicates with the main system on a chip here. This is an arm processor Complete system on a chip. It has an external RAM here and this E emce Data storage here that which is the ext4 file system. There's also a Wi-Fi chip an Ethernet port USB ports on this device So there's a lot of a lot of functions and features available for attack on this And then if you'll notice here, there's these headers that broken out This is the UART header and here is the JTAG headers So many many attack opportunities available and then also on the backside of this device There's also a flash SPI chip and not shown here So what I'm gonna do in this talk, I want to talk about some of the standard attacks We would make on a typical eliminated Linux embedded system There's kind of the boot process that goes through and I'm gonna list the attacks that we can do at each of those stages But I'm gonna list them from reverse order. So I'm gonna start at a fully booted system and work backwards So it seems like I'm going backwards But that seems to make the most sense to me because we're gonna start with what most people are familiar with and then work Slowly backwards from there So starting at the bottom, right? That's when the system's fully boot up. You've got basically just a Linux kernel running So your typical attacks against a Linux OS work here as well where you're looking for what ports are open What services it's running? What attacks do I have against those? Like is there a talent running right isn't an old version SSH is there any HTTP vulnerabilities on anything that's hosting there those typical things are always available to attack at this run level It's basically just a running Linux system Hardware-wise, there's also your these attacks against the you are in the J tag They're not specific to this boot process, right? They J tag can really be attempted at any time them chip is running However, I'm gonna list it here just because right it's the first it's the first stop So J tag on this particular device was disabled I'm not sure if it was via fuse or if it was collines, but I was not able to get J tag or SWD working on this on chip So that was a good plus for them. The you are did output, but it would not accept any input from the command line Some it's always worth a shot right to connect to that And maybe if you press an almost right skill be presented with a command line I haven't I've seen that before but in this case the kernel was not accepting any input from you are it would Output to the you are so why the kernel would booting was booting. It would show some messages there By way, okay, so the next attack level is while it's booting I mentioned that you are showing me debug messages or kernel boot messages. We're just somewhat helpful But nothing in there was Easily broke the thing down One things you can do though is you can try to interrupt the boot, right? You can hit certain keys to try to stop the links kernel from booting Maybe if you can change the boot arguments You can have it boot into single kernel or single user mode or the change the init level so that doesn't run all its scripts But then again that requires system accepting you are input in this case It did not so I couldn't really affect the boot process there Another thing I could try which I did try was glitching So glitching is where you can cause an Data errors in between communications. So for example, this used an external MMC chip for its EXT for boot partition So it's full partition actually and so if you can affect the boot lines in between the two Then when the kernel tries to mount that EXT for partition You can you know tie one of those lines to ground so the communication fails and then you're hoping that the kernel is going to Say oh, I couldn't mount this partition and drop you to some kind of limited shell At which point you have a shell and you can stop glitching the line mount the system yourself and muck around with it That wasn't the case here The kernel did a panic when it can't mount the EXT for file system And that's kind of the right approach when you're developing embedded systems, right? You know exactly what hardware you're expecting and so if anything varies from that you should just panic You don't want to kind of fail open and that's what this guy these guys did So that was a good design decision by them It meant I had to look lower down in the boot process for another attack method And that's the next stage before that would be the second level boot loader This is very often you boot not always the case But in most products we see you boot is the tool used and what you boot is it's a very lightweight Boot level that knows how to read external EXT for file systems So what it can do is it can read the EXT for file system find the kernel image Copy it into memory and then jump and pass execution to that kernel That's basically what you the you boot a stager is whole purpose is to do is to find the EXT for file system read the kernel image off of it and then boot to it pass the execution there The you boot itself has some parameters that it affects that it passes to the kernel And it also by default has like this three second delay where you can get to this very limited you boot console And change some of these parameters These guys had disabled that three boot boot delay second delay so I couldn't just press a key and Get to that you boot stager. I wasn't able to get there The the you boot itself though if you have ways of mucking with it You can actually boot off of a TFTP image right like you can tell it Hey, there's an image out here kernel image use that image rather than the limits You're told to find so what if you have access to the console you get a lot of You get a lot of control and you could probably break the system in some way or another But I needed to find a way into that you boot image and I wasn't able to get there at the stage I'll actually come back to this and say the work around there And so the very earliest levels is the first stage boot loader This is typically some very small flash embedded in this system on a chip itself and this flash is too small to hold the you boot loader itself and But it's too small to be able to just read in the xt file system And load the kernel in an image it it's basically like a mini first stage Which is responsible for finding the you boot image and then jumping to it Because it can't read the xt file system and it's too small to have the you boot image It has to find the you boot image somewhere else And so in this case it was an SPI flash chip to the side On the backside of the board that flash chip so this little flash image can read that SPI flash It'll read the you boot image and environment variables from there and then jump to the you boot image This flash chip we could reflash it. However, this is kind of a dangerous or excuse me The first stage boot loader which is running on the system on a chip We could reflash that but if we broke that or if we mess that image up You've now break the device because that's your first stage. That's all you have to access So it's a high risk, but possibly high reward area That's not what I wanted to attack What I when I realized that this was reading the you boot image office separate SPI flash chip, then that's what I attacked Here's an image of me with my gator clips attaching to the SPI flash chip And I'm reading off that flash and on that flash contains the you boot image itself and the environment Variables for that you boot image one of those environment variables was the one that specifies No boot delay. So on the left side of the screen here You can see I've got this highlighted this boot delay equals zero that was in that firmware image And so all I need to do is just change that bit the zero to like a five or something And then I can have a chance to enter the the boot console and start mucking with the device However with SPI flash chips, you can't just change one device You it only supports writing pages at a time and before you write a page you have to erase that page So there's a little bit of a risk there, right? You're gonna erase the page and then reflash it and something goes wrong You've kind of broke it But that's that worked for me I was able to change that bit you erase the flash and then rewrite it and so I changed this fireman variable However, what I missed is that the environment variables for you boot are There's a CRC attached to the beginning of it that says hey if this has been broken or if it's been altered with Don't trust these environment variables And so when I changed that boot delay parameter, I didn't change the CRC that went with it And so it was marked invalid Luckily for me though you boot when it found that the environment variable was corrupt It went back to its failsafe mode and its failsafe mode had a three-second delay So I was still able to get in once I got on I was able to restore the original environment variables or just fix the CRC So now I had access to the U-boot command line and on the U-boot command line. There's a command called ext for load And which will mount which will read a file of an ext file system into memory And then I can view the contents of that memory So on the right side of the screen you see that I've read in the contents of Etsy shadow, right? Like now I can see the root users password hash if I crack that hash Then I can just s h into the box and that that was really exciting. That was my hope However, I was never able to crack this hash if you do crack that hash I'd love to hear what it is, but as you'll see in the future. It doesn't even really matter what the hash is So getting the Etsy shadow file was no help to me However, I mean mentioned earlier on the slides that I needed the ee crypt tool I was able to read the ee crypt tool out like this to this hex dump and just use like xxd to extract that ee crypt tool to a binary and start looking at that binary and As I looked at that binary I was looking for what it does to decrypt the firmware I never saw on the command line a key passed into that tool So I figured it had maybe the key hard-coded inside Turns out it didn't have the key hard-coded per se However, it generated the key in a hard-coded fashion So on the left side of the screen here are four different d-word values And what the tool does is it will generate a shaw on those values You know funky way So there's these blocks on the left side and it's just it starts to generate a shaw value by Hashing block one two three four and then block four three two one and block two three four one You know just hashes them in in multiple times and just a random order But then the output of that hash is the key now it was really kind of interesting Why did they decide to do this right like I think there was somewhere in their mindset. We know it's bad idea to Include a key hard-coded key in a binary But instead so instead they're gonna generate the key based on some hard-coded values The end result is the same I still have your key right like it didn't help anything It just obfuscated it slightly and so I don't know I don't know if they realized that this was really a no-op for that for a reverse engineer So having that key I was then able to decrypt for more images and start looking at the files on disk And so here I am on slide 19 the whole all the slides up to this was really me leading up to getting access to the firmware As soon as I had access to the firmware within the first hour of just looking at the files I found in my first I found my first RCE on this here is a Upgrade function that's exposed via the HTTP interface and it calls it makes a system call And you can see the args parameter takes URL parameters and just depends it without doing any kind of user sanitation So we can append commands to this and run things on the target and so With this I had now an execution on the target. However, it's through the spunky upgrade process and it's it's slightly blind So I didn't I wanted a cleaner way to get execution I wanted to be able to just like SSH into the box and so what I did is I changed the root user's password However, I still couldn't SSH into the box. So I checked the settings for SSH to make sure you root user could log in and root user could log in with the password But it was surprising to me that I couldn't log in even after I changed the password I knew what the password was And it turns out as I did some more dicky digging into them Module that they have their own custom PAM authentication module in line So for those who are not familiar or just a refresher, right PAM is what's responsible for doing authentication on Linux And so you could write a custom module to handle custom authentication this custom module that they wrote intercepts all Authentication attempts it looks at what type of attempt it is and then takes a different kind of path based on that But it doesn't pass the attempted on down the line So if it says it fails, then it doesn't try the typical Etsy shadow user password. It'll just fail So that means that I could change the Etsy servers Etsy shadow password and has no effect Because this module intercepts the request and that's why it didn't matter at all if I cracked that hash or had the root user Password because this is what I needed to match to to get authentication as I looked at this tool when it does for an authentication was very similar to the Mobile operators tool it will take your you it will take your serial number and your username It'll generate a string based on that so it's like username at and phase energy comm hashtag serial number It'll compute an empty five of that and that hex digest is the actual password So once I had this I then had SSH access into the device And that was it I was at SSH access. It was really quite interesting. However, you notice here It's not just SSH authentication of this affects it affects the HTTP login it attacks It affects other things and this is really a bad issue Because this means as soon as you know the serial number, you know the password for any one of these devices Furthermore, you root anybody who owns one of these devices can't change their password They're stuck with that hard-coded password that the module uses you can't remove that module It'll break a number of the services that the in-phase controller relies on so that that One decision of theirs led to three different CVEs in addition to the other that I found before I'm all having to deal with poor password management choices and it made me wonder Why why did they do something to cripple the security of this so poorly? And I can't speak for the developers. I don't know what they were thinking But I can make some assumptions and here's what I think the issue was as I looked at this I think they knew hard-coded passwords are bad ideas and they knew we can't have one password for all the devices That's a bad security issue So they devised a system where each device has a unique password But they didn't realize the other things they broke when they came up with this system And so it was kind of a oh we fixed this one problem, but made for others, right? It's um, I Think it's a we know that thou shalt not do X But they didn't they were just following the letter of the laws and not really understanding why that's there And so their workarounds actually weakened the system in a lot of different ways So Really to hack the device is all I need is the serial number And that serial number is available on the board itself It's both available as a number right here or as a small qr code right there Or you can also get the serial number on the back of the box itself. It's on the bottom of the box itself It's on the inside cover of the service panel It's also on the shipping on the outside of the shipping box that they shipped to you And then if those aren't enough there's an XML page that's publicly available where you can get the serial number It's on the home page of the device and then finally if you can't find it there It's also put in the title bar of the home page all of this unauthenticated So the serial number is everywhere and with that serial number you can log into these devices. So What does that mean right like what can we do with this? Well Probably could be turned into a large botnet or something or or you know a DDoS tool Let's typically what we see IOT for but I was trying to see what else could be done with this one thing I found interesting is there's an option for the device to make a Open VPN tunnel back to in face and that's so that they can then you know SSH into your box What's shady about that is that that means? in face has a SSH open VPN tunnel into your home network or Business network, whatever the case may be which some people may be a little Uncomfortable with I what I didn't check however if that tunnel means that you have a open VPN tunnel back into in faces network I don't know if that's isolated or not. It would be something worth looking into but I didn't want to get into any legal troubles So I didn't poke at that myself And moreover. I found something slightly more interesting that that piqued my attention As I was reading about this I learned about the solar energy Credits basically as you're producing solar energy you in some states you can earn these solar credits a lot of companies Need to meet certain regulation about how much clean energy they're using and so these solar credits are something that's tradable on the private market so if you're able to alter the device you can potentially Claim to be making more solar credits and you're actually generating and then trade those on the black market for money Right, you're basically laundering your own money or printing your own money if you have access to one of these devices so I played with that a little bit and Here is where I decided to muck with how much energy I'm producing you can see I wasn't producing very much Energy until the 25th when I suddenly started producing 900 kilowatts of electricity and so I was like, oh, this is awesome And this is their online system here. This is their cloud monitoring base So my my device have reported this information up to their cloud their cloud accepted it and says yep This is what he's been generating for the last few days every hour of the day even at night The system didn't seem to complain at all But I thought okay, okay. Well, if I can go that far why stop there let's produce 101.21 gigawatts, you know like a bolt of lightning constantly coming through my household That should generate a lot of solar credits Now to be fair. I didn't ever assign I signed up for the solar credits I produced this energy or the cloud says I produced this energy, but I didn't sign up for the solar credits I didn't want to get any kind of legal trouble. Hopefully there's some kind of process in place Which would kind of vet how much credits I it says I'm awarded and I mean no home solar system Should be able to produce 1.21 gigawatts of power But who knows maybe there's something there maybe not We reported these issues to the vendor a while ago. They said they have a fix for them I have not seen that yet, but Yeah, it's definitely something that is concerning and I definitely want to I would look into Just as an apply side is it going to white away slide? By the looks of this and what we're seeing in other products embedded hardware security is improving a lot of the easy wins Aren't there right like the telnet's gone the you are at port being opens gone j tags being locked down a lot of The things that we've been harping on for the longest time are improving However, the industry still has a long way to go to make fully insecure systems a lot of the time Hardware attacks are becoming more difficult as a lot of things are being disabled However, they're still glitching. They're still the tax where you can pull the firmware off of SPI chips like I did that can Can still ruin your your setup This one I just have to say it just because I see it so much Never never roll your own crypto if you think you have a problem that's unique Like just just just think for a second. Am I really the first person to ever have this issue? Am I really the first person to design an IOT device where I don't want the same password for all the devices? because You're not and so look at the industry best standards and see what they're using rather than coming up with your own solution coming up with your own solution rarely rarely works and Then just that last point I want to point out that encrypting protecting your firmware images really goes a long way to prevent attackers From looking at things not necessarily doesn't make your system secure But it does prevent the casual observers or those researchers from doing a quick gander and pulling some easy wins So protecting your firmware does go a lot to keep things secret with that I appreciate You guys taking your time to listen if you have questions or ideas you can reach out to me That's my email address. There's my Twitter handle professor plumb There's two underscores there because I was really late to the Twitter scene But I'd love to hear comments or questions or thoughts or things you want or suggest I want to look at in the future and I have some free time. I love the kind of doing this kind of stuff on my side as well So I hope you guys have a great death con and I will see you around. Thanks