 Hello, I'm Didier Stavens, a senior handler with the Sense Internet Storm Center, and in this video I want to show you how we can parse and analyze JSON data to track Bitcoin transactions, for example. So this week, Johannes received a new type of extortion email of a sexual nature, and here is his diary, he wrote about it. This is the email that he received, and of course there was a ransom to be paid with Bitcoin, and here you have a Bitcoin address. So at the Storm Center, we wanted to monitor the different Bitcoin addresses that we had obtained, and I'm going to show you a method here, how to do this. So let's go back to my diary. So Rick figured out that Blockchain has an API and that with this URL here, you can get the status of different Bitcoin addresses. So let me take this, and let's go to Terminal, we have typed the command Curl to retrieve the data. So this is the URL, and let's go back to Johannes' diary entry, because there are in the comments some Bitcoin addresses, and we can retrieve data for those addresses. Let me copy this, like this, enter, and now I have JSON data here with the information for the address. So you can see here addresses, and then the address here, and final balance zero. So for this address, there have been no changes. And what we want, we want to be able to monitor several addresses and be warned, be informed when transactions occur for those addresses. So the API that we use here can query more than one address. So let's try for a second one. So let's take this address. So what we have to do is just make a sequence of the different addresses that we want to query. Now, the separator is the Pi character, but of course here in the shell, this has a special meaning. So we have to encode this with its hexadecimal value, and that is 7C, and then I can paste here the second Bitcoin address. Okay, and here you can see we already get much more data. So there is probably transactions for that second address. But of course, that's not very readable. What we want to do here is extract out of that JSON data just the address and the final balance for each address. And that's what I'm going to show you here. So I'm going to use a tool, JQ, to analyze and parse JSON data. So let me parse, sorry, let me pipe the JSON data that we retrieve directly into JQ like this. And then you already get a pretty print of the JSON data. It's already much more readable. So here you can see we start with a dictionary. Inside that dictionary, there is an entry called addresses with the two addresses that we requested and the final balance for those two addresses. So that's essentially information that we want to extract. And that's what we are going to do with JQ. So the Q in JQ stands for query. You can query the JSON data. And what we are going to do here, we have to escape this. We are going to select the entry in the dictionary with the name addresses like this, you see. And now we have an array, those quotes here, square brackets, that's an array. And inside the array, we have two dictionaries with the data. So we want to operate on the fields inside those dictionaries. So what we have to do now is to iterate over the array to have the individual elements of the array, the dictionaries here. And that can be done with the following operator. So we pipe, and this is a pipe in the JSON query language. It's not a pipe in our shell here, but it's a pipe for the JSON query. And we want to iterate over the different elements in the array, and you do this like this. With this command here, now you can see the square brackets are gone. So this is no longer an array, this output here is two dictionaries. And now, just like we did with addresses, we can here now select address. So like this. And here now you see you have the two addresses. The other information we want is the final balance like this here. And now you can see also we have the final address, final balance for each address. But what we want actually is that this to be on one line. So what we are going to do is turn this into an array. So those two fields that we retrieve, we will put them inside an array like this, the JSON array. And now you see we have the two values inside an array. And once we have this, we can convert this to a CSV file with the following operator at CSV. Because that takes an array, arrays as input, and converts them into CSV lines, like this here. You can see the two. Now you can see here that the quotes are escaped, because this is outputted, like JSON data would be outputted. But we can ask JQ not to do this with option R, Rau, so that we get Rau data like this. And then we have our address and the amount. Now the amount here, that's not the amount of bitcoins here in the JSON data, the amounts are expressed in Satoshi. And that is 100 million of a bitcoin. So if we want to see it in bitcoins, we have to divide this by 100 million, like this. So for this address, there is roughly 0.29 bitcoins that have been deposited in this address. Now you can see also we have here the progress indicator from Curl. We can also get rid of that by directing Curl to be silent with option S, like this. So now with this simple one liner, we extract just the information that we want, the address and the bitcoin final balance for that address. Now of course, if we want to monitor this, we want to know if there are changes and just executing that command again and again, we have to compare it with previous versions. And that's why I have a tool, it's called what is new. So what we are going to do is pipe all this output in my tool, what is new. And what is new requires at least one argument. And that's the name of the database, the database that what is new is going to use to tell us what changes have been present in our output. So I'm going to call this sextortion like this. And then you get this output, if you run it the first time, you get this here. And if we look here, a pickle file what is new, sextortion has been created. And if we run the command again, like this, then we get no output. That's because the output is exactly the same. So every line has already been seen by what is new when it's stored in the database. So there is no new output. It's only when there is a change, a new line, a different line that what is new will print it. And I'm going to simulate a change by adding a new address. So let's go back to the diary inventory because here we have a third address like this. So and I'm going to add this to our query. So %7C and the address like this. And then you get one output that's for the new address. And you can also see here that we have almost 0.44 bitcoins in that address. And of course, if we run this again now, then we have no output because there have been no changes. So this is how you can, with some simple tools that you can run on different operating systems, monitor for JSON data like a Bitcoin balance of addresses.