 In this module, we're going to give an overview to the area of complex data analytics, touching upon many of the major themes that we'll expand upon during the rest of the course. Computers do analytical processing of data. In the past, this was largely about individual machines acting on individual, well-defined data structures. We called this data analytics, which is the use of computers to analyze data and find meaningful patterns within it that can be used by organizations to make decisions. Today, computing is evolving to cloud platforms, advanced algorithms and big data, and we can call this advanced analytics or complex analytics. Complex data analytics is the use of advanced algorithms to process big data structures. With the convergence of cloud computing platforms, advances in algorithms, the growth of unlabeled big data sources and now the Internet of Things, the ongoing revolution in information is entering a new stage, with the capacities of information technology being greatly expanded. The creation of personal computing, the Internet and mobile devices has created a flood of new data sources. The response computing is moving up from individual machines with well-defined instructions acting on well-defined individual data sets to now running on clusters of machines on massive amounts of unstructured data and using qualitatively different algorithms in the form of machine learning. In this process, we're collecting ever more data about ever more aspects of our worlds. We're bringing that into data centers and applying ever more sophisticated mathematical models and computer science methods to building algorithms that allow us to look into this big data to see what we've never seen before. A world that was previously only accessible through our imagination is being presented to us as real data and visualizations. Our data production is currently on an exponential growth curve with no end in sight. The global information network is now growing at some 205,000 new gigabytes per second. A constant barrage of web searches, e-mail, e-commerce transactions, chats, blog posts, social media feeds, data streams from production lines, cars, closed-circuit TV from financial markets, transport systems, mining equipment and buildings, all creating a continuous stream of structured and unstructured data. More information crosses the internet every second now than was in the entire internet just 15 years ago. As we begin to instrument our worlds, with sensors and mobile computing, our every action becomes data. It's sent to the cloud where huge modular algorithmic frameworks process and cross-correlate it with the data from everyone else. Everything starts to become data. Your movement purchases traffic and the data gets moved to the cloud where it gets processed and compared with data from other devices. It's no longer just what you do, but what everyone else is also doing. And in this data, we begin to be able to see and connect our individual actions with those of others and the whole of the systems that we form part of. When hundreds of millions of people and devices start to contribute data, we can start to see patterns emerge from across society or across the whole world. But the challenge is that 80% of all this data that is created is dark, unstructured data. This is data that the computers we've developed in the past 40 years are not able to analyse effectively. We're missing 80% of the knowledge inside of this data. Unlocking this unstructured, complex set of data sources requires new models and algorithms. And this is the other part of the puzzle that's clicked into place only just recently. Not only do we have a new computing infrastructure available to organisations on demand and a wealth of new data sources, but now we have a new paradigm to algorithms in the form of machine learning systems. The algorithms of the past were well-defined rules that were pre-specified and hard-coded into the software that were mechanistic in nature. Recent breakthroughs in machine learning, neural nets and deep learning techniques have opened up new possibilities for processing large and unstructured datasets efficiently. Today, a deep learning algorithm can easily deal with tens of millions of parameters and billions of connections, meaning they can do things that were previously unimaginable. Drive cars, detect security anomalies, analyse job applications, process insurance claims, coordinate traffic and the list is ever expanding. These machine learning algorithms often take the form of self-organising computational networks as exemplified by the hugely successful approach of deep learning. This approach enables computers to act on large, unstructured datasets and to drive insight from them. As a consequence, algorithms are no longer confined to the internal workings of your computer but can now expand out into the world, acting on ever larger, more complex data structures. An algorithmic revolution is underway, as we shift more and more of our systems of organisation to cloud platforms. At the heart of these platforms will be advanced analytics, which is used to coordinate and optimise the network, whether we're talking about car sharing platforms, e-commerce or logistics platforms. With the current rise of cloud platforms, we're in the process of converting centralised closed organisations into large, open networks. These networks will be based around market dynamics but to coordinate and optimise such complex systems where we'll require the use of advanced analytics. Just as the vast user networks of Facebook, Uber, Alibaba and Amazon are coordinated via advanced analytics, the same will be true for almost all organisations in the future. Mastering this new paradigm means understanding not just data science and machine learning but also how they operate in the context of this emerging platform economy. The last major component is the integration of complex analytics with the emerging Internet of Things. Machine learning will be delivered as a service over the Internet and the smartness that it delivers will flow to all kinds of things as physical systems of all kinds start to exhibit new forms of adaptive, responsive, autonomous and smart behaviour. The Internet is starting to come offline into the physical worlds and machine learning is a central element to this as it enables the ingestion of large amounts of unstructured data. It enables machines to interpret and understand the physical environment, human behaviour and likewise interact with people in a fluid fashion. Not only do these advances in algorithms enable mass automation and the proliferation of autonomous robots into the everyday worlds but more significantly advanced analytics is increasingly being connected into whole physical infrastructure systems. The smart grid will throw off massive amounts of big data and be coordinated via complex analytical systems performing dynamic load balancing, dynamic pricing, performance reporting, predictive maintenance etc. The same will be true for transport systems whole metro systems like that of Dubai are now automated the same for mines and for fleets of ships for example Royals Royce is partnering with Google's cloud machine learning engine as they research and develop the next generation fleet of autonomous ships. The rise of big data and advanced analytics represents a profound change in both how we understand the worlds, make decisions and act on those decisions the depth, scope and significance of which is difficult to overstate. In a recent paper by Ericsson the authors capture some of the significance of the process that's underway when they note in contrast to digitalization which enabled productivity improvements and efficiency gains on already existing processes datafication promises to completely redefine nearly every aspect of our existence as humans on this planet. Significantly beyond digitalization this trend challenges the very foundations of our established methods of measurements and provides the opportunity to recreate societal frameworks many of which have dictated human existence for over 250 years. Advanced data analytics can be interpreted as simply how we manage our worlds in an age of complex systems. The technology holds out the possibility of actually seeing and understanding the complex systems that now run our worlds from transport networks to social networks to cities and global supply chains we actually have the possibility to manage these systems in a new way. Shane Gordy founder of Quid states it clearly when he says we live in a very complex worlds, there are 7 billion minds now and these 7 billion minds have created a world that no one of them can understand and yet we still have to make decisions. We have to decide whether or not to send troops to Iraq and we have to decide what to do about climate change and we have to decide how to deal with the global financial market or some want to stay still. Complex analytics enables a new form of data driven understanding to our worlds in that it enables us to visualize and in some ways see these systems that have become so complex that no one person can comprehend them. Big data and new visualization methods can abstract away from the underlying complexity to present a quick, high level overview to an otherwise impenetrably complicated data set. Financial markets that today are hidden behind layers of a pack complicated obscurity could be seen and grasped by every trader in the market. Billions of data points from around the planet could be ingested, cross-correlated and visualized to deliver a real-time vision of global security threats to everyone on the planet via their smartphone in a way that any one of them could understand in a few seconds. The threats of climate change, the risks of cyber security, the real social and environmental impacts of your current purchase all could be made transparent to any one of us enabling us to take responsibility for our actions and incentivizing us to make the right decisions. As MIT professor Alex Pentland puts it this is the first time in human history that we have the ability to see enough about ourselves that we can hope to actually build social systems that work qualitatively better than the systems we've always had. This is not just true for societies, it's also true for all of the complex systems that now make up our engineered environments. Likewise, datafication changes the actual nature of how decisions are made within society. Organizational decision-making processes have undergone a tremendous shift in the past 20 years. Enterprises are changing the center of gravity in their decision-making units from human expertise to big data-driven systems. This shift can be attributed to people's limited information processing capabilities in relation to the explosion of data. Take for example, the shipping company MERSC Lines which operates a global network with a total sea-borne freight of over 2 million containers that travel to 350 different ports as they work to move about 15% of the world's sea freight. The company estimates that they're spending more than $1 billion a year moving empty containers back and forth. No human could begin to reason about how to effectively coordinate such a system but MERSC is using data and analytics to automate and optimize where every container goes next and thus strip out the waste of resources in the network. This is, though, a relatively simple example compared to the data challenges that our organizations and societies face going forward. As the sea of data gets larger, the haystack gets larger and it becomes more difficult to find the needle while those who fail to evolve get lost in the noise and paralyzed by the complexity. The winners of this game are those who can use this technology to see through the complexity, to find the signal in the noise with which to move fast and strategically in so doing radically outperforming their peers. In an information economy, it's not the big fish that eat the small fish but the smart fish that are able to see what's coming and adapt the fastest that survive. For large organizations to be those smart adaptive fish is a huge challenge but mastering complex analytics is at the heart of that. The key feature of successful organizations in the age of datification is their ability to capture and effectively analyze the wealth of data available to them and quickly convert it into actionable insight so that they can effectively adapt and respond. Not only does complex analytics offer new ways of knowing our worlds through data and visualization and new ways of making decisions through advanced algorithms but mass automation, likewise, offers new ways to execute on those decisions. For better or for worse, mass automation of physical systems and basic services is now here. Around the planet, from Germany to Japan, physical systems are being automated and connected up to cloud platforms. With the rise of cyber-physical technologies and automated systems the nature of how we manage and control our environment is also changing. As Steve Law, the technology writer for the New York Times puts it Indeed, the long view of the technology is that it will become a layer of data-driven artificial intelligence that resides on top of both the digital and the physical realms and today we're seeing the early steps towards that vision. Although Steve Law's statement has a touch of science fiction to it the surprising thing is that science fiction appears to be becoming a new reality in our worlds. The more data we get and can effectively use the larger the problems we can solve but these powerful tools also have a dark side to them. Such powerful technology also has profound philosophical and ethical considerations coupled with it a dark side of extreme concentration of power of control and manipulation on an unprecedented scale of the civilization spinning out of control the stakes have never been so high but these issues need to be addressed in the context of the underlying technological changes that are happening and we'll address each of them at the appropriate stage as we go through the course.