 As we discussed in the past module, analytics is all about using data to answer questions. But how we approach doing this is very much dependent upon the complexity of the system we're dealing with and the kind of questions we want to ask. Analytics has been used in organizations in a formal way for over a century now being pioneered by the work of Frederick Taylor in his analysis of work processes and Henry Ford's measuring of his newly established assembly line. This is a relatively basic form of analytics where we're studying a very limited number of components and processes in a somewhat linear fashion. This linear approach has largely dominated business intelligence until quite recently but as we'll discuss in this module, things have just got a lot more complex and our approach to analysis is likewise changing to become a lot more sophisticated. This is now what we call advanced analytics or complex analytics. Whereas with the more basic approach to analytics, we're asking relatively straightforward questions about a relatively simple system that is limited in scope. With complex analytics, we're trying to answer complex questions or another way of saying it is that we're doing analytics for complex systems. To give this context and relevance, we can note how our world has just got greatly more complex along almost every dimension over the past decades. Whether we're talking about telecommunications, transport, international politics, supply chain management or the media landscape, the number of nodes and the degree of connectivity has greatly increased. In many cases, such as in financial markets, it is an order of magnitude greater than it was prior to the 90s. This new level of complexity that has emerged within virtually all of our systems of organization has major implications for how we should approach analytics. Living in a small town, you can simply walk around and talk to people to find out what was going on. As we form larger organizations during the industrial age, new tools of communications emerged. The telegraph, telephone and postal system enable people to communicate effectively within these large organizations. Television and the newspaper inform people of what was going on in their region, city or nation. But today, we find ourselves embedded within vast complex systems, networks that often span around the planet. Corporate supply chains, huge social networks, sprawling metropolitan areas, the global air transport system, the global biosphere, financial markets, these are things that shape all of our lives. But somehow, we don't have the means, the vocabulary or the methods that actually grasp them in their complexity to see them or to really know what is going on in the system. As a consequence of not being able to see the workings of these systems and thus in some way manage them, we get financial crisis, we get environmental crisis, we get violent social outbursts because we can't see the mounting tensions. Large corporations drop off the S&P 500 faster and faster because they can't see through the complexity and respond fast enough. In these increasingly complex environments that organizations operate in today, traditional conceptions of linear cause and effect breakdown, things become ambiguous and simply left without interpretation. We find ourselves in a reactionary state, continuously surprised and shocked, volatility increases, we start to see only some of the trees and no longer the whole forest. As a consequence, our organizations become incapable of acting decisively, knowing what to do and making important long-term decisions and investments. As this complexity proliferates, we are burying the proverbial needle in the haystack. More data means more noise. We stop being able to hear the things that we need to be able to hear. This is the same problem for businesses, for individuals, for researchers, for policymakers, for anyone living in this world of globalization and information technology. So how do we make sense of a complex world? We need new models and new ways of looking at the world, but just as importantly, we need new tools and methods for amassing and processing data and information. And this is what this course is all about, these new data sources and tools to support a more comprehensive understanding of the complex systems that we are now challenged to try and understand and manage. Whereas the traditional business analytics of the past has been based upon well-defined and well-structured data sets that were limited in size and complexity. Today, we have a wealth of new data coming from a myriad of new sources and we call this big data. Whereas the data of the past was structured into specific vertical categories being used to answer specific questions. This stream of unstructured data from a multiplicity of sources enables us to create context by making connections between a diversity of data. We no longer just have data about the products we sold and look to see how changes in the price changed the sales. But we now have a massive amount of data from different sources that can be used to find much more complex patterns and correlations. Data about customers, about location, about all the other products available. Specificities about the store where an item was sold or the time of day, the weather, etc. and all of this can be put together in new ways to find new patterns that were previously hidden. Whereas in the past, if we wanted to know where a fire might break out in a city or an accident on a highway happened, we were inclined to look at the phenomena itself. But with complex analytics, insight may come from a completely different realm that has nothing to do with that actual activity. With analytics, we look simply at the event. With complex analytics, we can now look at a network of data points to create some kind of context to the phenomenon so that we're no longer dependent upon simplified mechanistic cause and effect descriptions but can begin to look at things in a more realistic fashion as a network of interacting factors. John Kelly of IBM talks about this change as such. What is different now and has changed is that it's no longer about taking this data, putting it into a computer, running a calculation and getting a balance sheet answer. What's important now is what is the context of the data? What is it connected to? What effect is it having on data around it? It's basically a network of data. It's no longer sort of tabular columns, a rows of data, it's interconnected systems. Just as data is no longer a single thing, so to the means with which we process it is changing, in the past we used mechanistic rules like formulas in a spreadsheet that were dependent upon strict well-defined data sets as input. Today we're moving from rule-based mechanistic algorithms where all components are pre-specified and well-defined to computational graphs, which are networks of nodes that learn through self-organization. Complex analytics gives computers the capacity to understand data in new ways like humans do. This means that computation can now act on data derived directly from unstructured contexts and real-world environments. One of the best examples of complex data analytics is WebSearch. WebSearch was one of the first widely used applications of big data and advanced analytics. A search engine like that of Google or Baudu looks through massive amounts of data and within seconds analyzes it in a multiplicity of different ways. WebSearch has given us the capacity to look into big data, to look into all of this information we have and we can think about the effects of that on transport, on research, on almost every area that now depends upon this big data of the internet and complex analytics in the form of search engines. In this respect, people often equate our newly found technological capacities of complex data analytics to that of the microscope or telescope. Jay Walker of Ted Med describes this revolution well when he says the microscope in the 1650s and 60s opened up the invisible worlds and we for the first time were seeing cells and bacteria and creatures that we couldn't imagine were there. It then happened again when we revealed the atomic worlds but now there's actually a super visible world coming into play. Ironically, big data is a microscope. We're now collecting exabytes and petabytes of data and we're looking through this microscope using incredibly powerful algorithms to see what we could never see. With these new technologies that we'll be talking about in this course it's like we're building a new kind of instrument like we're building the telescope for looking into complex networks. For the first time, we're able to look at these complex systems that are all around us which have a structure, a pattern and even a beauty that are invisible without the right instruments. The implications of this are huge in terms of how we understand the world and our place within it and of course how we make decisions and act on them.