 السلام عليكم أهلاً وسهلاً ومرحباً بكم بعد الأكل والشرب سلا تكونوا بنفس القوة و بنفس العزيمة كما كنتها السباح الآن نوصل الحلقات للنقاش و الآن سنتحدث عن موضوع مهم جداً وهو سيبر سيكيريتي as the main challenge for the ICT industry in the next years so for that I have a pleasure to receive my friend in Tunis Mr. Mansour Shehry and he will be the moderator of this session Mr. Mansour the floor is for you Thank you very much Thank you Mr. Chair It's really a pleasure to be in the beautiful city of Hamamat and we really thank Tunisia for the great hospitality and I'm very honored to be with a great panelist with me today starting from Mr. Enigo Herrera the associate professor of economists from the University of Madrid and most of you who worked with the Acti know our chair for the Acti and the great work that he is doing for the last years and how he helped to improve indicators Mr. Enigo will present to us the summary of the Acti work for 2017 so the floor is yours and we have around 15 minutes Good afternoon to everyone I just wanted first of all to thank also the opportunity to be here presenting the results of the whole expert group on telecommunicators and I wanted as well as the chairman did to thank the hospitality of the Tunisian authorities really we are feeling perfectly comfortable here so thank you again for the hospitality and the organization now what I will be presenting is the work that has been doing during the last year both in the online forum in several subgroups that I will be most specific on them and in two Acti expert group meetings together with the EGH by the way that were assembled during the year so there is plenty of work that has been done and has been agreed also already throughout the year in the last two meetings as you will see the objectives grossly speaking are to improve indicators you will see some examples to redefine indicators and to propose new indicators and you know I think we achieved among all of us we achieved quite a bit on each of them I have to say two things on methodological issues during this last year there were three subgroups a group of people open to everyone that focused on one specific issue and later on came up with a proposal in the Acti meeting a proposal that was discussed and discussed and re-discussed and then later on converged and agreed upon these three subgroups have worked wonderfully and it's a very good instrument I believe for the development of more focused discussion and focused proposals that we need to come up later on with because sometimes and you all have the experience in any kind of group where we are many discussions go away maybe a bit ambiguous so this subgroup creation has been extremely helpful also in the EGH I believe to focus the discussion and to make up a proposal a formal proposal at the very end and also for the consensus building and also for us we try to make step forward always but always try on a consensual basis so I think it was more or less achieved what I will talk about is these nine more or less eight topics that I have put forward there are several indicators as I said some of them are improved indicators some of them are redefined indicators and some of them are brand new indicators and at the very end I will mention as well some words on the future work that is to be expected first of all I would like to say one the first indicator that we were dealing with more than a year ago in the previous acting meeting there was the proposal to talk about fix network coverage indicators I mean one of the deficits that we are facing is the fact that we are not really measuring fix or even mobile I will talk about this later on mobile coverage not coverage but deployment of the network indicators there was a big deficit in terms of fix networks and we know that the reality is that the reality is that fix networks are spread over very heterogeneous countries so we came up with the idea why don't we create a subgroup this was more than a year ago why don't we create a subgroup to deal with the issue see if there is any interest see how the people are doing it by the way in each country it is of interest to end up proposing to measure fix network coverage or deployment for that propose Joao Noronha who is present here from Anacom Portugal I want to thank him really warmly because he was the leader of a subgroup dealing with these fix network coverage indicators and what they did was a tremendous amount of work which was to gather all the different methodologies that many many countries are using in measuring fix network deployment methodological issues arise here a plenty we won't get into them Joao don't worry about this but there were a plenty methodological issues but we needed to converge and to get something done so first of all it is important it was this was the discussion that we held in the Acty and everybody agreed that it was important to measure fix network for policy proposals for investment proposals for many proposals right so we thought that ok let's do it and let's try to measure this fix network coverage you know that fix networks may have very different topologies they may be XDSL based they may be fiber based cable we didn't deal with wireless but fix networks these are excluded for now at least just not to mix and so the focus was on fix network coverage indicators and we agreed at the last Acty meeting to start collecting in the long question that you will all receiving like March April next year in the long question that will be asked what kind of coverage do your fix networks have and this is a challenge and we are aware of it for all of us because we all have different methodologies some of us they do not measure this but we need to make even approximations to really grasp this indicator this is an important one a lot of discussion went on with yeah this indicator will not be included in the IDIs so far of course a lot of discussion had to do with methodological issues you know that fix network indicators you can really grasp them from many points of view you can go for operators data you can do your own measurements you can rely on other type of measurements so there is a lot of scope for each country to organize the fix network coverage as he wishes but we need this data and soon for next month next March so it should be should be provided there are two big issues among others on collecting this fix network thing which is the fact that what is a fix deployment what is a fix connection deployed and it means to be available to the final user available not active necessarily and this is highly important for us to know then what does coverage mean what does available means what does deployed connection fix connection mean and this goes for any technology well it means that the network provider is going to be able the last mile connection the very last meters of the connection to the fix wire network within and the household of course within a very short period of time few days and without the need of any extraordinary commitment of resources hence availability means that the network connection fix is really available if not at your home very very close to it and this means deployed connection which is what we are aiming at measuring and the big problem which is has to do as you know with the overlapping in some cities in some densely populated areas as you know there are several two, three even more fix networks deployed already we don't want to count twice or three times the same household or the same unit so this overlapping problem it depends a lot on the methodology that you are following and it depends a lot as well but we tend to assume that unless you provide fine information on it the overlapping should be perfect hence you will pick up the coverage of the biggest deployed network operator in place now don't worry much because that will be a methodological document that will be provided to all of you based on the work done by Joan Naronia and the team that will be a methodological document to be distributed to everyone so that at least some issues will be clarified so this was one which is a big advance on it a second big set of discussion topics had to do with OTT IP convergence and IoT the internet of things as you know behind these very nice keywords there are hundreds of topics and problems and measurements possible we are and this is a recurring topic it comes up very often really discussion on this and we have a very important observation which is some OTT services they are clashing with traditional services and they are even replacing them absolute substitutability among some like voice of IP video calls of IP content of IP many so what do we have what are people really measuring in terms of OTT services and IoT so the problem that we faced and this was very clearly stated I think in the last equity meeting correct me if I'm wrong it was that the national regulatory authorities usually they lack the legal competence to really demand require data from this OTT operator to be provided to the regulator hence this is a reality and it was pretty much general not in all cases of course so we started discussing about this remember in the last equity meeting and at the end we have too many things behind OTT IP convergence and IoT we have too many things here behind when we split up the discussion a bit between OTT services on the one hand and IoT Internet of Things services on the other hand so we will have like two different discussion topics in the online forum this on the one hand and on the other hand and this goes in the perfect collaboration that we have between the ECT and the EGH we suggest that maybe from the demand which could take care of the discussion on how users use OTT services at least a few subset of them so this was more or less the conclusion that we reached at the last equity meeting five minutes thank you now then I will run another topic that thank you another topic that was widely discussed and here there is a big leap forward I think I believe and it has to do with wireless spectrum allocation talking about fixed network coverage well okay there is no measurement being provided at the world level on spectrum allocation availability and we discussed about this and we thought it was important to really develop some measurement based on this so since the topic is kind of complex and we need some collaboration among other branches of the ITU even so it was decided to create a subgroup that from now on and for the foreseeable months will be working on some proposal which is spectrum allocation or assigned spectrum that everyone has so I expect this is a big challenge as well to collect this kind of information it's not that easy so I expect everybody cooperates with this and there will be more guidance on this and it will be discussed of course before implementing it right most probably in the subgroup of course we need the collaboration of the ITU telecom development bureau and of course the radiocom bureau it didn't start yet to join the subgroup now another big leap forward was the revision of the LCT price data collection and methodologies the price data collection you know is highly important for all of us there was a tremendous amount of work being done year after year on price comparing prices among countries which you know is a highly complex issue so and there was the belief that we needed to update the baskets and the methodology itself so there was a subgroup from Sri Lanka from Lin Asia she couldn't attend the meeting but I also wanted to thank her because she produced a very nice paper together with her team on a proposal and the proposal goes like this basically means we were like flexibilizing some issues like for example now you will be picking up tariffs based on the most of contracting the service and I'm talking about mobile services so you can pick up prepaid is the majority of the contracts in place the benchmark of course will be based on the least expensive offer of what well of several baskets that we redefined we updated many many baskets but not only updated them as you can see here there is some novelty here which is we introduced the bundle which we thought it was very important to pick up the bundle of mobile services voice together with data broadband together with voice on prices so that we can compare across the countries we also redefined the expected minutes of use of voice of SMS and of data we defined a low and a high consumption basket and for the rest we kept only the data only mobile basket which is important for many reasons and we kept as well the only voice mobile basket right and besides we keep measuring right which we redefined a bit the levels of consumption this was a big we will monitor as well the banding because the banding is becoming prevalent all over the world so we will be and this has a lot of implications for the baskets for the average and for the benchmark in exercise and also there was a compromise to review the ICT price baskets periodically we don't want to because we agreed upon in the last estimating after some work now there was also the topic on should we redefine broadband what is broadband is it 256 kilobytes per second should we define it 4 megabytes per second threshold and there was a lot of discussion going on on this but at the end we decided also for practicality reasons I think to keep the definition of broadband and reporting it because of the speeds of the broadband fixed broadband connection that you are enjoying and you know that we had only three intervals before from 256 up to 2 megabytes from 2 megabytes up to 10 megabytes and then from 10 megabytes up we had the third category well now this third category we are splitting it up in three branches which I think is important which is the branch from 10 megabytes to 30 from 30 to 100 megabytes and up from 100 megabytes up to infinity so this will be the new three broadband speed tires that we are going to be using up from next year we are aware that some countries because there was a lot of discussion on this may have difficulties in collecting this data with these precise speed thresholds so for these countries they can keep for a couple of years they can keep the old thresholds that they are the ones that I am showing you here for a couple of years and of course this will not come the new version of the speed tires now there was a lot of discussion on cybersecurity indicators I guess later there will be a presentation on this this was an important topic it was raised in the last meeting we at the very end what we ended up concluding was that there is no one single measure there is no one single indicator really on cybersecurity we couldn't rely upon it is a very sensitive sensitivity issue cybersecurity is being measured in very heterogeneous ways from very heterogeneous institutions sometimes public sometimes private sometimes you don't know so this topic at the end we faced a huge difficulty in trying to get one or two indicators that would potentially be comparable we couldn't find them really at least for the time being so what we end up concluding was again that maybe from the demand side if we narrow down the cybersecurity topic the users are really doing in terms of cybersecurity and on this I recall the previous presentation by the OECD which had some very interesting suggestions on this and we want to explore but more from the demand side what can be done potentially on cybersecurity now the last topic before the future work that was deal with was in a previous extraordinary meeting in March that took place to improve upon the IDI index which I know that is matters a lot to everyone so we had a joint meeting and there was a subgroup by the way led by Ratty who is here thank you Ratty who came up with the proposal at the end that was long discussed and you know I am being conservative here it was lengthy discussions so at the very end we ended up concluding and agreeing on a set of indicators that would improve the IDI the main ideas are the following one indicator has been dropped two has been dropped one has been redefined and five a new and we are coming from 11 indicators up to a set of 14 indicators that are going to be used from next year onwards in this IDI and let me well end the indicators the structure itself doesn't change we have access indicators usage indicators and skills indicators to construct the whole index and what I would advance to some of your comments is the fact that why don't we just decided upon this why don't we just let it go and see how the properties are and see what the deficits are which there will be and see how over time we can improve upon that but let it go for a while because we just change it and we cannot keep changing an index every year on the future work this is the last topic I promise on the future work I insist that this is open to anyone who wants to participate there will be a subgroup to be created on what spectrum allocated spectrum and capacity all over the world and this is going to be a big leap forward I believe there will be a topic on quality of service there is a lot of demand which are meetings now 5G related indicators and IoT Internet of Things availability which we think they are related this is open up as a new topic for discussion for suggestions for indicators for experiencing for many things in the online forum and for the next meeting as well and also for experiencing indicators on OTT which there is a huge demand on them and just by thanking you do a participation and by encouraging you please to do participate in the ECTEGH meetings do participate in the online forums do make suggestions experiences, indicators problems that you face because this helps a lot coming up at the very end with a nice set of results and in my name and under the name of Annie Baldeo who couldn't show up here the vice chair for the being as chair for the ACTI and also thanks to Alexander from the EGH for your great work that's been announced in the World Telecom Development Conference in Argentina it was mentioned the resolution number 8 that the former mandate for the group and it was approved by the city and three members so thank you very much for your work and for Alexander as well now we're going to move to Mr. Savio Di Nicola and he he is the officer of ACOM from Italy and he will be presenting the new matrix for broadband and cyber security so please give him applause this is the outline of my presentation I will start discussing about broadband matrix and then we will focus on the Italian project MISURA INTERNET when we discuss about broadband matrix we don't specify a single well-defined matrix actually we speaks about many different matrix just three of the principle matrix that is bandwidth, loss rate and delay but there are a lot of other magic that affect broadband performance for example if you ask to online video gamer they are fanatic about speed and they can tell you hundreds of parameters that can affect your speed connection but if you give hundreds of parameters to consumer probably you are not increasing their benefit there is a trade-off between information overload and consumer matrix so what you obtain probably is a digital tower of bible this is a picture from the cover of a journal of the computation machinery association and they have noticed that there are in the computer science thousands of different language help to computer to communicate but from a regulatory point of view what you should obtain you should have different reports that explain broadband matrix but they have different approach for example you have a project about university or private company but they have different purpose from the regulator from a regulatory perspective you have three main objectives you want to increase awareness in consumer because a lot of time consumer don't know what really affect the performance of your line sometimes they think that if there is the name fiber in your commercial offers this means that you achieve the full speed but a lot of time there are other parameters that concern then you want to facilitate to compare offer because now quality is something that can lead to an higher market share because now the fight is among operators on the quality more than on the price and finally probably this is the most important things since we are an authority we can certified the speed that you can achieve if you read one of the report of other company private or public you don't have a certification of your speed inside with this project we will see you have a certification of your speed so this is a project that start many years ago in 2006 because we need to have the collaboration of old operator not only of the major company but also the small operator we built this technical board and all operator participate to this board and sharing ideas fighting but finally we reach an agreement on the technical also on the financial parameter because you need to found this project so what we decided to measure we measure speed delay also service availability this is the four parameter actually we need to collect a lot of sample and you don't need to collect samples just in one time slot because as you know the network load can change during the day so you need to collect sample in different moments of the day after 24 hours the procedure is finished and you collect all the sample then you cut the tails then you evaluate a part of from the average and the standard value also the guaranteed speed that means the lowest percentile and also the maximum speed the highest percentile from an architectural point of view we have adopted two different type of measure one centralized measure that is done in the internet measure that is done on the client side this means on the computer of the end user so when you are an end user you go to this website Misura Internet and you have a public area where you can download general information about discussion frequently asked questions and definition and a private area and register with your profile and what is most important with the name of your commercial offer then you can download the software and to start the software on your computer before starting the procedure the computer check that some of the premise that we already mentioned are not influencing the measure that is for example the CPU or the hardware or the Wi-Fi connection and so on after this first check-in it starts the measure and after 24 hours it ends and you obtain this certification so when most of the case unfortunately the advertising speed is different from your certified speed you can ask to the operators or you can withdraw from your offers without penalty you have these two options so this is what happens on fixed broadband lines but of course during the years we evolved with this project and now we can also provide measure over mobile network but with different architecture because mobile network is different from a point of view even a meteorological aspect can influence your measurement another thing that we have done is to simplify this procedure now it's no more 24 hours because it's a long time for a user to wait for the certification but there is a speed test that in one hour can give you the certification and finally we are testing and probably next month we will release this new version of the software broadband lines above 30 megabit per second so we are aligned to the new indicators presented by Nego so what is the remark and the conclusion that now competition is no more on price but it's shifting towards quality a lot of mobile company they advertise that they are the speedest so without certification by us another thing that with this huge project you need to develop deeply technical aspect you need technical board and discuss among all operators not only between the incumbent and other operator but you have to convince all of them that what you are doing is beneficial not only for consumers but also for operators also because they have to pay for this project so you need to convince them and finally since as you know technology run more than regulators you need to monitor your monitoring system I remember when we started this project in 2006 for us it was achievable speed and no one had this type of line in Italy but now 30 megabits it's not more sufficient for the market so finally some reference and this is the last slide that I hope you can enjoy more than the presentation I don't know if you have any question Thank you very much Mr. Salvia for very very interesting subject and the new matrix and now we are going to pass the floor to Sirin Tilly the CEO of National Digital Certification Agency of Tunisia she will have a presentation on the majoring cybersecurity effectiveness so the floor is yours so my presentation will be dealing about how to measure cybersecurity effectiveness so now it is needless to say that ICT and broadband are key drivers to economic growth to drive economic growth and enhance well-being we have a lot of people doing shopping on the internet banking water and electricity supply also we can find them on internet social networking etc and all these changes that happens to our society well actually it is bringing with it a new set of challenges like cybersecurity challenges are you ready enough for this digital new digital word are we protected enough against security threats so if you want to be able to know whether we are secured enough or not if you want to manage your cybersecurity not to be able you need to be able to measure it so something that you cannot measure it you cannot manage it so here I'm giving you some statistics taken from the European the European NISA ICT that shows that the main challenges now for defining indicators is to identify appropriate indicators for your system and network so you need to measure your security but you do not know how to do it you do not know how to choose the indicators to be able to measure the security and you also do not know what is the like to say data to collect what is the needed information whether you are ready enough to go on internet and to do some electronic transaction so we need to build up a framework of indicators that will be able to tackle with these two main difficulties to find security indicators so if we were talking about security cybersecurity indicators we can split them into three main categories we have the country level categories as an example we will show in the coming slide the global cybersecurity index and the individual level are you comfortable enough to do electronic transaction to do banking on internet so this deals about individual comfortable are you comfortable with that and as an example I give you the euro barometer on cybersecurity and for the last level it's the organizational level and as an example we will show the information security indicators developed by the European security indicators actually are moving from the quantitative indicators qualitative let's say the country level and individual level will see that it is more related to qualitative indicators whereas the organizational level it is more based on quantitative indicators so for the ITU global cybersecurity index it is divided into five categories we have the legal category are you comfortable with that and as an example we will show you the number of indicators to deal with cybersecurity is there enough technical institutions and framework to deal with cybersecurity is there a policy coordination and strategy related to cybersecurity is there enough research and development and training programs related to cybersecurity and is there also enough cooperation between countries from the survey that they have conducted on the 193 member states they have raised this map that shows the level of commitments of countries and the highest level of commitment is the red color and the lowest level one is in the red color so just give you an overview of how countries are dealing with cybersecurity and now we move to the other level that is related to the user and in the case of Eurobarometer they are measuring the attitude of European towards cybersecurity what is their feeling when they are using internet and from the statistic that collected during the survey it shows that cybercrime is actually at the same level with drug trafficking and with terrorism they are giving a high importance to cyberterrorism so authorities and government have to put enough measures to supply trust and confidence in user that are using internet and another interesting information also that you can see in this special Eurobarometer is that the most important thing people are caring are actually giving importance to is their personal data they do not want that personal a misuse to have a misuse with their personal data and also with their money when it comes to paying what people are caring about and on that point I would like to point to stress on something it's related to personal data that recently Tunisia has ratified the convention 108 related to the protection of personal data just to show the commitment of our government to do protection of personal data and in France they are working now on a new regulatory framework to organization and to banking account on some other examples now we move to another kind of indicators it is more likely to be quantitative indicators you need to measure as an organization you need to measure to see whether your cybersecurity tools and cybersecurity processes that you have inside your organization are enough in order to guarantee a good level of assurance in order to make sure that the security is effective and that it is worth the investment that has been done and to benchmark the effectiveness of security measures and we need to have a framework that will be recognized with reliable statistics so with the Etsy the information security indicators there is a huge documents that have been developed by the Etsy we will focus on the IC01 that will give you for organizations to use to benchmark the security to benchmark the level of security there is around 100 set of indicators but in order not to get lost within this 100 set of indicators you have another guide that will tell you what are the indicators that are relevant to your activities that will guide you to select the relevant indicator for that and to help you to some other also document related to that that will be able to see whether your security operation center is secured enough are you capturing the right information are you dealing correctly with the event related to cybersecurity but we will mainly focus on the two first part of the IC IC indicators so they are split into two categories we have the security incident categories and we have the vulnerabilities categories actually an attacker when he is targeting an organization he first of all he will try to look for a vulnerability for a weakness in your system in your network and to take advantage of this vulnerability in order to gain access to your network and to your organization and in that case you will have a security incident that you need to deal with very quickly so I will give you some details some examples of these indicators for instance as a vulnerability indicator we have for instance a system that has not been patched as you know there is always some vulnerability that are discovered in operating systems in software and so on and you need to make sure that you have the appropriate patch to deal with this vulnerability to eliminate this vulnerability from software and from your system and you need to have a security policy that will tell you how to apply the patch and you need to make sure that the security policy is followed and that you are doing the patch in the limit of time that has been defined in your security policy another vulnerability is related about passwords using weak passwords is not good to protect your asset so you need to have also like a sort of policy for the password that will tell you how long the password should be what are the characters that are not protected with antivirus or firewall and you know vulnerabilities on workstation are to be taking care very carefully because this is how ransomware actually will be introduced within the workstation and this is how companies can lose a very huge amount of money so these are kind of indicators as you can see there are very quantitative indicators and they will give you a lot of indicators related to security incident in that case the damage has already occurred so denial of service attack this is what happens recently in UK and it targeted a lot of hospitals and here we need to know to measure how this denial of service attack is happening actually how many traffic is coming from the same IP address within a very short period of time and there are enough resources and enough tools to be able to collect such information also there need to be a coordination between organization to collect such a information of denial of service and distributed denial of service there is also another example of indicators that is very interesting to mention is the malware you have to know whether there are attempts to install malware on the workstation within or an IPS an intrusion prevention system this is a system that will let you know what is actually happening within your network within your system another example is the phishing attack the phishing attack is sending you an email and claiming that it comes from your bank where actually it's an attacker that is sending that email and within that email they are pushing you to put your password and your login information so these kind of phishing attacks and this is one way for attackers to spread like malware and run somewhere within your system and network so one kind of things that you need to measure is how many attempts of malware are done on your organization and on your system network so just to give a full image of this is that the indicators are very use set of indicators and it's very difficult to find out what are the indicators of the organization and a good starting point is to try to have the needed tools for that it's try to have the needed a team based on in your organization that will collect this information that will see the that work on that information currently the information that is taken from the log from your system that is taken from the user themselves and also to coordinate with other organization within the need to set up a legal like a big framework related to indicators that should be applied on a national site and also an international site so thank you for your attention thank you very much Serene for great presentation and now we're going to move to Ivan Ivan Varejo the market analyst for the ICT data and statistical division of the ITU Ivan will present the real speed of broadband so the floor is yours Ivan thank you my presentation will be complimentary to that delivered by Silvio in that he looked at real broadband speed measurements and use them to inform and empower each individual user whereas in my presentation I will be looking at all these individual measurements put together combined in order to derive insights on the aggregate metrics of real broadband speed in a country let me start with some context this pilot is part of the project big data for measuring the information society in plenary two yesterday we had the presentation of pilot projects all of them relied on data coming directly from operators mostly call detail records and IP detail records the Swedish project is different from them in that data are not coming directly from the operators but data are collected from an online speed measurement platform which makes the scope and methodologies of this project it's about the online platform it's called it's developed and maintained by the internet foundation in Sweden and it looks like it's shown here in this slide there's an online web where people can access they see this they click on start test and then a series of measurements are run the results of these measurements are stored and recorded and this is the raw data that we used in our project this online platform also allows to have an application that people can download and run the same measurements from a smartphone on the relevance of measuring real internet speed I will go fast because already has been covered by Sylvia's presentation but this is highly relevant and this question is can crowdsourcing internet data be used to measure real broadband speeds and this question is relevant first because advertised speeds which is what operators make available are important but are just a small part of the total path so do not fully characterize the end users quality of experience hardware based measurements so cannot be implemented at least not in a regular manner in all countries and so we are left with software based crowdsourcing I would just like to mention some of the statistical issues that need to be considered when dealing with this kind of data which are very similar to the statistical issues of non probabilistic samples here they are listed I would just like to mention the most important one is following the behavior if I'm surfing the internet and I don't have any problem most likely I will not go and check my speed if I'm surfing the internet and I'm facing congestion or slow down I have more chances of going and testing my internet speed so that makes that the population that we have in the records does not fully correspond to the whole population of internet users so I would just like to mention what's the most common approach is that one of using ground truth data that you know are correct to calibrate your records I would also like to mention Siri already said that there are a lot of technical issues on the measurement that need to be taken into consideration I would just like to mention a major one which is on this slide you have the diagram of an internet connection starting from point number one the computer at home and ending in point number five the content located in a server somewhere in the internet so advertising speeds which is what the operators report measure the the throughput between points two and three which is only a part so depending on the results and also answering to different questions in this table I've listed some of the most well-known online speed measurement platforms Akamai Ukla also some nodes which is hardware based and the one we use this is just to signal to you that they are measuring different parts of this path and using very different technical implementations that one platform is better in absolute terms that the other is just that some of them are better for answering some questions whether the others are better for answering other questions so that was it for the methodological boring discussion now let's go and see the data so this is the data we have from 2011 to 2016 in dark blue you see the observations for fixed broadband speeds so the tests run within Sweden and you see that there are about 15 million per year considering that the number of fixed broadband subscriptions in Sweden is around 3.5 million in principle this should be a good enough sample a few words on the data processing just to say that we receive the data from the internet foundation in Sweden we were able to store it in an SQL database then we transfer it to the cloud using Amazon web services we did the heavy lifting data processing using Spark so that we could do in parallel and speed up the process the resulting tables were then transferred back locally and with R the analysis was run and these charts were produced just to signal to you that this we were able to follow to use the cloud for the processing because confidentiality of this data was not that high you probably cannot do that but it was okay so some results if you look at the blue bar that's red bands column so our measurement you see that in 2012 average download speed was about 17 megabits per second and in 2016 52 megabits per second so okay we see that it has increased now how accurate is that if we compare this measurement with the results of other platforms also from Sweden so there's sort of a consensus that there's an increase but you know we cannot be very sure based on this on what's the actual speed customer self now on the actual sample that we had in in our project the chart on the left-hand side is actually the sample we were using divided by the operator so the number of observations or the percentage of observations per operator if you were randomly sampling so sort of the ground truth you can see that there are some differences for example the part in red refers to the proportion that relates to small operators which is larger in our sample than in the reality and the light blue is the part that relates to the incumbent which is smaller in our set compared to the reality and the differences matter we have statistical tools to provide a clear reply to that so if we run a chi-square test between the two we have a very clear result saying that the two samples cannot belong to the same population which brings us to the conclusion that we have a selection bias in the records we have so to finalize just some takeaway points where needs to be considered from where to where are you measuring and some other details on the implementation second the fact of in-house congestion if you are measuring from the laptop to the end there will be a part from the laptop up until you reach the ISPs network which is beyond the control of the ISP so if you are measuring that and you are trying afterwards to check this data with internet service providers that might tell you that you will be right then that is the issue of selection bias that needs to be taken into consideration and finally a very important point is the importance of ground truth data usually panel data or official house of survey data in order to calibrate your set or at least in order to check whether you have biases that would invalidate the results here just for your reference some references on what I mentioned and that is it for the presentation thank you thank you very much Ivan for interesting presentation now I would like to open the floor for questions if you can please get the sign of your country up and we will have three questions and then we will get back to the panelists to answer them so please if you have a specific question to one of the panelists please mention that so we will start with Mexico I'm Salo Sanchez from the Mexican Telecom Regulator congratulations to all panelists for your interest presentations I have one question for the Telecom Regulator from Italy obviously these kind of tools are low cost to obtain really valuable information and that's clear I think for everyone but how do you compel users to use mobile or fix broadband applications in order to receive information from them because I have heard other regulators that are coping with how they incentivize the use of these apps from the people at the end and this is something that it will be interesting to know from your side thank you now we are going to have a question from Portugal thank you Mr. Chair concerning the presentation from Italy I've been knowing what were the results of these tools were consumers using it to complain do you have an increase in the number of complaints or do you have an increase in the number of contract terminations and also in the case of operators are they improving the quality or are they decreasing the advertised speed that was my first question the second question concerns Ivan's presentation we have in Portugal conducted a survey to determine exactly who are the people that are using these speed tests and what we found out was that there is in fact selection bias people use these tests when they have problems or when they buy a new connection also there is a very few number of people that use these speed tests but these few people make lots of tests which means that when you use this data you have to average by IP address otherwise you will get even more bias and number three the tests tend to be concentrated in certain socio-demographic categories people with more literacy levels will use it and people with less literacy levels won't use it and I was wondering if you could explain something about the methods to correct these bias thank you we will have a question from UAE this is Saad Naif from Federal Competitiveness I have noticed that the data for the new indicators are already there in this the IDI report 2017 but I believe I wonder if the new ranking as per the new methodology but as per 2017 data not the next year data is available if not we would like to have it so better benchmark with other countries and if possible in open data format not in PDF format thank you UAE now we will go to Salvi to answer a question from Mexico we try to answer to both questions of course we have a selection by also Italy we have noticed the same problem that even highlighted and usually people with a problematic connection let me say use this software so if you saw the report a lot of them failed the test means that the average speed is lower than the advertising speed and this is probably one part of the questions the other one was the reaction of the operator if I understood and usually first of all the operators try to improve the quality so they say okay let's try to speed up a little bit your connection and in most of the case they succeed and the user remain with that offer in some case but it's a small part not the majority they they can't provide they advertising the speed and in this case the majority of this minority case change operator without penalty in other case as you said they reduce the price of what they pay because they pay they choose another option with a lower advertising speed I don't know if I have answer to both questions okay now we Yvonne want to respond to Portugal question yes so I think regarding the I just covered there surely in the presentation selection bias as Portugal was raising there's also the issue of unequal participation when these tests are run not compulsory but at the choice of each user the way that this platform Breadman's column has of differentiating or counting each individual user when they when they do it online but also in the app is using some cookies so they have a way actually of getting of that unfortunately in the data they shared with us we could not we did not have this information so we also had the the issue of unequal participation on how to correct the bias I think this is still sort of a research question open some of the results that have already been successful involve the calibration of the data with some other so for instance I think that it was the statistics office in the Netherlands that at some point in the past they they used they had a panel consumer panel in which they introduced the same measurement platform and then they run some control test which they had the control with and also they this way they were able to link these results also with demographics and then using these panel data now had the results of the test they were able to actually correct the data from the non-panel data that's an option but that has also some cost implications in terms of in terms of adding that to the panel there are maybe other options that the colleagues from Flowminder mentioned in their presentation from yesterday but this is this is open for yet for discussion thanks I'm thank you Mr. Chairman yes I would like to reply just very briefly to the if the replication of the results given the new indicators that are being proposed and agreed upon if the replication is possible well this hugely will depend on the quality and the amount of data to be collected from ITU in the very near future I recall that there are five new indicators this is a tremendous effort for everyone to provide the data later for ITU to collect it to validate it to check it out and then to run of course some test on how the rankings will look like but I recall that this will be in the long questionnaire you will be receiving it by merge so that I take profit of your question to insist on the importance of providing all the data that is being requested in the long questionnaire otherwise some exercises will be impossible and I recall that there are five new indicators which means that these indicators will exist on it with one observation thank you thank you Nego I think due to time and a very heavy subject that we had in this presentations from the great panelists in here we're gonna I'm sorry that we're gonna close the floor for questions and I'm gonna pass the floor to Mr. Chair Mr. and thank you for this moderation thank you for the panelists for the deep information that we get and for the question because the questions were very interesting from practical perspective