 Hi, my name is Vinod Kall and I work for Intel Corporation. I do a bunch of audio drivers for audio DSPs for Intel and today we are going to talk about ASOC topology. This is the kind of work which we did as part of Intel DSPs for latest Skylake and beyond processors. So a bit of history before we go started with this. So ASOC topology is not a new concept. It was actually like three or four years back. Credit score to Liam Goodwood. He started writing this when he was working for TI back in the OMAP days. And in that what we did is basically have a simple DAPM description for a DSP. And that DAPM description was essentially coded up in the user space and sent down to the kernel so that drivers can load this description. In this case, once we started upstreaming it, it is actually not so nice to have the DAPM description in the user space because that essentially exposes all the internal details to user mode. So that's when Liam came to Intel and we started re-looking at this and started cleaning it up a bit and started upstreaming. So initial version of the patches were accepted in version 4.2 and eventually other features made it into the main line and ABI is now deemed stable from 4.9 onwards. The subsequent ALSA lib support was made available in 1.1.0 version of the ALSA library. So why do we need the ASOC topology or why do we need the topology description in the user space? So assuming a simple DSP graph like this one in the picture here, you can have a DSP which probably has two PCMs and two I2S endpoints and between them you can have modules doing the audio processing like say in this example AB and CD doing the front-end audio processing and then on the back end you do the post-processing in ENF modules. Now since it's a DSP, what we get from customers and vendors is oh, you know this is nice but I would like to do something else, something like this where I replace some of the modules in your system by my own modules or by a third-party module which is kind of okay. So I would have initially coded all this particular description in the driver and then when I get the request I would change the description, modify it, put these two new modules inside. Now in a kernel if you do that probably you will probably do a build flag or a module parameter to differentiate between these two graphs. So you're happy, you deliver the patches, you probably upstream them and everybody's happy. Then another customer comes and says, can I do this? I don't want this module, I don't want two PCMs, I want only one PCM backend and I want instead of your modules I want some other module called Z. So you go back to the drawing board and say oh, how do I figure things out and try to change the description again and so you do that, another guy comes and says you know what, I need a third PCM probably, I want to do a low latency already buffer application, can you do that for me? So this kind of cycle happens endlessly that you keep on getting requests from customers and you have to continuously modify your graph description because it's a DSP here. Everybody puts DSP for scalability and flexibility purposes and if you're not able to customize it, it doesn't really make a lot of sense. So that is when we started looking at ASOC topology, Liam was working in TI and then we started collaborating and then he came over back to Intel and that is when we started re-looking at it, how do you want to do things? So the way we have done this now is we have a simple description in user space, we call that a topology configuration, then we have some tools which can build that configuration into a binary description, we call it topology.binary and that binary we pass on to the kernel. So kernel loads that binary description and then creates the whole DAPM graph and widgets and associated elements for it. So now as we say in the problem, we had a problem of being able to customize, not able to customize for the customers. Now instead of having module flags, build flags or all sorts of things, we can just simply provide a different configuration file to different customers, they can build it for their own and use that and load it in their own systems. So this way we solved the problem of configuring the system, scalability and changing the graph per user or per vendor for us. So how do we do this? Okay, as I said we described the topology in the configuration file, this is called topology.conf, you can see the sample configuration files in the ulcer library right now. Since we are using it in the user space, we need user space tools and utilities. For this we have our own good old ulcer library to do the job. So we have APIs in ulcer library which can parse this configuration file and build it for us. The syntax of configuration file which we'll look in the future slides is basically the old UCM style. This is what people used for UCM description. For main purpose to reuse the UCM description was so that we don't have to rewrite the new parser in the ulcer library to the parser syntax. We already have ulcer UCM parser. This is built using ulcer utility tools. The ulcer utility provides a tool called topology tool which we saw in the previous slides. So you can use a simple configuration file and the topology tool to build it out. Along with the whole configuration file it allows a manifest. This is just sums up what you are going to do with the configuration as drivers can use it for bookkeeping. For DSPs, apart from the description it is pretty much important that we provide a way or mechanism for drivers to know something more about the particular element or particular widget you are going to do. Typically it might be for let's say a coefficient for the particular module which you need to run at the initialization time of the module or it may be some magic numbers which you would want to communicate with your firmware. So that is not a standard description that is a specific driver specific or firmware or DSP specific description. So for each element in the configuration file we also allow a private data. So for us it has been very useful because we can describe all that what we want to do for DSP and all the magic numbers DSP needs to put that in the private data and send it down to the kernel and kernel can forward it out to the firmware. So what are the information do we have in the configuration file? First, if you want to describe a particular configuration for a driver, assuming we are a kernel driver for a codec what we do is to define the control. So similarly here we will also define the controls like mixer control, enumerated control, or byte control. Byte control is specifically important in terms of DSPs because you would typically have a module which would need a bunch of parameters to be configured. So you can send some 100, 200 bytes to the particular control using byte control and in this case we also did something called as TLV byte control because traditionally also controls have a 512 byte limitation which we can work around by sending it as a TLV type of few kilobyte of data or more. So that is why byte controls are important here. Then we define for the particular graph the DAPM widgets what we are going to create and to link those DAPM widgets we have the DAPM graph associated with them. Then as we saw in the previous diagram in one picture we had two PCMs in another one we had three PCMs. So we allow people to specify how many PCMs you have to use because we believe that for a system PCM is limited just by your DMA channels probably for a given system and then people would want to have multiple PCMs on a system or even lesser PCMs on a system so that is kind of a software concept which should be customizable. So we provide people to specify front end or PCMs and if you are doing some processing from the modem to the codec we specify the configuration for dilings and backends. So how does the configuration look like? First to specify a mixer control we define it by term section control mixer with the mixer name. Mixer may be multiple multi-channel controls so in this case we can define the multiple channels with the channel declaration. The channels can be mono or stereo to be simplified but if in case we have multiple multi-channel control you can define front left, front right, rear and so forth and so on these are all ALSA standard control names. Then if we are in kernel when we define a particular ALSA control with the ALSA control we typically define the callback ops which are basically set get an info callbacks which are called these callback ops can be one of the set of the standard callbacks which are like wall s, w, mixer, bytes, enum, strobe, range we are allowed to define all this along with that if you are in the kernel what you can do is to specify one of your own driver specific callbacks typically we do it with the external keyword when we define ALSA control in the kernel so similarly in this case you can define ops control we'll have let's say info function as my standard wall s, w but in case of get input I want to define my own function so you can define a value here which will correspond to a driver specific value when the kernel loads this control it will try to find the value and associate it with this control so this way you can either use the standard values or your own driver specific values the control always has an axis this specifies how a user mode can access this control which may be a read control a read-only access or a write-only access a read-write access or a TLV access or TLV read access, TLV control access so you can specify what the axis of this control would look like along with this you can specify the maximum values this control can take whether it's an inverted control or not if it's a TLV type of control what is the TLV data associated with it and for DSPs private data second type of control is byte control you can declare that with the section control bytes with the name associated with that particular control most of the things are similar to other control, mixer control type you can specify along with that a byte control whether if it's a DSP in a codec probably you would have a register associated with the particular codec and then you can define what is the register values what's the base, what are the number of registers what are the mask values and what are the maximum values this control can take if it's an enumerated value first we need to define a text section for the enum values it would need to take so that we define using section text with the name of the enumerated values and with the set of the values then you define the section control enum for the enumerated control with the name of the control rest of the things are same but the only difference here in this case would be the text section in this text if you can see we have an EQ1 control which is referring to the particular values you would have given so this way you can define a control which has set of the values and the control itself so this is all about control moving now to Dappam widget a Dappam widget typically you would define in the kernel with a macro and you will pass on set of the values these are kind of similar things what we do here we define a section widget for a Dappam widget with the name of that particular widget and first we define what is the type of the widget the type of widget can be AIF in, out, PGA, ADC, DAAC what else PCM which is a die-in, die-out, input, output these and few more are allowed types of widgets so we define what is the type of the widget if it's a stream widget type you need to associate a stream name with it so you can define using stream underscore name value then the typical stuff what we define in the kernel it's the same thing here as well so is the nopm valid or not what is the register bit, what is the shift, what is the is it inverted along with that there's a control, there's a fail in also control also Dappam widget called a subsequence not many people use it but it's in opinion it's quite useful for the DSP guys if you have bunch of widgets for example PGS and when you want to do a DSP sequencing your sequencing is pretty much defined by what your firmware can do so in this case a firmware is enforcing some rules for the ordering of those for the creation and enablement of that particular widget you can use the subsequence field within that widget type to define what comes first and what comes last so that is where subsequence field for at least for us is pretty important because DSPs can have different kind of sequencing requirements and you can satisfy that with this value then event type what are the types of events you would need so if you have a Dappam widget and you want to have your own event callback for that particular widget this is the value you can use so you can define event types in your driver give them indexes one onwards and then once the kernel loads this particular widget it will try to map that type with this value given here and set that particular callback for event in this widget now for the callback we also have the flags like standard flags pre-pmu, post-pmu, pre-pmd and post-pmd so if you want to specify the flags for which the callback should be called we can set it here and finally if the Dappam widget has a mixer associated with it like a mixer control you can define the mixer name here or if it's an inamirated control associated with it you can define the inamirated control here and in the private data so we have defined the controls now we need to associate them controls all the way into a DSP graph so you can define that with a section graph and as you do in the kernel when you have the Dappam widget same control source you would do the same thing here but it's just a comma separated value if you have no control no need to keep null as we do in the kernel we just leave it blank finally the PCMs so PCMs in this case are of two types one are the front end PCMs and second are the back ends which are basically the i2s ports so first we define what are the capabilities of your PCM that we define using section PCM capabilities with the name of the particular capability here we can define what are the formats this particular PCM can take what are the channels minimum channels and maximum channels we can have for this PCM what are the rates we can have for this PCM once we have defined the capabilities as you typically would do in your driver it's kind of analogues to that we define what is the configuration that particular PCM is allowed to take so for the configuration we have typically two things playback and capture depending on what all you need to support so we define that using section PCM config with the name and then for the it has two sets one is the config playback and another set is the config capture in the config playback you define the format rate channel and if you have a TDM slot you can define the TDM slot as well so you do this same thing for the capture this specifies what is the configuration which is allowed to take you can have multiple capability sorry you can have single set of capabilities for the PCM and multiple configuration so once you have defined these two you can define the PCM the front end will look like something like this which is declared using section PCM ID is the one which is used to bind it to the PCM associated in the driver and then you can define the die name that is the name which will appear when you create the die and you define the playback if it supports playback for this we would have previously defined the capabilities you can set the capability here for this PCM and the configs it supports so you can have one config and multiple configs depending on the use case you need similarly for the capture you would do what are the capabilities for the capture what are the configurations required for the capture now along with that typically we sometimes need to have playback and capture in symmetric format so you can define symmetric rates or channels or sample bits true if they are not true you can set them to false if they are true you can set them to true and associate again a private data which your DSP can use after PCMs we can define die links so die links are something like codec to codec links in the DSP where you are probably getting the data from modem and sending directly to the codec and so forth so you can define those die links in your topology as well so that can be defined using the section link name in this you can have a stream name associated that is the DAPM widget it will be associated with it and you can have multiple hardware configs these are the hardware configs you will apply for that particular die link along with the configs kernel will pick one config which will apply which will be applied for this at the default that you can specify using default hardware config then the usual symmetric rate channel and sample bits if it is true or false you can specify for the die link typically when driver kernel creates a die link it also creates alsa control for it if you have the multiple configuration the config 1 to n in this case would correspond to the enumerated values for that particular control last the back end i2s so one of the back end configuration is actually a bit different here because we don't define the back end as such we define the configuration which can be applied to the back end this is for the simple reason that your front end PCMs are software driven you can define as many as PCMs as your hardware limit or if you can define only one or two based on the uses you want to do on the DSP but your back end are actually limited by what your hardware supports if you have only two i2s ports you can do more than two so it's a hardware thing so that should be enumerated by the hardware you can scan how many ports you have and create your back end links and die links but we allow the configuration to be specified so for example in one particular board you would want to run it at 48 kHz but on a different board you wouldn't want to run it at 96 kHz or 41 kHz so from topology we allow people to specify what are the formats you want to apply for that particular back end configuration so that you can define using section die and then for the PCM playback you define the capabilities for the PCM capture you define the capabilities you would like to have you would like that particular back end to have so we have been talking about private data a little bit more details because this is kind of little interesting for us so how do we define private data so for each of the modules you can use a section data to define private data it can be of multiple types if it's for example a particular binary coefficients for a module let's say equalizer you can define a file where you have all the binary coefficients it can load and set it if it's just a simple byte you can define a comma separated byte value or a word value or this thing shots along with that what we also added is the support for tuples so byte's word shots are fine but they are not human readable that means they are very very hard to change so instead of having these binary values if my DSP needs value A, B, C for a particular use case I want to change those values if I can make them human readable it eases people's job to customize those values for them so that is why we came up with the support concept of tuples we'll explore more in the subsequent slides so once you have defined all the private data you need for a particular module or a particular element you can define section you would have defined as a section widget or section PCM or section mixer control for that we will have the data section in that you can specify all the private data you need it can be multiple data sections for the private data because typically if you have one data section you can have only one type of data so if you have a combination of bytes and shots you would need to define first as bytes and then as words and then add both of them to the data section so what is this tuples all about so what you can define is tokens so it's simple key value pair kind of work what we have done here you can define multiple tokens token 1, 2, 3 and they would mean something which your driver would understand for example for our modules we need to pass a UUID value so one of the things what we do is what is the UUID of this module which we need to send it down to the firmware and we can define a UUID token for that and this is how we code it because tomorrow if somebody is integrating another third party module which has a different UUID they know oh it's a UUID this is how I need to replace it so that way instead of binary byte values or hex values it's simple to customize for user so how many types of tokens we can have first is a tuples string you can define a string which you can pass to firmware or your driver you can define a UUID tuples dot UUID or you can define a boolean value with tuples dot boole or you can define bytes simple byte values you can have multiple tokens for different type of byte values which your DSP would understand using tuples dot bytes or again shots or words similar now we have defined the configuration file we have defined the private data we have defined all the widgets in the configuration file what do we do with this configuration file we build it how do we build it also it has a tool available it's called also our topology so using the minus C option you can give what is the configuration file for this particular system you want to run and with the minus O output would be whatever binary file you need to write this binary file would eventually need to be installed in the kernel so that your driver can pick that up so this particular tool uses also our topology APIs which were defined for this work in this the also topology tool basically uses snd underscore tplg underscore build file API it's a single API which does full job for us it essentially takes this configuration files passes through it finds all the sections finds all the data for the sections and creates the elements for it if you don't like this tool you can write your own tool additional C APIs are available in also library for parsing these API looks like this so first what you need to do is to create a new topology instance this you can do using snd underscore tplg new so it will define a new instance for you and once you have the instance created for all the elements which we saw in the configuration file you need to call snd tplg add objects the object may be control, mixer, byte, enumerated or it may be a die it may be a dapple widget it may be a map once you have done adding all the objects if you want to have manifest block in the file you can do that by invoking snd tplg set manifest API and once you are done with everything you can ask also library to build it out for you snd tplg build by specifying an output binary file and at the end you free the binary so how does the binary look like so binary has header associated with it this header perceives each and every block of data we have in the binary file first we have asock as the magic number so that we know this is our binary file and then second word will be abi file abi right now is version 5 then we also allow people to override and do their own stuff if you want to do that you can specify your own vendor version so that core will ignore it and directly pass on it to the driver if you are not doing that you can define it as one of the standard topology types which we were discussing like control, mixer, widget, tie, PCM so that you can define the topology type field after that there is a field for header size this is typically used to check if you have any abi mismatches if you are defining vendor version you would need to have your own vendor type so you can have use the vendor type field for that and after that if for the particular block what is the payload you are defining so that payload size will be there followed by index and count of this block so this particular header is there it's not just a file header it's a block header so we will have multiple blocks so the binary file will look like something this so you have a topology header and followed by the manifest manifest is always the first block because as I said it's used for bookkeeping operation so it makes sense to tell the driver first what how the manifest looks like then again you have a header for mixer controls you define all the mixer controls in this section together followed by another header for enumerated controls followed by enumerated control data and the header for byte control followed by the byte control data the header for dapm widgets followed by dapm widget data header for dapm graph followed by the dapm graph header for the pcm followed by the pcm and die links and dies so this is how the complete binary file look like so we have done the whole user space thing where we define the configuration we now have built it now we move the gears onto the kernel side what do we do in the kernel so you have this binary file created you install it into the kernel and then driver would pick it up how does driver pick it up we have a nice API call request from where so this is what we use to load the file from the user space into the kernel so your driver would call request from where API once we have the file loaded into the kernel space we pass it to the ASOC course using snd so ctpl g component load so this component load will load the topology file for that particular component the argument it expects is the first is a component it may be a platform component may be a codec component depending on your point of view and then we give it our own topology ops so we look at the ops a bit later and then you give the firmware file which you want to load so what does this API do the topology component load API will start processing you have the file with bunch of things in it followed by the block so it will start to process the headers one by one and process the blocks based on the type it finds first check the valid header did we get bonkers or did we get garbage from the user space so we check we check for the size this helps to detect the magic ABI mismatches or wrong data then the magic number is the magic number right is the ABI version right so when we started we started with ABI one and as I said anything it was a team stable so till four it's all development versions and four onwards we are backward compatible so if you're running five and you have a binary version four it's supposed to work well actually it does and then once we had done we load that particular header once we loaded the header we will find the type of the header and type of the block which is following this particular header and we will invoke a particular function which is specific to the header element loads which we have in next slide once we have loaded all the elements that means our job is done so we mark it as complete if you remember in driver when we define controls or when we defined widgets we create a template for those widgets and only when the card is instantiated then only those widgets are created so similar concept is applied here in the complete if the card is instantiated we actually go and create those widgets if not we just keep the templates at that point of time and once everything is done we call the topology complete it's done so what do we do in the each element load type so in the element load type first is the K control element load this is called for mixer widget and byte control in this as I said we will typically call add K control this is something you can open code and your driver or call the micro which we have and then call sndsoc new for those particular controls if we have a graph type of widget that's the map we call sndtplg dapm graph element load where we will find all the elements in the graph and start to add those dapm routes by calling sndsoc dapm add routes then if we have a widget we will call dapm element widget load and in this case we will call dapm new control for the pcms we will call register the die and add those dylings based on the pcms type and as I said if you have a back end die we will not create the die for that because that's something done by your driver so we will find that die find and set the stream information and the die flags for that as you would have specified in the configuration file same thing goes for the dialing because dialing is again set by your hardware or your driver we specify the configuration so in the link element load what we do is find the die link and then set the hardware format and set the flags for this then in the manifest type we will just do the manifest load and tell driver this is the manifest details in the header we saw that there was a vendor load function and vendor type if at all it's a vendor load or vendor load type then in that case we don't do any handling in the kernel for that or in the core for that the driver is supposed to do it we can't implement it in our driver but if you want such a flexibility or if you have a different use case you can use this this is also the default handler in case driver kernel doesn't recognize the type of it and it will call the driver's load function so how does driver specify the various load functions hold on before that what are the ops topology ops the control has so when each of those widgets have been created or you are invoking TAPM to create widget or PCM or element load for you in that case we had the private data so how does the private data get communicated to the driver so for each of the object we can specify what is the object callback so object callback can be of type control widget die link manifest and complete so for each of the operation assuming you load a control in that case while you are creating that control you are in that particular control context after the creation of the control we will invoke drivers control callback so the driver we know this is the particular control getting created this is the context of this particular control and it can go and load up its private data or save or whatever it wants to do with that particular private data so this is the information for drivers so similar callback is given for widget die link manifest as well as as well as vendor so in the previous slide we saw that once done there is a topology complete callback so that topology complete callback which in turn will tell driver that okay I am done processing with whole topology this is the end of it so if you want to do some step at the end of it you can use that particular hook so how do the IO ops look like so as we discuss we have the control where we can have a standard set of IO ops and if you do not want to use the standard set of your get info functions you can use your own driver function to do the job so that is declared in driver as your topology control ops where you have the id field this id field has to match with the id you have given in the configuration file that is how the core matches this is the particular callback you can specify and you can actually do a mix and match you can specify a separate id for get you can specify a separate id for port you can specify a separate id for info typically its get put would be paired and info can be separate or you can have a standard set for info but you can use driver based values for get input sorry so this is all about topology ops lastly how does the manifest block look like we have been talking about it this is how the manifest looks like you have the sum of all the control elements in it you have some of the all the graph elements, PCM elements dialing elements, dial elements and the private data so typically we use it to allocate memory because we know we are going to get 50 control so lets allocate memory for 50 controls in the driver so that we can keep the driver context for it and in the private data we have a generic firmware specific information we need to load for our DSP so that we can specify in this case here which is not like a particular element type of information which is a global DSP information which you want to know so this way we make our driver more scalable and kind of little more agnostic and independent of the firmware in practice lot of things are still not so great because still you have a dependency on firmware but we try to abstract as much as possible using those values so this was a manifest block what is our future work so one of the things we have today is all the configuration files which we have upstream for our projects they are in the ALSA library which kind of gets updated once in six months so during the plumber's audio conference discussion we discussed that we should separate it out to a separate gith which is just a configuration file where all the UCM configuration and topology configurations are so that they can get frequently updated and picked up by distributions so that work has been started right now from the user space side we only support ALSA library we don't support we don't support tiny ALSA that is something which we need for the non PC use cases to be done so one of the things we have in the widgets was a particular index field so using index field right now everything is zero for us because we load all that at a single shot but what we have the flexibility is to specify different set of graphs using the different set of indexes so you can say I want to load index 1 now and I want to load index 2 for a different use case and index 3 for different use case so you can specify sub graphs not the complete graph in one shot but different type of sub graphs using those index values so this work is still to be done okay that's it from my side any questions that's just a binary file with the format we gave but I've seen that pretty in number one so that is basically for whatever you got the driver we have a default configuration which has a sample graph so that has been compiled into that particular binary file and that is what you are loading in your system so if you want to customize the graph or you don't want some parts of the graph which is probably not of use for us or you want to add more things for your graph what you can do is change the graph compile it and install it so as I said we have this request firmware so request firmware picks up the file from standard parts like live firmware so you would need to again go and install that particular file in your standard directories of your distribution so rather than starting with the binary file you need to go back and start what is the configuration file for that particular platform you are using right now it's in also lib right the Skyler configuration file is already there it will have the links but it will not have the controls defined so the whole graph we don't code it in the corner so without if you kind of just remove the dfw file it will fail to load the driver will fail to load not precisely so if you remove the file the driver will fail now you know okay good so that ui kind of thing yes we have been talking about it but then that will be a vendor specific ui because like for intel we will do our own stuff which understands our graph some other company may do their own so probably there won't be a standard ui associated with ulcer utils which is like for us it's a configuration file which is a standard but on top of that yes there can be a ui and I think we will have some ui in future available for people to use uis are nice right yes so the configuration may not be different a lot but then the private data where the secret source for the particular dsp lies in that is where we need a lot of values to be configured yes yes yeah Sunday patch yeah yeah ulcer live precisely yes yeah that is like what people are looking at that and runtime you define a particular section of the graph and then you say okay load this particular thing yes it's essentially for that purpose only yeah any more questions yes so fix up is nothing but your hardware configurations so if you remember let's go back to that slide where we specified the back end this is the link front end and next to next year you are here defining what your physical die configuration capabilities are so this is your fix a function implementation actually can remove that so right now in the upstream driver we don't actually have die support we have the all widget widget topology because we wanted the whole topology to be done first because that is where the most of the configurability is actually coming from the pcms where the like little later in the game it got upstreamed in only 4.8 ish time frame so pcms support in the driver right now is not there it will come eventually yeah right now we don't have pcms support pcms need to pcms need to be there in the machine driver for your particular machine yeah it's not supported today okay controls widgets dappum graph dappum everything is there apart from pcms okay so if you are familiar with a soft driver it's a simple stick codec driver so what you would do is you would have dappum widget section where you will define all your dappum widgets in a table and then you will have a map table you will have a control table and then when you initialize your particular codec you will register that particular codec with a core and in that you will give it all the values in the table so that is how those used to get loaded and that was the only way to load now we can remove that hard coding from the kernel and put it under the user space and customize it it is not useful for a lot of codec vendors because codecs are typically fixed function so it doesn't change but for codecs with dsp this thing will be useful in fact for one of our earlier like metfield code if merifield code if you look at in the upstream we have the whole graph coded in the kernel and it's difficult to customize and that's where we started looking at how we can customize it differently because I have whole of the graph added in the kernel using the same set of declaratives where we have defined app on widgets, we define controls, we define the map and then we load it always we can't change it so in this case right now we have Skylake shipping on lot of devices and people are doing their own customization for that any more questions? thank you guys