 So good evening everyone. Hello and give Andreas Tiller a warm welcome for His packaging both of the our science package here at that conf 18 and in shoe Hello everybody Thanks for coming well formally I had the DB and meet both and the DB and science both which is Which our teams which have a lot of applications Written in a lot of different languages and now I'm talking about something which is based only on one technology there are statistics program and We have Since a long time several our packages inside devian, but we have recently founded a team Which uses a common repository on Salsa and uses a common mailing list to the dust discuss common problems I'd like to introduce this team and I would like to show you some of the work the team has done to make Our packaging really easy so regarding the team We have here is kind of statistics about people Uploading our packages This is basically dirk edelbüttel is a main our package are who has done our since obviously since 2000 and Has a lot of packages He does not want to be member of the team for whatever reasons and the other people like me and Sebastian Dylan and Graham and so consider themselves members of the team and have uploaded a lot of packages Well, it looks quite impressive this graph what I did, but it is more or less everything Autogenerated so this is just firing up the program check that everything went right and signs a package and upload Because it became very easy to build our packages and we we needed to do it because of a transition so we need to touch every package and That's the reason of this really high upload for this year in 2018 so This is about the people who are discussing as I said we created the team in 2018 so we have no statistics about the years before Maybe I'm a little bit chatty where I've just contributed a little bit to the To the technique we we are using and explained a little bit and that you here see The active posters on the mailing list and you are kindly invited to join this list if you are interested in our packaging This is a developer mailing list which has Even less contributors so forget the last one that might be spam, but there are the people who are Contributing to the development of packages. This is also the FTP master So there is some some technical discussion here going on So if you want to have a contribute, it's better to use the user mailing list which is called deviant minus are at least deviant.org This is it statistics about the people who are Fixing bugs in our packages so every package can have a bug And so you also see the history of dirt little and the other people also to fix bugs which is in part You have a test suite for a package and for some reason it fails because it's might Need new dependencies which are not included or not yet packaged and then we try to fix bugs in our packages in general, I need to say that the R code itself is quite good tested from the from the C run team, so Usually you find here Real packaging bugs which we fix and the the R code is quite of a high quality These are the contributions to the Git repository of the R team My high level of contribution is directly connected to the uploads I did because Every upload has maybe five commits and if you multiply the number of my uploads It's it reflects this high number is as I said is basically auto-generated So don't be afraid that you need to do so much work. It's Not something you should be very astonished about so From my work in the deviant mid team. I found some team definition, which is not True in all teams, but I like to promote it anyway because I had the experience that when I had a problem in the middle of the night and Saturday night and wrote to the mailing list of deviant mid and Went went to sleep and I opened my laptop on the next morning Somebody else that really solves the problem and this is this is my dream And this is the good thing in deviant that The people are usually very helpful very kind and very competent and can solve problems You have and you have a hard time to deal with and so as for me a team is if you wake up in the morning and Realize that somebody else has solved your problem from yesterday So what about the R packaging? Recently We switched from CDBS to Depp Helper Who is familiar with deviant packaging, please rise your hand You are I know you are probably as well. You are too shy. I don't know. So we have some technique Thank you. We have some technique to build deviant packages, which is basically In the easiest case a package can be built with just using a template and it works and Our packages are usually that of that kind of easy packages So we use Depp Helper and everything works out of the box it provides a quite comfortable interface and The good thing about the DHR helper is that it generates Dependencies of packages automatically. So you don't need to care about dependent packages Since the our package usually have a description file and it includes all the information You need for the packaging and DHR is reading it and creates a working control file We have also an automatic testing which is Something it's called auto pkg tests which usually needs to be manually written and you can write some kind of Sophisticated tests but for the very basic thing in our it's just firing up our loading the module check if everything is Okay, and this can be done for every our package It's and the Depp Helper is doing it for every our package. So this program is done for some packages we have more manually written tests and If we have these tests and inside the test some Other dependencies are needed. They are added automatically to the recommendations of the package so a user who installs our packages our package can Immediately also run the tests which I think it's quite a good thing because he can on one hand test And on the other hand he has a working example for what the package should do so the user is finally enabled to Follow what we use to test for the package We have two very very useful helpers for instance. It's one is team age make are Which is used to create a new package so a package which is not existing in Debian then to go to the see run mirror see run is a central archive and with DHR you can create this package with one call and There is also DH update are if there is a new version of the our module and You have new dependencies or so DH update are is reading the meta information of the our package and Updates the control file automatically for you. So I said everything is automatic. So I cared for DH make are DH update are and then these strips are doing the work for me If somebody knows an our package which is not yet in Debian I do a live demo and finalize the package with some manual editing of the copyright file and maybe Adjusting the description formatting. So everything else is done automatically This DH mark are creates a Debian directory, which is really nearly ready and and this is Specifically interesting if they're not existing our package needs further dependencies, they are also created so you Recently some colleagues of mine asked me can you package this our package they had I needed to have 22 our packages because all of the dependencies and I finished it in one working day including the writing the ITP I Usually the Debian packaging you write an intent to package bug to say hello my friends my developer friends I want to do this and I prepared also something to Get all the information also for my packaging in some text. So Well, it was not a stressful day. It was a kind of boring work to Upload all these automatically generated stuff, but Well, it's one of my record 22 packages per day. It's not usual, but since all this stuff is existing I was able to do this. I have also a Script, which I call routing update I'm running these if if a new upstream version is available and I can show this because I have written also a script here is Here is Some text file which is auto generated twice per day Which lists all the our packages which are outdated. We have one Column which version this is a version inside a Debian and one column upstream This is what upstream has and for instance, we see that Akron DB blur, but whatever how it's pronounced is not up-to-date. And so I say and I can continue with my talk and then we check what happened and I've got a hint that there is another Talk here at the Debian conference about auto package. It's something like this, but this is really simple I'm keen on hearing this talk. Maybe I can enhance something this script calls use can use can is an Program which goes to the URL which is specified in the Debian watch file and checks if there are new updates or not and it downloads the upstream version and The routing update scripts and injects in new version into git do some other updates like for instance, we we have changed Infrastructure from Elliot to salsa. This is checked here And maybe there's new standards version and new the paper version everything's update and then the DH update R script is called which I mentioned before to check if there are new dependencies and In case you want to fix some bug in some R package And there is no new upstream version. You need to add minus 4th option, then it will also work But otherwise if there is no upstream version, it stops because there's no there is no Routing update to do. Let's check. It's currently building Or it has now downloaded the new upstream version and it's building the package we can continue and as I said the the packaging is automatically created and Now you need to file a bug report and this is some Text form you need to fill in but in principle the information that is needed for this bug Is all in the packaging and so if you call this ITP from Debian dear you get a text and you can copy this in your Mail program and don't need to fill in everything manually. So this is This link is a little bit smart. I have in the end. I have a link to all to this PDF And then you can click on this link. This is where I I'm watching the new Upstream versions. This is updates twice a day. So I'm I'm quite behind Upstream because it was one criticism of Duke Edelbriddle was that The Debian R team is not able to cope up with upstream Well for me, it's rather I'm I meant I just learned in the talk from the DPL that it's I'm the person who has a Largest amount of packets uploaded. I was not aware that that's that's really that much But I care about a lot of stuff. So they are packaged have not really first priority or others but with this technique I Checked this this text file in the morning. See are there some new upstream I fire up routine upstream do something else and once this is ready. I upload and so it's nearly no work anymore to update the packages This script is Created by some Call to UDD UDD is a ultimate Debian database which contains a lot of information and specifically for this case The information about new upstream versions and so if I see that something happened we I check and Look for new updates. So the package is now built It says that there's some documentation Not in user share doc where it should be Yes, it is some lintian information. Maybe I should care at some point in time but I'm Do not do it because Our users usually know where they can find their documentation. So it's not really relevant. I'm check what what was changed Changes here. This is how I checked it's out from from the git repository in salsa The the new upstream version was included then it got the new standards versions. It was probably formally 4.3 or so, I don't know I can check the diff and Finally, I uploaded to unstable or they haven't changed like entry and upload to unstable So nothing really happened with this package and then I can say Get push push it to the Debian mirror and upload the Debian package and it's done While I thought I can even upload now then you see even if I was talking I was able to do an upload Push also the text I find the package in the build Area and this one I I usually check the The changes the source file you see also see here These are the changes of the package the new upstream version standard version is this and I Sign and upload the package. This is now signed and uploaded and the next my next task would be this one if I want to continue these four packages can't be updated because we are lacking some New dependencies of the packages which I which are in the new queue They are uploaded to the new queue, but we need to wait for the ftp master. So you see Even if I'm talking I can update in our packages. It's very brain-dead easy if you have Solved your task to find a new package new our package. You can tell me which package is not yet in Debian I just do it No, not yet. So if there is anybody who knows some our package with not in Debian, please let me know Which one? profit, okay, there you see If not the way we can try Debian R we have the scripts So it's called prepare missing C round package and missing C one package and you say profit Let's see. I think the last line is missing somehow and I keep on talking and maybe it's doing something so we have some open issue because this in in Debian you can't you are not allowed to add binary Blobs into a package and usually Javascript which is compressed is considered not added So Some our packages contain Javascript We usually it's jQuery or so and then we remove it and link to the Javascript package But recently they found that even the HTML documentation Contains some JavaScript blocks which are not edible editable. I Created the script which just removes these are had HTML 5 Okay Yes, it's a different field, but to make me able to upload the package I Can't do it with a lintern error. I agree with you. I but yeah, but but my question is The documentation is auto-generated right we can auto-generate it afterwards, but I'm not I Have not written a single line line of our code. I don't know how it is done if we remove the doc how to Recreate this HTML doc When then we put it in the paper and this d r d h r helper and then it will be recreated this was this would be perfect Do you have some some idea how it is done? No Maybe if we find out this is the only open the issue I see in the developer that we remove some docs and We could recreate because it's auto-generated so we can do it as well, but I have no idea how so let's have a look How if profit was packaged or not? profit That way profit Oh Yeah, something is if if the package is doesn't exist It's a profit was neither found on C run or bioconductor. It's also checking in bioconductor and Maybe it's doing something now. Are there are some questions Meanwhile, yeah, please use a mic. So if I mean that the crown So as code quality is known to be extremely high I mean, I know comparing the seat on to the crown I mean that the one reason why crown is so strict is because you can actually install everything on every So that's nice. That means I mean you have automatized already everything Why don't you out to my ties everything can run it automatically upload every new package everything The only thing is signing you get the depth if if you have an update You get a new package and you just well once a day or whatever whenever you sign the stuff Well, there was an effort to do this automatically I think it stopped it was run by or Have the best memory for for names, but there was an effort which Was packaging everything but the only thing what they can't do is Creating the final debian copyright file because there might be some codes So the copyright file need to be manually audited and then it needs to be processed by a new and FTP master. This is a blocking. I have 40 packages in the new queue So if you want to really so I'm packaging at request If there is some need either by the packages themselves Needing new dependencies, then it will be packaged or somebody says please package then I do it and I think really in general. This is a this is a wider problem of of the of debian that the middle layer Dealing with the middle layer problems is not sourced at all we have are you can install every our package directly from ground Without yes, and I did this because I needed some packages and dependencies on all this And you always come into the hell that some packages there and update and the same is with Java script The same is with Java this James is Ruby all these kinds and this well, I mean in this sense you are you're right much further away from or much better in a much better situation than many other of the middle layer middleware stuff Languages, but still I mean this is not something you can solve and actually I Would wish to see more actually There was the question not only once should we do our packages at all? because This is a valid question, but my answer to this is you have Different user application who depend that depend from our packages And to to package these user applications you need to pack a package and this is kind of a network which is Yeah, so the answer for me is to To serve our users, right? It's necessary. So now it's this Has some package you can find the repository here I show you the package is created in a sub directory which is called zero and prospective Just the name I used and there you find other packages. I I Did this way before you see our current profit. Well, it was stupid. This does not exist But our current profit was packaged and it obviously needed an extra package extra distro Yes, and you see the debian Directory is created. It has even a test an autopick a gist. It has a change log file For the change log file you need to find an ITP back you need to file it manually Also something we can get rid of actually I hope right yes Asking for it. Yeah. Yes. I know but it's still stupid for this amount of packages. Why would you I? Agree, but but this way to get rid of the the boring work. I had these ITP here So you get you get all the information you just copy this one into into your email program And then you are ready with the exception of this is it you need to Maybe you need to fix the formatting here, but the everything is done for for the dependency and the same ball I can't profit. So this is basically what's what's done here. You have also the the copyright file The It's a license is filled in here's one fix me because I have no chance to find out the year when the copyright starts I am to be honest. I invent this year because It's hard to know when they started I look at the change log if it's a one point something I say it started 2012 to 2018 whatever Nobody really cares and this is all you need to do you have a rules file Which looks like this? This is a very simple rules file if maybe some Permissions need to be fixed manually with with an override, but this is basically done. You can go on and get build package Start with get build package here. It starts building so Some formatting some some year when the copyright start and then you are done This is what I wanted to present here For the copy we don't need anything. So I have this this presentation is online and If you seek for Andreas Taylor talks, you will you will find you will find it easily I forgot to put the link but you don't need to take a photo of this one This is I should have put it on the beginning where we can do put the discussing But but there's this document is empty now seek for Andreas Taylor talks and you find my talk page and Any questions? Yes They describe Updates to the latest standard version Well for the our packages. It's always it's it's match the standard version. Yeah Can you please so that there be a policy? Sometimes change. So there might be also changes that That might concern your packages So if you just upload without checking the what's changed between the standard version You might corrupt other person's machines, maybe Well, it's so far for the jury But the our package install all in one directory and if one package is wrong then all packages are wrong. So Lin-chan is checking the policy and Lin-chan is I check Lin-chan Always and I don't upload the package with a Lin-chan error And I I very very rarely upload with a lin-chan warning. So some lin-chan issues remain, but I'm quite strict. So I Just increase the policy. I use actually a CME. There's a pearl team has this config model editor And CME is included in in this DH update are and so it's it's quite safe to Blindly update update. It's just a theory what you it's true, but it's not a practical case. I The Intel Mkl library was recently uploaded and finally entered eb and what is the status with our support for the blasts? Don't know. I don't know. Okay Can't comment on this. So we just check It's it's building now the dependency. So it's it's it will build and then if you are interested I can really upload this Yeah So And finally somebody will might be happy or not As I said, I'm packaging our package at request. So this was not a not an honest request So I don't upload it, but if it's honest then It's an honest request. It's a good package about Prediction, I mean Arima models and all this kind of stuff prediction from from data. So it's a it's a good package So I hope by Facebook originally. Yeah, well, okay, then then we fill up the new queue with two other packages Did you think about Just running you a superscript on all the packages of ground No, because I have some other things to do But wouldn't that be an I mean that's the whole point I mean would be nice if all the packages are in deviant Well, why do you want to run my script and do this what you are asking me? Yeah, feel free and would be really happy. I have not written this script to do all the work But to make others a very smooth entrance to do this work themselves This is what's the point of the script. Yeah, but I agree with If I it there was this automatically generated deviant Packaging stuff which is perfectly possible and in theory to do this but I think there were some problem It was stopped overnight, but it's it's not my personal interest to to have everything of our But I have the interest to to support My target audience, which is medicine and so with all our package they need this is bio conductor these are some our packages and All its dependencies and it's kind of a dependency tree and I I'd simply volunteered to Care for all other packages that are in deviant science and so Because somebody needs to do this, but I cannot promise that I keep on Doing this all myself because it's so easy everybody can do it so Mike should I do this as long as very easy I can do this but yeah It would be very welcome if somebody else would join this anybody else I Had a very similar problem back 10 12 years ago when I started packaging take life and all the c-tun packages and there was the General Opinion on the deviant developers mailing list and all this kind I should not package each Single package they would just reject it because there are too many new packages. It seems this has changed, right? I mean because nobody complains that you are loading for a thousand packages, right? Nobody really complains. Well, the only valid complaint would be FTP master because they reject they didn't complain But they are I say not super motivated to do it right now But to I need to say well I do not want to blame the FTP mother if I had told them I Need this our package to fix this RC bug or so they are super fast and they do in it in 50 minutes But if it's a random our package, I have no motivation for them to do this It's it's in the queue as fast as the queue is processed and it's fair, but I If you go to I'll see and ask FTP master, please for this reason. I need this package. They are super fast All these and this is what Relevance, right? As I said other people say well, it's a question if it makes sense to to package So many are packages because you can also install it from our but Every user you use as a package has is in its own home directory and It's if this makes sense It depends Mighty user systems real mighty user systems are rare nowadays Well, I don't know what's the definition of real user system What an eight one active human user on the system? Oh No, well, I've seen it where it makes a difference if every user installs everything or if it's installed for the system So, okay, so no further question. We Don't need. Oh, yeah, welcome. Please takes a mic. Oh, okay Easy yes, it is what we just did now. It's the prophet is the one he asked for This is the one this is But thanks. Thanks for doing this So I will okay to use us was asking we're asking for Akron prophet and I will upload it and so that's fine. So we can You can stop here now. What do we have a question about our packaging? Then takes a mic Not so specific. So I think I will come to you back When we talk face-to-face, okay, like, you know a dinner But don't don't be shy at first and then my second answer is I will go swimming now and we'll be late at dinner But I'm here the whole week Awesome. So let's go swimming together. Yes, fine. Sure Let's go swimming to but I'm I'm swimming so I quite concentrated and I'm quite fast I'm a trained swimmer. So you will not be able to follow me and talk while swimming We can meet in the middle. We can meet in the middle. Okay. I've been training. Okay, so fun That's fine. So let's let's stop here and have fun and swimming. Okay. Thank you very much