 Hello, welcome to my session about the year 2038 and what it means for U-TEMP, WTEMP on 64-bit systems, especially on Windows as you can see. My name is Thorsten Kuckuck, I'm the scientist engineer at JUSA, I'm leading the Q&A team. I'm now with the company Science for over 24 years and looking back at our over 25 years as a software developer in various areas starting from Q&A to U-TEMP and similarly allowing user-friendly stuff. So, in the year 2038 on Unix the 32-bit Tine-T counter with an overflow. If you never heard about this, then there is a really good Wikipedia article about this, but in short the Tine-T is a Unix counter starting at the January 4th 1970 and increased with every second since it's a 32-bit signed variable on most operating 32-bit operating systems with an overflow. And there is no universal solution in the C-language, any change to the definition of the time T-data type would reside in IPI breakage and code compatibility problems. Everybody who is currently working on changing a 32-bit U-TEMP system to use a 64-bit Tine-T knows what I'm speaking here. It's although not possible really to frame the signed in-tata to an unsigned one because then you will use the possibility to present a date before in the 1970s and the fault is similar to increasing it because you also have to look at the source code using this variable if it is okay how to use it. Increasing the size of time T to 64-bit, the existing system breaks the layout of structures and binary interfaces of functions. Again, the people currently working on 32-bit distributions switching to 64-bit time T now what I'm speaking about. So now 32,038 is far, far away and many people say that they will only support 64-bit architectures until then which have already 64-bit time T today, so why should they care? The answer is very simple, if you look at the header files of GDPC, there is this define for compatibility between a 32-bit and 64-bit user land at the same corner in the same system and this defines the seconds as a 32-bit int and not 64-bit or the system default. Which means even x8664 with GDPC on Linux is affected by this and U-Temp will stop working in 2038. If you look at the affected API by U-Temp or the double temp you will find a long list of functions and if we change the struct we have an IPI break it for all of them. Except the one where you set the part to the file that stay compatible. So two functions out of the list will stay compatible, the rest has an ABI issue. And the affected files are run U-Temp, VALOG, W-Temp, VALOG, LastLog, VALOG, B-Temp and maybe more who are using this struct. Now of course people think already signed some years about how to solve it. The one idea of course is to use 64-bit time t everywhere. So adjust the struct in VDPC to use 64-bit time t in the struct U-Temp always on 32-bit and 64-bit systems. But this is a big headache and it's in the same scope as the 32-bit link distributions going through who changed the time t bit. It's a massive API break it, since you cannot use simple versioning or for structs or variables like the problem is solved for the rest of the 32-bit time t transformation in VDPC. And the worst thing is there is now API for LastLog for example. You have a header file which defines the struct, no functions. So you would need to touch every single application to do it correct and work with it. Some people tried some proof of concept implementations how to solve it on VDPC level. But the developers decided that they don't want to change the current U-Temp interface anymore but they want to declare it as legacy and drop it in the feature. So that makes it pretty hard to come up with a solution on a low level. Additionally the current U-Temp implementation or design, it's a design problem more or less, has a security problem. You can make a denial of service attack so that root cannot lock in anymore and you cannot solve it on VDPC level but you need a demon for it. There is a bug report in the VDPC tracker for this one who want to know more details about this. Another idea is there is free space in the U-Temp struct in VDPC. The VDPC developer reserved it for future enhancements. Now one idea is can we not use that tool for and store the bigger value there. The first problem is VDPC developers don't want to touch the U-Temp code anymore but drop it. So at this moment already it's clear that it will not happen but there are also some big technical things, logical. And one of these problems is we have an application compiled with the old VDPC with the old struct U-Temp. It writes something to U-Temp then we have an application which is using the new struct already. It reads the old entry, it does not find its data. So it will not really solve the problem but will create a huge amount of problems because you have to change all applications at one time. You have to migrate all existing files and so on. So let's start with the easy files interfaces we have last log. Last log stores the information when a user did log in the last time. It's storing the username, the port from our TTY and the time. And it will be found at the next log in time as last log information. So as I said there is no API for this. There is only a struct and header file which means every application has to implement it at their own. It makes it pretty hard to convert it to something new because you have to touch every application. Additional, we have the problem that only log-in in SSH have real support for last log. Else, there is a PEM last log module with some distributors and some applications uses and configures other not. So on my desktop at home if I type last it claims I did log in the last time last year November. Even if I did log in at that day because somebody did not configure this display manufacturer to use the last log PEM module. So it's also not trustworthy so we need something completely new. Besides that the current last log format has a problem. It's using sparse files and with big UIDs you get really large files. And at the moment where the file system or tools don't support files with big holes and don't support them then you have a problem there. So we did implement last log 2. It's replacing the last log implementation by using an SQLite database to store the data. It's implemented as PEM module so you can configure it for every service where you want that it stores. You have the last log information and that it falls at the next log in time. You have it in one place code so it's easy to update, to adjust and to make sure that it is stored in a safe way, the cool way and always the same way. We have a last log compatible last log 2 command where we created a compact sibling tool. It can import the old last log file and the future is that the Utilinux maintainers want to merge last log 2 into Utilinux. So as base and drop their own last log implementation. Then open through the tumbleweed, micro s and I'll reuse it already signed several months. I got doing all the time only one bug report because somebody was missing one last log. Now that was the VTEMP. For this we didn't got any bug reports, nobody noticed it just working. That was a good thing. Now WTEMP, this file records all log ins, log outs, boots and shutdowns of a system in a historical way. They are beside this for entries, some other types which can be stored in the pre-system D times or maybe still today with some for init daemons. The run level is stored there but the system D it does not exist. That's the concept. Then it stores the new time field. When I did the analysis during the year who is using WTEMP and why I found one application which is reading this field but I didn't find any application writing that. So at just time who is reading this field will never find this in a VTEMP file in my opinion. And then there is all time where I only found several comments what is it who is using that but I couldn't find any usage or definition of it. And the file is written more or less by all applications performing a log in. So additional problem with the VTEMP format as we have today is that there is no common rule what I should write into this file and at which time. So if you take VTEMP as example the GNOME display manager it creates always two WTEMP entries for every log in and log out which is not a problem per se but confusing. Then you have like DEMP which is always using DevConsole as TTY. If you look ever at the last source code then you will see that last is using the log in name and the TTY to find the matching log out and log in entries. If you have now always the same TTY and if users log in on several X displays or something similar then with this last is no longer able to match log in and log out records and what last is displaying is wrong. It looks like that nobody really cares I didn't find a bug report but I could reproduce it that last is creating wrong output in this cases. And then some few terminal emulators write WTEMP entries others do not so you have inconsistent data. Who is using that file? Last is the most prominent and often used application to read this file and then there is LS log ins who is reading it and some few others but it's really rarely seldom to access this file. The solution here is also to use an SQLite database to store the data one entry for log in and log out. Since we have a database we have a unique identifier uniquely for this entry and can update the same entry. And we also wrote a PEM module to update and to have all code on one place like this last log and we have a last compatible command which forces the information. And this is where I got in all the months we are using it now one bug report because once option of last was not implemented but will follow the next days or weeks when I have time. So we integrated this already in OpenSUSE as the PEM module is installed by default and configured by default. The applications and their configuration are adjusted we have compact symbolings and until now as I said there was one last option which some user found out is missing else it looks like nobody noticed that we switched from VTEMP to VTEMPDB already on OpenSUSE. Now the next step will be to disable VTEMP support in all applications so that they don't write the file anymore. And now the most interesting one is the UTEMP file. The UTEMP file allows one to discover information about who is currently using the system. Again there are now common rules what you should write into an UTEMP file and there is quite a lot of discussions between different developers of different tools which ends in that for example XTerm and Consola write UTEMP entries. GNOME terminal does not write this entries. And if you call Uptime you will see that Uptime reports the users locked in on the system. But this is pretty meaningless because it doesn't tell you if you have 10 different users on the system or if there are only one user who is running nine XTerms and maybe 200 GNOME terminals. So it doesn't tell you anything this value. Also the API has no error reporting so if the UTEMP functions don't return a new entry then it does not mean there is none it only means for you nothing you don't know. And the design issues including the denial of service attack also exists for this. If you lock the UTEMP file you cannot lock in any more as root and you can lock it as normal user. Other problems is that user names and devices are limited to 32 characters. If you look at the header file for the kernel or glibc normally you have much bigger limits or no limit at all. Which means if you store a user name with more than for example 33 characters it will be truncated same for device names. And which is also a little bit a security risk is that the strings are not null terminated if they fit exactly in the variable in the length of the variable. So if you read such a field you have to look for the zero operator or that you read 32 characters. And I did read a lot of code to understand what applications do with the UTEMP file and the data it contains. And a huge amount of them have in the one or other place a problem and will read behind the string. Because they did one of the effects in some cases wrong. So when I brought up this problem the first time there was also a nice in this regard comment from Luca one of the system media that pass on Linux weekly news. And and thought we should stop limitating Linux. Because we use we should not look at the least common dominator Unix of 30 years ago nobody remembers anymore. Looking at the commercial Unix 30 years ago which has this limit happens before to remove them because Linux in reality don't have this limit happens and be put as I said not limit by us by the past. To now find a solution how to replace UTEMP at first I analyzed who uses this file and information. And one is the inner demon for process management. Not with system D so we don't need that. Then we have a huge amount of applications writing UTEMP entries for others. Log in SSHD, XDM, GDM, SCDM and all you can continue this list. They all only write the entry they never read it again. And then you have a few applications which read this file and use this data. It's not most like W who uptime wall write. QEMO, Samba are also some tools which read the information and display them. So it's a limited number of applications. And then I grouped them in common usages. Most of the application only use the UTEMP file to count the number of logged in users. So every graphical terminal is a new user in XTEMP. But if the graphical interface is using for example GNOME terminal which is not writing this file then it's not a new user. It's already as I said they are confusing. It makes these numbers not really reliable. Then you have tools for information about logged in users. It's who, it's W or the Samba and QEMO are using it for this case. And then you have the wall messages sent to our logged in users. For example if you shut down the system. And here it's local logged in on console users if you log in, remote logged in users SSH. And then you have the desktop users. So our GNOME user in the team tested it. He was using the GNOME terminal. He did not got a wall message. The KDE user got it. Even if you had no terminals because KDE creates a dummy UTEMP entry to get these messages. So in my opinion this is something the desktop developers need to solve in a better way. And not working with fake log ins or expect that every desktop user has an XTEM or a console open which is creating an UTEMP entry. So the solution for most of the problems is to create a new library with a demon. Such implementation exists already like UTEMPS. But this only solves the 32 bit time t problem and is it cruelty problems. It does not solve the API design problems. It does not solve which is now error handling the access of the struct. And you need to review all applications independent if it is a 32 bit or 64 bit applications that they handle the 34 bit time t correct. And it is another demon running on the system. But we have already demons on running on the system. Why should we create another one? And if we have to review and audit all code using the UTEMP struct and functions. Why not create something completely new without all the problems? Because writing secure new in a library or whatever is as expensive as reviewing and fixing all the code and searching outside there. And we have a demon which is already running on system D systems. And which is collecting all the data already sent several years and that is system D log in D. It is used on all system D based Linux systems for the non-system D one. There is for example E log in D which provides the same functionality. It is made for and this should be hopefully secure. It has also said it has all the data already collected via the PEM system D PEM module. And there are several interfaces to it. You have log in CTL, you have adipose interface and the system D. And with the system D signs the last release or 254. It also provides all functions you need to replace UTEMP. And this is looking then in this way the mapping. So for the UT type field only user process is supported so we don't need that. UT PID, we have the SD7 get leader function which providing exactly this PID. For the TTY we have the SD7 get TTY function. The TID is the session ID. The UT user is the SD7 get user name function provides it. The host is the SD7 get remote host function. UT exit and UT7 we don't have replacements in the system D. But I also haven't found any application using this two variables from this tract. And the log in time we get with SD7 get start time. There is a little key but in this because you can already start a system D7 before you log in. So this time is in this case not really correct but it's the best we have in this case because it's really a question if I start a 7 which time counts when I start the 7 or when the data is the user access attached to the session. So it's also the definition. And IP variable is then the UT host it's identical in this case. If you know the UTAMP ABA function it's in the end you call setUtent. You have a while loop about every entry you pass a struct you call endUtent and you have all information about the log in users. So if you use the system D the SD get session function is identical to the setUtent function more or less you initialize the data structures and you get a list of active sessions then you iterate over the sessions. You can fetch with the functions I mentioned on the previous slide all information you need and you can get much much more information which UTAMP does not provide. So the NULIP developers already use this and query also additional information from log in D and somehow encode it in their tools. For example if you use who from core hotels with this you will see additional output which UTAMP normally does not provide and in the end you free what you got you are done. If you look now at the common usages from the beginning for what UTAMP is used counting the number is really simple you just call SD get sessions with a NULP parameter and you get the number of logged in users. And since we have one instance log in D who controls what is a new session and what not this number is now really reliable. You exactly know how many different users are logged in on your system Informations about logged in users are found in who and we with this Compatible versions of W and who are and similar tools already exist and most all are already released. Other like Zumba and QEMU are currently work in progress. And for wall messages work for local and remote console users because log in D now who logged in it does not work currently desktop implementations. As I said this is something the desktop people need to solve. The display made for our Greta has a TTY it gets the messages they only need a find to provide it to the logged in users who are not using a terminal but maybe only a web browser or something similar. An analysis of the packages on the open to the tumbleweed installation brought up a lot of packages which are affected and need to be changed. Everything which is dark green on this list is already able to or is already using log in D instead of U-temp and there are already released verphins which means on open to the tumbleweed for example a micro s you just need to use them and they will use log in D and no longer U-temp in the background. So either the projects brought their own patches on support for it like the GNU-Lib developers that's why quality e-max and other GNU applications will support it latest if they update to the current GNU-Lib version or they accepted our patches like PEM, PROC-PS, system D, monitoring plug ins. Fedor also created their own application support. The light green one is this application which use U-temp but which we don't need to touch because it's enough to modify the PEM configuration and add PEM system D to it. The blue ones are the one we don't need anymore. We don't need a helper application who writes to U-temp because that's done via the PEM system D module. The orange applications are the ones where we have patches and which are already part of open to the tumbleweed but which are not yet upstream. Open SSH for example does not react at all to our mails and pull requests. Others, I didn't have the time yet to speak with U-temp maintainers and then of course we also have some maintainers who say 2038, it's far, far away. I don't care, go away. And the red ones are the ones where I'm asking if we really need it anymore. Act for example was the last patch six years ago but they are convertable if somebody has time and so then we have some applications writing U-temp entries like screen script, what should happen with them. So in my console opinion and that is what most people also agreed with me in the discussions in the last month is that Xterm and console and similar tools would not write U-temp entries like Nome Terminal. They would not fake entries so that they get the whole message but they need to be solved correctly in the desktops. And then we have cases like screen-wise U-temp entries but it's not a log in session, it's not a real session. Tmux does not write U-temp entries on all Linux distributions so it's a big mess there and Zudo is the same, Zudo writes something U-temp entries so does never. Discoffs with the upstream system D maintainer thought that they don't think that Zudo could create a new session and that's why they block it in their code. I can partly understand it and agree to it but yeah, that's something which is currently interesting. So what is the status of this in regard to open to the Tumbleweed and open to the Micro-S. So we have system D with 254 signs last week and with this we also release co-authors to Linux, proc.ps and many packages more who are now using log in D. So since one week open to the Tumbleweed and Micro-S are using log in D and no longer on U-temp for the most applications and I haven't seen any comment I announced on the Tumbleweed mailing list, no reaction, no bug reports so it seems like nobody is really noticing it that the informations are now coming from another data source because the output is the same. So what's missing is system D is using internal still U-temp for the warm-up messages. I fixed that two weeks ago so with system D 255 we can really build a system D without U-temp support at all. Coomu and Zumba they need to be converted. Zumba I have patches, I tested them, they are working, I only need to contact upstream. Coomu would be even easier than Zumba and then I hope that we in some months can disable writing U-temp entries at all and have a system which is in regard to U-temp, W-temp and last log here 2038 clean. There are some references. I wrote a block with a lot of background information and feedback from others. There is a Git repository for me where I collect some files with lists of all projects and what is done and what is still open to need to do, a document which describes how to convert from U-temp to log in D. I have also directory with patches for packages which are still work in progress or where I didn't have the time to submit them upstream and there is a repository with binary RPMs for Tumbleweed with all the RPMs with the patches but it means everything is in Tumbleweed that is not needed anymore. Are there no questions? I don't see anything in the publicum because it is so dark. Are there any questions about this? Any concerns? Other ideas? Oh, there is one question. I don't know if I have a microphone. I think I come here and then I repeat it. It is my better. That is the question was if we already tied it with a computer running in the year 2040 or something similar. Currently we have test cases which emulates this by calling all this code with effect time. We have not installed a real system because this is only about U-temp, V-temp and last log and it is not about all the other stuff which is still using 32-bit time T which means currently, no, until one or two months ago even our installer was not able to install on such a system. We have not been able to install on such a system. These are other teams that are working on it. I know Debian is working on replacing or switching from a 32-bit time T to a 64-bit time T on their 32-bit distribution. They are also working in progress and that is why I got so much help from the new developers because the Debian people complained to the new developers and that is why I got so much help from Debian. So it is currently everybody is doing single steps but we don't have a full thing which could work together. But this already solves some problems like security issues, no error reporting and similar things. More questions. Yes. That is a single missing feature I mentioned which is working in progress. Reinhardt Max, I think you know him, promised me to implement it but he didn't find the time for it yet. I think I have to look at it myself if I am back from all the conferences now. But it is on the to-do list. The question was the last comment has an argument that you could list the users, you could specify a user name and then you could get all the information that you need to list the users. This is what I mentioned is missing but we will come. Yes. Because I don't have 15 more years, I have a few months for this. The normal desktop lifetime is about four years so they have plenty of time to solve it. But if you look at the enterprise server market or embedded market or edge market, then you can create now the distributions which have a longer lifetime and will still be available and running in 2038 and later. And this is a change you don't want to do with a maintenance update, especially as we have these users, I think the others have the same customers who want a distribution where we guarantee that without update and modification it still runs after 2038. So it means, yes, in some use cases there is plenty of time, in other use cases we have no time anymore. That's why I started with it at all. Because we have the time preffer, we see them. So more questions. Yes. I didn't understand regarding what? The question was about BI systems and having applications, 64-bit and 32-bit applications running on the same system, how it will work or not for 32-bit applications. The answer is binaries as of today who are 64-bit, x8664 or 32-bit, x8664 will stop working 2038. There is no way to get them running there. So this, oh, I see it's, okay, it's going to jump in. So you have to adjust this application and if you use log in D, all what I told about 64-bit is also true and will work with 32-bit. All the interfaces of lib system, D, of last log 2, all of them are using int64. So the different, and not in seconds, but I think it's micro, whatever system D is using there, I don't remember exactly, I have to look up. So we are self in this regard with this on both architectures and of course, since they're all used in an int64 type, they're also compatible to each other. Yes? So the question was, if we make, disable the code in Vlipsy, will check tools, create a warning, will that fail or not? So today, Muslipsy, for example, has the function and is returning in all this, no data. With Vlipsy developers, plans the same. How this affects static code analyzers? If they can detect it, I don't know. I'm not in that business. I spoke with our compiler people and they said, there are some ways to detect it, but there are so many options with casting and accessing the virus and so on. They say, we tried it and we got more false positives than real positives. So yes, I think it is a nice search topic to write a static code analyzer who is going over the source code and tells you where you have a problem with 32-bit time T in the year 2038. I don't know if somebody is working already on that. I haven't got any feedback about this. More questions. Okay. Then I want to thank you that at least a few people tried to fill up this big room and thank you for listening.