 Next is JM, I don't know what your first name is, so JM Ortega Sorry Hi Hi So you're gonna talk about Python memory management 101 deeping in the garbage collector the Python garbage collector You have it. So you're a software engineer and you're focusing on new technology open-source security and testing Yes, where do you work? I am freelance I have experience working with consulting but in the last years I I have interesting in other roles like freelance working with specific projects University for example, research projects These things different different projects that we can find in normally in In companies or in consulting for example, all right, where you're dialing in from? I'm from Spain From from where in Spain? Yes, from Valencia Valencia, okay Excellent. Okay. Good. So I think it's all yours now Okay started. Yes. Yes. Thank you Welcome to my presentation. My name is Jose Manuel Ortega You come from Spain and in the last year I'm interested in in Python with focus in in security You can follow me on Twitter and LinkedIn where I usually share content with security ethical hacking Python and Contains devos and these things also Technical Reuters and in the last year I have fully some books related with with Python security and networking The first one is Python for networking and security that covers topics related with Modules and tools related with pain testing and detecting vulnerabilities in servers and web applications and in the and the last one And the second one is learning Python Working that covers talking topics like socket programming How to design servers and even driver and key to tools and also there is an introduction to web web applications With dango and flash plain words Well in this tall I will try to explain the memory internals of Python and discover a hobby harness memory management and object creation the idea is playing Hobbies are created and deleted in in Python and home garbage collector functions to automatically release memory when the object Taking the space is no longer in use. I will review the main mechanism for memory equation on how the garbage collector works in conjunction with the memory manager for For reference counting of the Python objects and finally I will comment The best practice for memory For memory management such as writing efficient code in Python scripts. I Start with some definitions Memory management is basically the process of especially Allocation the location and coordinating memory so that all the different processes run is mostly and can Automatically access different system resources in Python the memory The memory management is done automatically And it's an another language like CAC plus plus is managed by by the developer In Python dynamic memory management is an important part of programming language design and memory management Involves cleaning memory of objects that are no longer being accessed When you are working with with Python or order language like CC plus plus and you have different allocations for your code For example, local variables are located in the stack space. We can see that in the reading in the green in the green square Memory can be stored or on either a stack for starting memory A memory allocation or in a heap for dynamic memory allocation and this dynamic Memory is a heap that contains objects and other data structures used in the in the application Memory management in Python is handled by the Python memory manager and the memory allocated by The manager is located in the heavy space dedicated for Python This means that all Python objects stored in this heap are Privat and inaccessible by the developer and from the developer point of view Python provides some core API functions to work up on this private heavy space Additionally, Python has a built-in garbage collector collection to recycle You lose memory Allocated in this previous heavy space. The idea is that every function is a secure in on a stack memory And each object Reference is also created in the stack memory. However, object values are created in heap memory It is a heap allocation stored in a memory which is the memory that is used in the application at global scope and This global variables can be used outside of a specific Method and can be shared within multiple functions globally In this example under the main method An object with value 300 has been referenced With the name x and there is another function than the tools the square value in So in both functions the each object is referencing to the same memory location When the sqr and then when the sqr method is returning the reference object Sita is no longer in the scope therefore the reference cone of 90 90 hundred is reduced to zero and the return value is assigned to double variable and also reference to 9900 Later we will review. What is the reference counting? Well, we continue with Python objects in memory in Python is variable as an object Python objects it can either be simple like numbers or strings or containers like Dictionary digitalize and list furthermore Python is a dynamically TPA language with means that we don't need to declive ties for variables before using them For the developer point of view we can We can use the del extrusion for for the For delete an object and we delete an object and try to access it We get an error starting that the object is not defined The most important thing at this point is that del extrusion doesn't delete the object But rather dissociates Reference and delays obviously the name But doesn't delete the object the object is Is in memory? The is available in in the stack space only disappears the reference to to the heavy space We start analyzing mutability and immutability of the objects in this example We can see that an integer object is immutable science with an object issue is updating the reference is not changed In this other example We can see that at least object is mutable science in with an object is Updating the reference is changed in this example. We declare a one list with two variable objects and when change the lead When change the the original lease this will affect both variables due to lease lease are mutable objects In this other example where for example when we try to modify a lease automatically Updates the least that is reference in this example. We can see that copying a lease means copying reference because they are Mutable objects How can we how can this check if two variables are referencing the the same address memory? Well in this aspect we can use the ID Method the ID method retours the location or address where the object is stored and Basically in this in this example, we can see that objects with the same reference will return the same The same ID ID value In this example, we have two variables with the same value With the ID method, we can see that it retours the same memory location for both variables Now if we are seeing a reference to another variable We can see that the memory location is exactly The same we have the same memory The memory address and if we compare two variables with the each operator It will print it will return true the true value as both variables are referring referring to the same object if we Following the previous example if we create Another object and give another value you will get different Memory memory memory location if we assign a reference to this new variable which was a late assign No, we'll point to a different in memory memory location Regarding the objects comparison we can use the as we have seen the previous example We can use the use this operator which compares memory addresses The use operator compares memory addresses not values Not values that are in in memory in this example We are comparing two we are comparing two variables with the same value, but the comparing result is different in the first example In the in the sample in the in the left We have two variables that have the same address Memory address and in the right in the in the second example in the right We have two variables that have different addresses and Internally the each operator is using the idea of method for ways then this compare this comparison in Python we as we have seen before in the memory manager is Responsible for periodically running to clean out our located and the look and manage the memory and like see Java and all the programming language Python manage object by using reference counting this mean that the memory manager keeps track of the number of of reference To each object in the application and when an object reference Count drops to to zero which mean the object is no longer be used in the garbage collector Automatically freeze them the memory for from that particular object The user developer Don't need to worry about memory management in principle You can you can you can control but in principle you you have not to worry as the process of Allocation and your location is fully fully automatically reference what is referring counting basically is a Shun approach that Python use for storing object in memory and basically satanic in which objects are Located when there is a reference to them in the in the application Also a reference counting is used for gal vast collection as we will see later When in reference cone increase if we have a variable X for example in we have three cases when when this variable When reference cone increase for for for this variable when using an assignment operator In the second when pricing passing this variable as argument to to a function and The third the third case when reference coin is increased is when appending An object to to Alice in these three cases the reference cone of a variable is increased Python allows you to to expect the current reference cone of an object with the C's module keep in mind that passing Passing in the object to The guess right cone Method increase the rest the reference code by one basically every time a new variable a new variable points to the object And the reference cone is incremented in this example We can see that we can see how power to increase the reference code when assigning Alice object to another lease object By to automatically freeze object the object when is no longer required And at this point the question is how does Python know when the object is no longer required? Well, basically When there are no variables pointing to to to the object and In this case the object is not accessible and can be free And We have seen when the object when the reference cone is incremented and when There is the cremented Well, when the variable points to a different object or an object that not not a seed in these two cases The reference counter of a variable is is decremented in this example The reference counter for the lease is decremented When the lease object is a signal to to know me to the to the known a value And when the reference count arrives zero the memory used by the object is free In this example, we can see how Python increase the reference counting when calling the the function and Decrease the reference count when referring the the same value outside the function we have a Function where we are getting a reference reference count for the variable pass as an as an argument And we can see that before calling the function the value is 20 Inside the function has the 22 value because inside the function we We are using the variable. We are seeing We we we are passing this this variable in the in the function and when the tool outside In the function we get we return to to get the 22 value the same value before calling the the function What are the main advantages of using this in this approach? but basically is that it's very easy to to to implement for the developer and Obvious are immediately delayed when reference counter is zero, but there are Also disadvantages for for this for this approach is that is not free safe and also And the main disadvantages is that not has the capacity to detect a cyclic references We have cyclic reference we can have problems for for Free the Memory that that is using later. We will review the this aspect of cycling reference And the other disadvantages is that you need more space in memory science reference count Is a store for every for every object and you need And you need more more memory Well when the GC module with with the GC module that we have available in in the In the Python standard library you we can get The reference for a specific variable in this example we are creating a dictionary that contains a list And when getting reference for the internally we can see that appears the the list object Now we're going to talk about In the Python garbage collector that functions in two ways The reference counting that we have reviewed plus the generational g garbage collector Basically with the reference counting we have when the reference counter reaches zero then it proceeds with the leiton of the object But we is we have problem with cycling reference. We need to To use a other approach like for example using tracing GC What is a reference reference cycle? Well, what basically reference cycle or course when one or more objects are Referencing each other here to a samples when two objects are mutual reference And when Alice is itself reference And in in these two cases the reference count Is not possible to that that the reference cone arrives to zero and in these two cases Then the reference cone for this obvious is always at least one We have seen that when the reference cone draws to zero The Python interpreter automatically freezes the memory and DC and this classical reference cone is very effective But what happens when we have Referencing call in this example we can find a reference single cycle because one or more objects are referencing each other And in the the reference count and never reached Zero zero this code creates a reference cycle where the object list refers to itself and Hence the the memory for the object list will not be free automatically when when the function returns And the reference cycle problem Can be partly solved by reference counting. However, this referring reference cycle problem can be solved by change the behavior of the garbage collector In your Python application to do so for example, we can use the GC dot collect function of the GC module this method Basically try to solve problems with the cycle reference and Returns the the number of object it has colleague if it has collected and Allocated in the previous code as the reference count is at least one and can never reach zero We need for successful a garbage collector the objects by calling the this method we can compare Then the objects collected before and after calling this method before calling the easy collect We get a H Unrecovered objects and after calling this method We get zero Unrecovered objects and with this approach We can We can control better the use of the of the objects and the memory by the car bus collector Another interesting tool that we can use that we have we can We can we can use with Python is that provides a module called graph that lets you visually explore Python objects Grasp you can get objects and variables reference and view in a graph in a visual way The relations between them and with this tool for example, we can detect in I see ways our code has cycle reference and basically, we can see the state of your of your objects in in in a specific in a specific In a specific the bug or or execution a time Finally, I'm going to comment best practice for memory management Eh Well related with with the use of the easy collect method. We will cool the take exactly reference remember not to force garbage collector frequently the reason is that even After freeing the the memory the garbage collector takes time to to evaluate the the objects to be garbage collector taking up the the processor time resource Consuming consuming and the best practice at this point is using this method only when When you see on when you are analyzing that is strictly necessary for and for and for calling this the and before a calling this method Before calling this method, I recommend using the previous tool that we have seen for viewing the state of you of your objects in a specific in a specific time and see if your code can Can originate a cyclical reference from because the the the main problem The main problem that we can find is are the the cyclical reference if we don't have a reference a cyclical reference in prison ball week we can we have no problems but For complex applications that are calling functions each other Maybe it's it's it's most probably that we have cyclical reference. It's depending on the the the complexity of the application We can have problem or not Indeed in the official documentation where you can find the the methods supported by garbage collector interface Well in the documentation as well as explain how where is recommend Use specific method or not Knowing about memory Management Also has your right mode more efficient code. For example, we can use the the context manager for working with files And don't worry about closing files signs this approach automatically free Free resources Another best practice interesting for example, we are working with leases slicing leases slicing as fake affects memory allocation because it creates a new array separate from the original and and the best practice at this point is available Is using the this list function that is available in building Python functions Basically, the this list function assess three parameters the star is toba and the step and if we can and if we are we can avoid the The original leases slicing We can we can avoid for simple Memory leaks or or something or something of these problems Another practice is related with string concatenation When when possible avoid using plus symbol for string concatenation because strings are immutable and Every time you add them into the string a Python creates a new string and allocates it to a new address We are using a Lot of memory for for for for only a New string and it's taking of using this this approach. We can use the Forestry contact recognition is better using the join method that provides a parameter An iterable logic or also we can use the format method for for for for for for better For better the the memory management Why are we using for simple generators and heads first funtions that always to to create in tractor funtions the tractor funtions calls the special operator funtions and The the generator use yield to save which element we are on and if only that value moving to the next value only when the Only on the next iteration of the loop in this way we can save memory Since the generator only store the data needed for for each iteration Finally these are the reference you can find articles these are the articles and Oh and Python models and I have used for doing this presentation And that's all you can follow me in my YouTube channel. You can you can find all Other Other other tasks and components related with Python Security you are so interesting in for example in docker containers for example, so I have Tiles letting with this topic And that's all we have any any question You can you can do it? Thank you very much. Jose we unfortunately don't have time for taking questions. So there are three questions I see in the Q&A It would be good if you could go to the talk channel and then maybe you can answer them there And I would also like to ask the participants who had questions to ask them there again Okay, thank you very much. Let me just play your blouse