みなさんこんにちは。私は、ひたちファンタラのマサキキムラです。私は、クルマネティースリースセットの使用プロテクションを話します。PV、PVC、スイグレット、Beyondです。では、始めましょう。まず、私は私を紹介します。私の名前は、マサキキムラです。私のギッハンドルは、M-キムラMです。私は、HITACHIファンタラのOSSデビューバーです。私は、クルマネティースコミュニティースリースで、3年以上経営しています。私は、ローブロッボリングフィーチャルとCSIフィーチャルを作っています。私は、クルマネティースクラステルと外側のコミュニケーションを作っています。クルマネティースをクルマネティースリースで、です。このセンションは、クルマネティースリースで、クルマネティースリースで、クルマネティースリースで コミュニケーションは、これがこの紹介について、私は、クルマネティースの  existing protection features新しいプロテクションのフィッシュを 今、私はプロプロジェクトしています。私はF8QとDAMLをご紹介します。そして、このプレゼンテーションを終わります。私は、実際にプロテクションのフィッシュを始めましょう。まず、実際にアルバックのフィッシュを 使っています。つまり、ロールベースのコントロールを使っています。このプレゼンテーションのフィッシュは、誰がリソーシスを使っているのか、そして、どう?3つのリソーシスを使っています。誰?ユーザー?グループ?サービスアクラント?2つのリソーシスを使っています。どのリソーシスを使っているのか、ロール?クラスターロール?ロールとクラスターロールの違いは、彼らは、彼らの名前の方か?ロールは、彼らの名前の方か?クラスターロールは、彼らの名前の方か?ロール・バイデン・クラスターロール・バイデンの中、コンセプトの方か?誰?誰?誰?誰?例え、ロールビディングが使用者のアーリースにアップデートやコンフィグマックスを取り出すことができます。クラスターロールビディングは、EPモニターコントローラーで、リストやウォッチ、サービスやポイントやポーズなどのネームスペースを取り出すことができます。リストやウォッチ、サービスやポイントやポーズなどのネームスペースを取り出すことができます。リストやウォッチ、サービスやポイントやポイントやポイントやポイントやポイントなどのネームスペースを取り出すことができます。リストやウォッチ、サービスやポイントなどのネームスペースを取り出すことができます。リストやウォッチ、サービスやポイントやポイントやポイントやポイントやポイントやポイントなどのネームスペースを取り出すことができます。同じリストやウォッチの歯処を取り出すことができます。リストやウォッチの歯処は厚くなります。そのため、リソースを取り出し、そのため、タイムを前に取り出し、タイムを前に取り出し、実際に取り出し、実際に取り出し、まずは、コントローラーに取り出し、実際に取り出し、例え、公式詳細の一時解釈に詳細はいてても、実際に取り出し、実際に取り出し、実際に取り出し、実際に取り出し、実際に取り出し、 operated backs並行した,たきたつから分からない、そのために使用スイ pullたちになると、PBCプロテクション、PBプロテクション、そしてガーベージグレクターを使うファイナレーザーです。次の問題について説明します。また、ファイナレーザーのブログを使う場合は、ファイナレーザーを使う場合は、ファイナレーザーを使う場合は、PBCプロテクションとPBプロテクションを説明します。PBCプロテクションは、PBCを使う場合はPVプロテクションを使う場合は、PVプロテクションはPVプロテクションを使う場合は、PBCプロテクションを使う場合は、この説明は、クルバレンテストレージモデルで、この説明は、足りないので、説明してみましょう。クルバレンテストレージモデルの説明は、PVプロテクションを使う場合は、PBCプロテクションを使う場合は、PVプロテクションを使う場合は、PBCプロテクションを使う場合は、PBCプロテクションと、ぐず어서PBCの....ことが在 other汝の汝を試みました。Tuberエンジンになるため、PBCプロテクションを使う場合は、4个を記録してみます。 waste generation つまりは止めた場合は、4個を記録してみます。PBC Protection and PV Protection is needed as for the implementation of these features, it is very simple, for PBC Protection, there is PBC Protection Controller and it has finalized on PBC creation and when duration timestamps becomes non-near as a result of duration,PBC Protection is used by any ports and there is a finalizer if not used.PBC Protection works almost the same way.Next feature is garbage collector.Garbage collector is a feature to delete descendant resources on ancestor resource duration.So it is a feature to delete, not a feature to protect from duration.To manage multiple ports, Kubernetes has concepts of deployment and replica set.Deployment manages a replica set and the replica set manages multiple ports.In this way, users can create and manage multiple ports via deployment instead of creating multiple ports manually.On duration, users don't need to delete ports one by one.They just need to delete the deployment and all the managed resources are automatically deleted.To achieve it, garbage collector is needed.To manage the relationships, garbage collector uses all the references field in each resources.The field has identities for the parent resources.For example, a port has a reference to its parent replica set and a replica set has a reference to its parent deployment resource.The basic behavior of garbage collector is it updates a graph structure of owner reference on every create, update, or delete event for all resources.And it deletes reference to resources when all references resources are deleted.For example, it deletes replica set on deployment duration and it deletes parts on replica set duration.Then, finalizer plays a role to achieve additional features to guarantee order of duration.For example, it is used to block duration of replica set to complete until all ports are deleted.If block owner duration is true and foreground cascade duration is specified,finalizer is added to parent resource and the completion of the duration of parent resource is blocked until all its parent resources are deleted.Next topic is new protection features.Before going further, there is a caution.Slides here after contain features that are still under discussion in the Kubernetes community.Therefore, specification and implementation are subject to change.The features might not be implemented in the worst case scenario.As you heard the explanation for the previous slides, you may think that Kubernetes already have enough features for protection.However, this slide is to show that it is not enough yet.It became clear by the issue in this slide.The issue was a failure in durating a volume with Azure file driver and it is found out to be a non-driver specific.Let me explain the issue first.As explained to consume a volume, port, pvc and pv are created in Kubernetes storage model.Actually, to create the actual background volume and delete it, pv uses provisional secret and the secret can be in the same namespace.In such a situation, what happens if the entire namespace was deleted?On the namespace deletion, Kubernetes has a namespace lifecycle controller that deletes all the resources in the namespace.Therefore, all the resources in the namespace are requested to be deleted.As explained, pvc and pv have a mechanism that prevents deletion while they are in use, but secret doesn't have such a feature.So, the secret is highly likely to be deleted before pv is deleted.Then, pv deletion will fail because deletion of the backend storage fails due to the lack of provisional secret.Therefore, secret also means protection.I started proposing secret protection.Notice that secret that were differences from CSIPVs can be configured as parameters in storage class.For example, by writing pvc.namespace in storage class, you can specify the provisional secret namespace to be the same namespace as the pvc.It is used to make the secret per namespace or per pvc not to be shared across the cluster.So, you might be able to avoid putting secret to a different namespace to avoid this issue, but it will conflict with this use case.Also, even if you put the secret to a different namespace, there will still be a risk to be deleted mistakenly.This slide explains the first proposal of secret protection.As you might guess, the word first proposal means that it will be changed later.Secret protection is a feature to block deletion of secret while it is used.It is proposed as a similar implementation to pv and pvc protection.A secret can be used from pot, pv or volume snapshot content.So, the newly introduced secret protection controller adds a finalizer on the secret creation.And when deletion timestamp becomes non-near, it checks whether the secret is used by any resources.And it deletes the finalizer if it is not used.Feedback from community isUnconditionally adding finalizer may affect existing environment because there are other use cases of secret.Also, there are many differences.Like pot, pv and volume snapshot content.So, aren't there any features to block deletion of resource x while resource y exist.So, I studied considering a generic protection feature or in-use protection.In-use protection is a feature to block deletion of resource while it is used.It is for all kind of resources, not just a specific resource like secret.And it is used by other consuming controllers like secret protection controller.As for the implementation, I proposed similar implementation to garbage collector.But again, as the slide title shows, it is the first proposal.So, it is changed later.The basic idea is that if you remove the deletion logic for garbage collector,the remaining logic for guarantee of the deletion order is exactly what we want for in-use protection.So, a newly introduced use-y differences field is added in metadata.And it is used to track the reverse difference of using resources.The basic behavior of controller side is it updates a graph structure of use-y differenceand every create, update or delete event for all resources.And it adds the finalizer when number of referencing resource becomes one or more.And it delete the finalizer when number of referencing resource becomes zero.The usage from consumer side is on creating resource, it adds a reference to the reference resource,to the referencing resource use-y difference field.For example, on creating persistent volume that uses a secret,it adds a reference to the secret,to the pb's use-y difference field.Feedback from community is it's too complex to update inverted graph structure of all resources for this use case.Updating all resources means that we need to update which resources to watch when new CRTs are registered.So, it is a very complex work that garbage collector is doing.Also,managing inverted graph means that we need to keep track of all the changes.Failure in tracking changes means that they will be inconsistent.Also,finalizer blocks duration to complete,but it doesn't block the duration to start.It actually says that finalizer has an issue when it is used for protection.And this becomes the biggest issue.So, let's go back to the finalizer slide and see what this issue is.This is the modified version of finalizer slide.As explained,finalizer is a feature to block resource duration to complete.And the issue was that it doesn't block the duration to start.This issue happens when multiple finalizer exist.And the purpose of the finalizer are both quick point for actual duration and to delay duration.If multiple finalizer exist,order of duration for each finalizer is undefined.So, duration of actual resource starts even if finalizer for delay duration remains.So, it may be inadequate for blocking duration.The idea for solving this issue is shared from Kubernetes community.The concept of VM that it discussed in old issue long ago will be a real solution for this issue.So, let me share the summary of the idea.The basic idea is to protect from erroneous duration.Duration request should be blocked because it is too late to block the duration after the duration starts.To block request,paradating webfork can be used.The predating webfork is a mechanism to check the request if it is valid and block the request if it is invalid.To decide whether to block,a slice of string should be used as it is done in finalizer.It is to allow multiple entities to set and unset the resource to be protectedAs for the name of the concept,the word VM might not be familiar to non-native speaker of English like me.From the American Heritage Dictionary, VM is defined as the right to hold another's property as security for a dead old.And the difference between finalizer and reen is that finalizer blocks duration of a request to complete.And reen is block a duration request of a resource.With the concept of reen,inuse protection becomes a feature to block duration request of resource while it is used.A newly introduced reen's field is added to metadata as a slice of string.The basic behavior of controller is that it blocks a duration request for a resource if the reen's field for the resource is not empty.For example,the duration request for the secret in the example is blocked by inuse protection controller because reen's field isn't empty.As for the usage from consumer side,it adds a reen to a referenced resource on creation of a referencing resource.For example,it adds a reen to a secret on creation of persistent volume that uses a secret.As a result of discussions,it is decided not to add identities of referencing resources in reen's field because the field may become too big by adding such information.For example,if we add identity of using resource like persistent volume my volume to reen's and the secret is used by hundreds or thousands of pvs,the reen's field becomes too big.So,each controller need to track referencing resources.With this version of inuse protection,secret protection becomes a feature to block duration request of secret while it is used.The basic behavior of the secret protection controller is that it updates a graph structure on every create, update, or delete event for pot,pv, or vst.It adds a reen when number of references resources becomes more than or equal to one.And it delete the reen when number of referencing resource becomes zero.Secret protection controller only adds reen to secret when it needs to be protected.Then,the inuse protection controller does all the rest of works to block the duration.Alright,let's turn to demo.The demo mainly consists of two parts.First one is the behavior of inuse protection itself.And the second one is the behavior of secret protection.The second one has two parts.Protecting a secret while used by pot and protecting a secret while used by a persistent boring.The first demo is about inuse protection.Let's manually add reen's and see how it works.Any kind of resource works in the same way,but let's use secret here.I created a secret and there is no reen's field.So we can remove this secret.Okay,so let's repeat the same command again and add reen's and see how it works.Created a secret and there is no reen's.So let's manually add reen's by patching the secret.If we get the secret again,you can see reen's field is added.And let's try to delete this secret.Then the duration request is blocked by inuse protection controller.And let's remove the reen's and check that we can delete the secret.It still has reen's,so let's patch the secret to remove the reen's.Then there is no reen's anymore.So we should be able to delete this secret now.And we can delete the secret.Next demo is about secret protection.Secret protection is implemented as an external controller.So we need to deploy the secret protection controller first.I deployed the artwork and then controller.So the controller should be running in curve system namespace now.So let's check it.Here you can see that secret protection controller is running.So let's create the secret and annotate the secret to obtain the feature.Then you can see that the secret with this annotation.Just adding annotation doesn't block the duration of the secret.So the secret can be deleted.Okay, let's repeat the steps to create the secret and annotate it.To make secret protection controller to add reen's to this secret,we need to deploy a part that consumes this secret.In the official kubernetes website,there is an example port yaml that uses the secret.So let's use it to deploy the port and check that the port is actually using the secret.And you can see that the secret is mounted on the port.And the test secret is used by the port.So the secret protection controller confirms that the secret is used by the port.So the reen's are added by the secret protection controller.So we should not be able to delete the secret now.Yes, it is blocked by the in-use protection controller.There are two ways to remove the reen's.The first one is to remove the annotation to opt out the protection feature.So let's try it first.It still has reen's and annotations.And once we remove the reen's annotation,then the reen's are also deleted by the secret protection controller.So let's re-add the annotation and confirm that the reen is added by the secret protection controller again.And the second way to remove the reen is to delete the consuming port.So let's delete the port and check that the reen's on the secret is removed too.Yes, now we still have annotations but the reen's are deleted.So now we can delete the secret now.The last demo is the case the secret is used by pd.Let's create a secret with the annotation.To make pv use the secret,we need to prepare a storage class that is configured to use a specific secret.In the storage class,it is configured to use the test secret in pvc's namespace.Then let's create pv by creating pvc.We can see that the pvc and the pv for the pvc is created.And the pv is using the test secret.Then the secret protection controller should see that the secret is used and the reen's is added.So we can delete the secret now.Then let's delete the pv by deleting the pvc.Now you can see that the pvc and pv is no longer exist.And the reen's is removed.And we can delete the secret.That's it.Then let me share the answers to the question that I think that would be frequently asked.The first question is if there are any workarounds before reen is available.And the answer is that please decide the best workaround depending on your requirements.If you can wait for reen to be available, please use reen after it becomes available.Please note that this feature is still under discussion and won't become stable until Kubernetes 1.27.If you can't wait, the next question is does startup deletion matter?If no, you can still use finalizer instead.And even if yes, you can still implement your own admission well work to block deletion by yourself.For both finalizer case and admission well work case, you may need to consider reimplementing with reen after this feature become available.So you will need to decide with reimplementing cost in mind.The second question is whether there are any other use cases for reen other than secret protection.The answer is that I'm interested in applying this feature to only allow deletion via parent resource.The background of this issue is that some operators cause issues when child resources managed by CRDs are directly deleted by users instead of deleted from the operators from the parent resource.These issues happen due to the lack of the deletion order guarantee to solve this problem.We can use this feature to disallow directly deleting child resources by adding reends when deletion timestamp of all parent resources are not empty.Owner references field can be used to track the relationship that the child resource is still used by the parent resource.To conclude existing and new features to protect Kubernetes resources from error solution are explained.As for existing features, our bug can restrict who can delete.However,granted user can still erroneously delete.Finalizer can block deletion to complete.However,actual deletion may start.PV protection,PVPC protection,and garbage collector use finalizer to block deletion.As for new features,bin blocks deletion request itself.bin can be used not only by controllers like secret protection but also by users manually.The feature is still under discussion and can be used in Kubernetes 1.23.I'm targeting alpha in kubernetes 1.24.As for workaround and new features can be used.You can use finalizer if start of deletion doesn't matter or you can implement admission web work if start of deletion matters and can't wait reends to be available.Thank you very much for your attention.