 Hi everybody welcome to the multilingual content in the eighth session a highly evolved permutated API So my name, can you hear me? My name is Francesco Placela. My ideal name is Plach and I'm from Venice, Italy And I work as senior performance engineer at Taguan Constantin. I've been working on Drupal staff since 2006 and at the moment I'm the maintainer of the core language system and the core translation module and also the country by entity translation and title models You can see below my Twitter handle Watch out it has two underscores at the end. Not sure you can see them So a brief outline of what we will cover during this session We will start with a brief introduction a brief history if you want on the Drupal 7 API because it has some it has quite some drawbacks and it's needed to Explain how we improved it then we will turn to an overview at what we done in the eight On the entity API level because that was needed to work on multilingual content Then we will have a brief look to the content translation UI Gabor is Presenting a session a lap where he is showcasing all the multilingual goodies we implemented but I will give you just a as mold preview Since I've been working on that for a long time Then we will see All the entity storage and querying stuff that we've been building to support all multilingual content needs and Finally, we will have a look to the how How it's built the new entity translation API because we have one Finally, and then we will summarize what is left since hey, we're just in beta Okay, let's start with some With a look on the current D7 stuff we have Okay But most of you if if you are here are familiar with the first line, which is which shows how looks like the typical field data structure on an entity and You can see the usual field name the infamous lung code level and then the usual Delta and Collins levels This one is the piece that is causing troubles as you all know Because it's hard to predict What we should be put here to access the proper data? in fact a field may be translatable or not and so this means that that language long code variable may be an actual language long code or The language the online code or better the language non constant In the case the field is not translatable. This means we have to check every time for field translatability as you will know Then what we implemented in the field API we to deal with language we we actually Took two different approaches depending on the context when we have to Work with storage We act on all languages So we are able to load and store all the available translation field translation at once instead when Working with forms or rendering entities. We are dealing with a language at the time All these architecture provides a good consistency because Exactly as cardinality and Delta values Language is always there and is predictable. The data structure is predictable, but as we will We well know it provides a very bad DX for the reason I just explained So you probably well know what I'm talking about Then we have the entity language API in this heaven or Or Something like that Actually, we have an incomplete entity API in core. In fact, most of it is in contrib the the small pieces we have in core include the entity language function which As the name tells has the role to return the Language of the entity and then we have a couple of function useful functions in the entity API module The when you use the entity metadata wrapper, I hope you know what I'm talking about It's it's just like how many do know what the entity metadata wrapper is Okay, many of you are wonderful. So yet I don't have to explain which would be difficult So when you want to To work with language you have to with translation You have to specify the language you want to access Before and you have to call that language method Or you can call the get translation method on an entity object directly and both of those methods are Provided by a contrib module at like to at line Then we have the entity translation module that instead provides three hooks related to crowd operations that are fired when Translation are inserted updated and deleted So as you can see, this is hardly an API It's spread in three places and it has very inconsistent naming patterns and dx This is a big trouble for all the people needing to Write multilingual code in this seven Our field our content translation models in this seven Uh are the following we have actually two competing approaches because we have no translation in core That lets you create a different node And lets you assign a language to each one and basically you you can Put them together in a single translation set and then we have At api level the field translation So you you are able to as we see as we saw we are able to Provide a different translation for each field if field are translatable and we have just a core api but The ui is provided by the entity translation module So in the eight We we decided that two Two approaches were too much actually The main reason for not having two approaches is that Uh yeah set builders have to choose one and We don't want to confuse people and make them ask questions if they don't strictly need to Developers have to support both because most of the time The code needed to support one or the another is different And everyone has to understand both. So these all of these implies an increase cognitive burden and And also perative and maintenance burden because you have to write more code and moral Above all you have to maintain more code So after a very very long discussion huge debate during the Very early phases of the duplicate multilingual initiative Uh, we came up with an uh solution that is an unified model Actually, what we want is translate every piece of content applied applied sorry to an entity so um, basically we are merging the two approaches actually we are A field for each piece of data attached to an entity And but we have a single entity for each translation sets. So actually we are able to vary each single Piece of data attached to an entity per language Yes, one thing I forgot before Going to the next slide One of our main contributor hosereiro is not exactly a fund of this solution. So he was Um Proposing to revive it in contrib the to revive the old solution in contrib and we tried hard to make that possible So he will probably experiment with a translation model providing The old behavior because actually there might be use cases where the old behavior is better fit We will cover this in a bit more detail later So what's new in the drupal 8 entity api? Sorry, I can I cannot fight the right So, uh, what's new in the the 8 entity api? We have classed objects And that allows us to encapsulate all the data and all the processing logic of data We have interfaces and swappable implementations with which which means that actually we are able to provide a different class For the node entity for instance, and this can be useful And the interface on top of that allows us to easily swap it We have more swappable handlers what we use to call controllers in drupal 7. So basically we are Different handlers for storage, which is the only one available In drupal 7. Sorry, we have handlers for form so basically We have A handler responsible for generating entity forms and then single entity types are able to extend this handler and provide Customization just for their use cases We have a see with the similar approach. We have translation handler Which is in charge of altering the entity form and providing all the ui for translation the render Handler, which is responsible in the very same way to Provide a renderable renderable array for the entity and and that can be customized for every entity type And then we have other handlers like the access control handler, which behaves exactly the same way These all of these are very useful because allow to generalize all the common logic And just provide Very small customization for single entity types And this is very powerful because allows us to unify most of the entity of the various entity behaviors Then we have Maybe the most important piece Which is the entity field api basically This is an extended and improved version on the of the entity property api we had in d7 And it allows to provide definitions for each field that is attached to an entity And is the only way actually to define fields both It's a unified way to deal with fields And this means that what in Drupal 7 we were used to call properties like the node title are now fields And what in Drupal 7s we were used to call fields are fields again So everything is a field just what is provided by the field api are configurable fields That just means that the definitions are provided based on the configuration The the user enters through the field ui This is very important because it can provide A wave to the entity system to know A lot of things about every single piece of data it handles For instance We have that all field definitions are based on the tapered data api, which is another lower level and allows to provide data for Sorry, allows to Know which native data are assigned to each field Which is very powerful to build stuff like rooms or something like that And it works with any piece of data attached to any entity So now a small demo of what we build the content translation ui a small interaction the content translation ui basically with content with mean content entity that means that Any entity extended the content content entity interface is supported And we will see in a moment how many content entities we are able to translate with it just in core It exploits the form handlers concept I was referring to before and it Has column granularity when we talk about translatability, which means that we can define What elements are translatable even at a even lower level than field because we are able for instance in an image to tell whether the file Properties should be translatable or not or whether the alt or title properties are translatable or not And yes, let's have a look So as I was mentioning this is our content Translation configuration page So basically what you can see listed there is the the list of all the content entity types that we support in core And we have content The usual nodes and when we click on that we have another Way to Configure in just a couple of clicks the translatability before all field all available fields And as I was mentioning we can say that the file for instance is share among all translations But all the title are translatable See As you may have noticed we have taxonomy terms here Custom menu links here custom blocks here comments and users which means that any of these entity types is fully translatable now in core applause, please So let's have a look How this works Sorry as most green we created an article for instance And here is the usual Translation on review page. We can provide another translation as usual Here we have some options for the node. We don't have very much because most is covered already And here it is and then finally We create the third language and we have the usual source language select Here it is and we can create another yet another translation. So yes Here's the translation status. It's our language and then we can delete A translation if you don't want to be there And this works the very same way with for instance taxonomy terms The name is completely random Sorry, it's a bit difficult from this distance And here it is we have the very same UI for terms So I won't spend much time on this Gabor is doing a fantastic work is showcasing this stuff. But yeah, I wanted to show it just a bit And here it is Sorry forgot to enable the language switcher. I think it's not showing it Here it is Okay, let's go on Oh, sorry Just a suggestion So let's talk about entity storage What makes everything possible is the ability of storing Multilingual values. It may be it may sound obvious, but it wasn't Exactly easy to achieve So how we decided to to achieve this Basically in d7 we have just one single solution Which is the title module that allows to replace somehow we're doing very funky stuff somehow replacing entity labels and Relying on field translation Allows to more or less translate The full entity, but there are all the other base properties that are out of luck And we are able to Achieve at least this result because Drupal 7 fields have native native multilingual storage So once we decided that this was the way we wanted to To go through we it was obvious that we had to provide native multilingual storage or also for base field and the The right moment to implement this stuff came when we Spoke with the entity api folks that told us that they wanted to Provide a really cool feature in d8, which was storage and agnostic entities, which means that We are able to swap entirely the storage For the whole entity while Whereas in Drupal 7 we are able to do that just for single fields And since we to achieve that we are forced to build our code Without making assumption assumptions on the storage We are also able to make no assumptions on the SQL storage or the table layouts we are adopted. So we basically were allowed to revamp how the Default core table layout is structured so that it supports multilingual natively Sorry, I went a bit ahead. Here it is So this approach Implies That we need to load entities every time we want to access their data And when we want to query entities, we don't go through Directly assessing the database because this wouldn't work as soon as we swap unosql storage, for instance And instead we use the entity query. We should be familiar You should be familiar with the entity query concept because it's already there in d7. It's called entity field query and chicks provided and buoyants provided a revamped version the to version the version 2 Which is called just entity query and allow it's way more powerful and expressive and allows to write Many queries that are comparable with the the sql ones because it supports More or less joints and aggregations when I say it supports joints I mean that it has a syntax to express relations between entity types So basically you are able the The sql implementation the sql backend of the entity query system Is able to translate these relationships into actual joints And for instance a mongo storage for two completely different things But the the point is is we are not forced to Skip joints now because entity query the entity query sql backend allows for them And The the the very important point is that the entity sql backend entity query sql backend We have in core supports all the multi all the multilingual stuff we baked into storage and this is Actually what allows us to work transparently with any table layout actually So let's let's have a look to an example of how querying language querying works Can you see even you are at the bottom the code or is it too small? Is it fine Okay, so Basically the entity query API make makes no assumption on the language conditions so when you When you perform this first query You you basically are getting simply all the The notes that are published and promoted and this is true If for any translation, okay So this returns any node that has at least At one published or promote and promoted sorry translation As you can see the syntax is very similar to the btng and the dx is very is very similar So we had big improvements on this side So writing entity queries is much more similar and much much easier than in d7 Another example These language condition is explicit and allows you to retrieve old nodes that has an english Promoted translation as you can see and there's also a way to say I don't know which is the original language of the entity but but I want my condition to apply to that language Which is default line code This way we are retrieving all the nodes with the promote original values, okay So Some more details on how the course equals storage works basically What we now call bundle fields, which are fields that are somehow optional in the sense that they can be Attached to some bundles, but not some others Uh Are still still have per field tables, which is The usual usual field tables we are used to see in drupal 7 And they still have obviously multilingual support what we have now is uh four different table layouts dependent on the Properties of the entity type Which what does this mean? Uh basically An entity type can be Revisionable or not and can be translatable or not and this provides us four combinations So we have four different table layouts Possible we have a base table. We have a revision table. We have a base field data table Which means that basically all the information field information is stored in this data table Which holds the last revision And a field revision data table Which instead holds all the available revision you can think to the last two Tables like Merging all the single field data tables, okay, that's the same relationship they were Modeled on the concept of Field table field data table and for revision table, but they they can store multiple field values That's the only difference. So these are the full implemented the four provided tables and You can have different combinations depending on the properties. So a basic entity type would have only the base table Like I don't know the taxonomy term in drupal 7 Revisionable entity type might have only the base and revision tables like I don't know the commerce order entity type in drupal 7 Translatable entity type not revisionable would get only the base field data table or data table Where we would store all the translation And instead the the the base table would hold only the very basic stuff like uuid and bundle and stuff like that And Instead notes that in core in drupal 8 core are both revisionable and translatable get the more complex table layout Which has all the four tables and actually What we have is that The base table as I told you holds Only uuid and minimal stuff is basically used to allocate the Entity id revision table the same holds just most mostly only the revision metadata and The revision id and all the actual field data is Stored in the field data table And in the field revision table which is Very cool because we we are actually have Sorry, I didn't tell you one thing also on translatable field are stored here. So actually we can use the data table as At the denormalization table and perform querying only on this table We're at we're very seldom need to join the other ones when when querying, okay? So Even if we have four tables, we are actually able to To query just one single table Nonetheless these new table layout obviously imply A performance penalty for monolingual site which doing don't need such as such complexity Because when querying we we might anyway need some additional joins let's say when you want when we want to query on the uid and We have composite primary keys Which make queries load where and then on saving obviously we have more records to store. So this is This is lower We try hard to mitigate this factor and we came up with a really cool solution. I think because Basically what we came up with is generating the schema Based on the anti-type properties. So actually you don't need to provide your anti-type Schema in your install file anymore Or the the entity storage takes care of that Which makes sense if you think about it because a mongo storage doesn't need the SQL table And actually you just need to provide the definition for your entity type and the storage will create tables for you With the proper table layouts. So you don't have to Understand the very little details. You just define your entity type and it works And we recently also implemented A system that is capable of updating this schema based on changes on entity type and field definitions And I will show you An example in a moment This provides also An improvement a performance improvement improvement over d7 because We are able to skip field revision tables when the entity type is not revisionable And this saves a lot of Stores saving Records are not stored when they are not needed. Let me show you Let me show you another example So I basically Wrote a very small module to just perform a couple of alterations on the entity type and entity and entity field definitions It's just an example module It does very simple things Basically what we want to show it actually I was using to try other stuff. So I will uncomment just a couple of rows So what I'm doing here I'm adding a new field definition to know to the node entity type And I'm changing the cardinality of the node title from single value to multiple values So let me show you how the database look like right now before applying these changes This is sorry This is our regular base table as you can see it holds almost nothing And this is our data table I will probably need to delete this node because changes schema changes are not allowed when data is already created as for field data We will talk about that later So let's delete that So as you can see here, we have we have our usual regular title column Now I'm going to clear caches. So the system knows something has changed Let's change it. Okay, and then visit status report It's now telling me that we have updates because the scheme is up today is up to date We perform these updates and it's telling we have to create the text field and we have to update the title field We apply these changes We go online I wouldn't go online in this shape actually, but You will see why in a moment And this is what happens No title. It's a separate table Your smart audience you're starting to Know how it works actually add the uploads slide right next, but you anticipated me Okay, um So let me show you that this just works It's funny actually Here it is Can you see it? We have two titles now. So basically this is just a way to show you that With this system we are able to switch from I don't know Non-translatable node type Sorry Let's imagine and know that the node entity type and the site where you are pretty sure Multilingual will never be enabled You can just in a one line of code alter the entity type definition Strip out the translatable beat or set it to false Random dates and you get the regular storage you you are used to in triple seven and no Penalty happens anymore if you want Okay, let's go on So a brief note about views a brief note about views actually Views is more or less working with all of these although it's not able yet to Adapt to the changes in the table layout. We are working on that. It shouldn't be too hard But because right now we have these new entity data views Sorry entity views data handler, which is able to expose all views data Automatically for any entity type All the base defining all the base code and all the base definitions are implemented for any entity type And then can be customized as usual for specific entity types and actually These allow these provides the foundations for Letting views Adapt to any change in the table layout, but we are not there yet. We still We are listing when Entity type is translatable the records in the data table as I was mentioning before and these may cause some duplication issues because All the translations are listed. So you you need to have a filter on the translation language which Basically provides you only the translations for the current language. Let's say it's it's rather easy But we need to adapt. We still need to update the default views provided with core And we still need to implement some handlers to make everything work smoothly But we are working on that. So don't be scared if you see views not working properly. We won't delay. Well right now We have a meta issue. Thanks to jennifer hogdon Which is doing a great work with that and you can help her during the sprint if you want to Okay, let's Turn to the api side a brief note on language assignment and we implemented all A system that allows us to define Some configuration to provide the default language that should be assigned to entities when creating them and these configuration is used to determine the default selected in the entity forms And it's also used to Provide a default when entities are created programmatically. So this is consistent once Once we are able to implement also the language we jet Which are working on we are actually able to Unify all the logic Of language assignment for any entity type, which is very cool because it can be a bit tricky And no one has to worry about that. You just have to define A field a language field and then everything will will just work because the form will be we'll know how to handle it Just a quick note about that So the entity translation api, this is the last thing I wanted to talk about In as we saw in Drupal 7 we have no nothing like an entity translation api We have just a bunch of functions and hooks and so on but They do not work as we need actually, we came up with I think a very good solution that improves the DX of working with multilingual entities very much in Drupal 8 The main idea was provided by by krel and a big A big credit goes to him Basically, as you can see the in the code snippet what we are doing is A retrieving and a translation object, which is exactly A clone as of the original entity object, but with a different internal language Which means that we are able to Access fields in a different language The language is we specified As you can see the first line retrieves The the original value Then we get the translation object and then The same exactly the same code provides a translated value, which is very nice because The the api Transparenly hand lends and those field translatability you don't have to worry whether the field is translatable or not You don't have lancode level to specify all is internal. It's still there, but is internal and you don't have to worry about that And we also implemented native hooks and We have unified and streamlined all the Existing api as we can as we will see in a moment And you can find more information about this at that URL, which is the change notice about the this stuff It provides some examples And covers maybe covers and something we will Not talk about right now How do we access Field data actually as you can see in the seven We used to perform all these funky stuff because we needed the field definition We needed to Check whether the the field was translatable or not And then finally after determining which is the the active language. I mean the language that we are currently working on and We we were able to access a field value. This is really Not viable In the eight we have a way simpler code as you as you can see We just get the translation corresponding to the active language and access the value and that's it And one note about active language the active language Might not exist. We might not have a translation for that language. So we implemented another System to deal with that Okay, another example of this stuff. I will talk about the active language. I will help to determine the active language in a moment And and a few more words on this concept actually As you can see that this was the problematic code because you had to check the translatability of a field And basically you have to determine the active language if no language was provided, okay This was a typical pattern you defaulted to the current content language And then you were finally able to do the stuff you had to do with that data What we are able to do in the eight is a completely different Approach and It's way nicer if you want because you let's say you retrieve the current language But it doesn't matter which actual active language you use You retrieve a translation for that language and then you just call the the function With the entity object and the function can be written as if it were Uh Were passed a regular entity object if it needs language it can just access it through the language Method if it doesn't it can be written as a language diagnostic piece of code You don't need all that funky stuff we had before earlier this one This one is not needed anymore as you can see Which is very nice So as I was telling you you may need to determine the active language, okay This was typically the way we did that in d7 This is not exactly ideal because Field language performs field language fallback, which means that if a field value is not defined it for a particular field We fall back to another language. The problem is that this was originally intended to work With all field values together Because it was a way to provide Fallback to the entire entity translation, but we had no anti-translation concept in d7 So the side effect was if just a single field was missing You get just that field in a different language, which can could be very confusing In d8 we revamped completely all these stuff and basically Provided this single method that performs entity language negotiation Which means that internally the the entity manager Figures out which is the best most appropriate translation for the context its best And returns the translation. So let's say we ask The current language, but we have no translation for the current language The entity manager will check out will apply some logic Which is swappable and modules can alter it and Based on the context it's provided. It may Decide that another language is Appropriate and return it. So we still have a translation object to work on And the important another important thing is that if you have an if you have an empty value An empty field in a particular language. You don't get That in another language when displaying or working with the entity which could be very problematic This is the context that was referring to Let's imagine we have we are implemented We are implementing some code dealing with tokens And we have no line code specified in the options. We can default To this value, which just tells the system to default to the original language And then we specify this context here When we do that, we are actually providing more information to all the fallback system that actually can forward this information to the Hook implementations that can use it to decide whether a different logic is needs to be applied in Let's say when creating tokens If not default context is provided the entity view context is assumed This is a random string. You can provide any string that makes sense to you and the entity view context just Tries to return the the current language and fall back to the The next language Using language weights as configured in the language configuration UI And this works natively Fine with view builders that display the current language and form handlers that edit the Current language the current version of the the translation Another couple of of examples when we are dealing with translatable fields, I'm sorry untranslatable fields Even if we create Different objects the values are shared through and through references So if we alter an available foreign translatable field on the entity object And then we do it again on the translation object The values actually Changed as you can see We see we set more here and then entity field untranslatable as a value bar You just need to be careful and not Serialize these objects. Otherwise the field reference the references internal references will be lost And this won't work anymore, but this is not a common case And so as you as I was saying the actual data values are shared among all the Translation objects This is the last example. I think There is one more As you can see we have an api to Retrieve the the current language So this method just returns the the language associated with this translation object And we can retrieve The an entity the original version the untranslated version from any object So translation get untranslated or entity or entity get untranslated would return the same object as you can see here And then this is the typical way to get the original language You just retrieve the original translation object and retrieve its language and that's all And then you you are able to iterate over all the exist available translations You are able to check whether a translation exists at the new one providing a few default values Which is the same of doing this Aside from the fact that if this code is not installed in the system, you will get an exception And then this is important to avoid Breaking some code that might try to act after a translation has been removed When you call a remote translation on an entity object Any translation object that was instantiated from it with the removed language code Will throw in exception every time you try to access this field. So it's not possible to do Let's say try to update a value that then will be deleted. So we'll lose your data Okay, I wanted to mention One thing as I was saying before hose was not completely happy of this approach of having a single entity old in old translation, but as wappable Uh Classes allows us to provide a different implementation of the class object So he might even be able to keep the the very same interface the very same api But the backend might be implemented with different entity objects So this is I think a good signal that the api the api is well designed And then we have two just to store hooks because when Translation are added or removed from the storage. We get these hooks the update is not needed because we just have our regular entity update hook And this is a whole oh so What is what is missing? Sorry As I was telling you before we are not able to Switch table layouts dynamically yet because there are still a few encoded SQL queries in the storage handlers. So we are working on make those Query the the database dynamically So they are able to adapt to any table layout we support and this includes work on on the view side Then we have entity translation of testing which is a very complex way to to say that we are not Loading yet the the translation corresponding to the current language when when we Are upcasting so we are translating from a url placeholder to the actual object. So let's say our usual node one is provides us node one in the original language instead of Uh Taking you know into account the current language actually so this is an easy change But we want to do it as soon as possible because this way When you are working on a custom route corrals already work with this stuff But when you are implementing your custom routes the the object you will get when When Returning the related attribute from the request will be already in the Current language, so you will won't have to retrieve it And then we have a few Usability improvements for the content translation models Which is not really in the in a good shape on that side because we had more basic stuff to work on So these are all topics for the the Sprints upcoming sprints if you want to join us And then a last slide On the migration api which as you know is now in core and that will probably allow us to Support also cases where we want to switch from We want to switch table layouts when we already created data. We just have to integrate everything so that When we run a migration All the data is stored in the new tables It will probably require some code, but I don't think so much And then we have to write a migration to upgrade the old node based translations And migrate all the field data. So this is something That we can work on during the sprints also A few more links for when you will download these Presentation slides This is the celebration video. You have to watch it. I won't reveal anything about it This is the improvements meta issue for the content translation module This is the Drupal 8 multilingual initiative Official website, which is run by Gabor in a fantastic way And this he this is his Personal site with where it you can find a lot of usable information of our on our progress And again remember to attend the sprints Friday because we have all sorts of work For you. So you just you just have to come And that's it. I think that's a question question time So I want to answer actually the question that probably most of you has been wondering when you came here Because I won't tell you what is the permutated API actually, but you can discover it if you listen to the This sound so I will invite you to do so and if you have questions, I'm it I'm here. Sorry Okay So thank you